Review main Merged

80557[TASK] Add accessibility ci tests with playwright/axe

Frank Nägler CR +2 V +2 Hard +271 −16 updated 08/16/24, 07:37

Commit message

playwright/axe iterates over a list of web pages and highlights
accessibility issues. This tool is now applied as a CI job to
the TYPO3 Backend where the main backend layout and modules are
being tested.

Usage:

  # CI mode
  Build/Scripts/runTests.sh -s accessibility

  # Run against local instance
  ACCESSIBILITY_BASE_URL=https://t3main.devbox.local/typo3 \
  npm --prefix=Build run accessibility

  # Prepare CI mode, run locally (repeatable command will be printed)
  Build/Scripts/runTests.sh -s accessibility-prepare

Commands executed:
  npm --prefix=Build install --save-dev \
    @axe-core/playwright@4.9.1 \
    @playwright/test@1.45.3

Review

  • Code review CR +2 Done
    • Andreas Kienast +2
    • Benjamin Franzke +1
  • Verified V +2 Done
    • Andreas Kienast +2
    • Benjamin Franzke +1
    • core-ci +1 CI

Issues 1

Files 15

Review log 5592

Show 4067 earlier messages
  1. Frank Nägler uploaded patch set 1 08/15/23, 15:28

  2. Frank Nägler wrote 08/17/23, 08:18

    Removed reviewer Stefan Bürk.
  3. Frank Nägler uploaded patch set 2 09/12/23, 07:15

  4. Benjamin Franzke uploaded patch set 3 02/19/24, 20:12

  5. Benjamin Franzke uploaded patch set 4 02/19/24, 22:13

  6. Benjamin Franzke uploaded patch set 5 02/20/24, 05:51

  7. Benjamin Franzke uploaded patch set 6 02/20/24, 07:53

  8. Benjamin Franzke commented on patch set 6 02/20/24, 08:02

    A recent run of this pipeline can be seen on: https://git.typo3.org/typo3/CI/cms/-/jobs/3020497 It succeeds because the error count (threshold) of the staged `pa11y-results.json` file is not exceeded. That could be a "good enough" heuristic and seems to be the only one that pa11y offically supports. => Ideally we'd compare the logged errors against a baseline, we could do that using git-diff (as is `checkGruntClean`), but the TODOs listed below are still blockers for that. …and I'm not sure how "good" that scales for the daily dev workflow . Whenever structural changes are performed, this baseline needs to be regenerated (similar to the phpstan baseline).
  9. Frank Nägler uploaded patch set 7 03/04/24, 07:49

  10. Benni Mack wrote 03/04/24, 09:58

    Unset private
  11. Josef Glatz commented on patch set 7 03/05/24, 03:40

    i would definitely add an introduction text „about pa11y“ before the usage section in the git commit. - what is pa11y - and maybe: what are the primary goals
  12. Benjamin Franzke uploaded patch set 8 03/05/24, 06:36

  13. core-ci voted on patch set 8 V +1 03/05/24, 06:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53916

  14. Frank Nägler uploaded patch set 9 V +1 03/05/24, 16:42

  15. core-ci voted on patch set 9 V -1 03/05/24, 16:48 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53970

  16. Frank Nägler rebased onto patch set 10 V -1 03/05/24, 17:50

  17. core-ci voted on patch set 10 V -1 03/05/24, 17:57 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53975

  18. Benjamin Franzke uploaded patch set 11 V -1 03/06/24, 08:55

  19. core-ci voted on patch set 11 V -1 03/06/24, 09:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54005

  20. Benjamin Franzke rebased onto patch set 12 V -1 03/06/24, 09:24

  21. core-ci voted on patch set 12 V -1 03/06/24, 09:34 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54011

  22. Benjamin Franzke rebased onto patch set 13 V -1 03/06/24, 10:32

  23. core-ci voted on patch set 13 V -1 03/06/24, 10:39 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54016

  24. Benjamin Franzke rebased onto patch set 14 V -1 03/06/24, 10:45

  25. core-ci voted on patch set 14 V -1 03/06/24, 10:50 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54019

  26. Benjamin Franzke rebased onto patch set 15 V -1 03/06/24, 10:57

  27. core-ci voted on patch set 15 V -1 03/06/24, 11:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54022

  28. Benjamin Franzke rebased onto patch set 16 V -1 03/06/24, 15:20

  29. core-ci voted on patch set 16 V -1 03/06/24, 15:28 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54046

  30. Benjamin Franzke uploaded patch set 17 V -1 03/11/24, 09:56

  31. core-ci voted on patch set 17 V -1 03/11/24, 10:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54354

  32. Benjamin Franzke rebased onto patch set 18 V -1 03/16/24, 11:27

  33. core-ci voted on patch set 18 V -1 03/16/24, 11:39 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54621

  34. Benjamin Franzke uploaded patch set 19 V -1 03/19/24, 13:53

  35. Benjamin Franzke commented on patch set 19 03/19/24, 13:55

    Reply

    Done

    Reply

    => Will be solved by applying a 0-errors strategy. All issues will be resolved first and (module)URLs will then be listed in pa11y.js to enforce they do not produce errors again. We will not track a list of errors (errors will instead be logged as CI artifacts, in case of failure).
  36. core-ci voted on patch set 19 V -1 03/19/24, 14:00 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54695

  37. Benjamin Franzke uploaded patch set 20 V -1 03/19/24, 14:15

  38. core-ci voted on patch set 20 V -1 03/19/24, 14:21 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54697

  39. Frank Nägler marked it work in progress 03/22/24, 10:08

  40. Frank Nägler uploaded patch set 21 V -1 04/10/24, 07:27

  41. core-ci voted on patch set 21 V -1 04/10/24, 07:32 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55668

  42. Frank Nägler uploaded patch set 22 V -1 04/10/24, 13:45

  43. Frank Nägler uploaded patch set 23 04/10/24, 13:50

  44. core-ci voted on patch set 22 V -1 04/10/24, 13:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55685

  45. core-ci voted on patch set 23 V -1 04/10/24, 13:56 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55686

  46. Frank Nägler uploaded patch set 24 V -1 04/24/24, 09:05

  47. core-ci voted on patch set 24 V -1 04/24/24, 09:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56644

  48. Benjamin Franzke commented on patch set 25 V -1 04/24/24, 10:31

    Published edit on patch set 24.
  49. core-ci voted on patch set 25 V -1 04/24/24, 10:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56654

  50. Frank Nägler uploaded patch set 26 04/24/24, 12:05

    Copied Votes: * Verified-1 (copy condition: "**changekind:NO_CHANGE**")
  51. core-ci voted on patch set 26 V -1 04/24/24, 12:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56659

  52. Frank Nägler uploaded patch set 27 V -1 04/29/24, 10:59

  53. core-ci voted on patch set 27 V -1 04/29/24, 11:04 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56959

  54. Frank Nägler uploaded patch set 28 V -1 05/02/24, 11:15

  55. core-ci voted on patch set 28 V +1 05/02/24, 11:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57090

  56. Frank Nägler uploaded patch set 29 V +1 06/10/24, 20:35

  57. core-ci voted on patch set 29 V -1 06/10/24, 20:40 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/59076

  58. Frank Nägler uploaded patch set 30 V -1 06/12/24, 12:27

  59. Frank Nägler uploaded patch set 31 06/12/24, 18:56

  60. core-ci voted on patch set 31 V -1 06/12/24, 19:56 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/59362

  61. Frank Nägler uploaded patch set 32 V -1 06/12/24, 20:17

  62. core-ci voted on patch set 32 V -1 06/12/24, 21:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/59369

  63. Benjamin Franzke rebased onto patch set 33 V -1 08/01/24, 15:42

  64. core-ci voted on patch set 33 V -1 08/01/24, 16:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/62594

  65. core-ci commented on patch set 33 08/01/24, 16:46 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/62594

  66. Benjamin Franzke uploaded patch set 34 V -1 08/01/24, 17:21

  67. core-ci voted on patch set 34 V -1 08/01/24, 17:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/62600

  1. Frank Nägler uploaded patch set 35 V -1 08/14/24, 10:51

  2. core-ci voted on patch set 35 V -1 08/14/24, 11:38 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63370

  3. Benjamin Franzke uploaded patch set 36 08/14/24, 11:44

    Copied Votes: * Verified-1 (copy condition: "**changekind:NO_CHANGE**")
  4. Benjamin Franzke uploaded patch set 37 V -1 08/14/24, 11:58

  5. core-ci voted on patch set 37 V -1 08/14/24, 12:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63373

  6. Frank Nägler uploaded patch set 38 V -1 08/14/24, 12:29

  7. core-ci voted on patch set 38 V +1 08/14/24, 12:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63374

  8. Frank Nägler updated the commit message in patch set 39 V +1 08/14/24, 13:01

  9. Frank Nägler marked it ready for review 08/14/24, 13:01

  10. core-ci voted on patch set 39 V +1 08/14/24, 13:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63380

  11. Frank Nägler uploaded patch set 40 V +1 08/15/24, 10:02

  12. core-ci voted on patch set 40 V -1 08/15/24, 10:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63466

  13. Frank Nägler rebased onto patch set 41 V -1 08/15/24, 11:58

  14. core-ci voted on patch set 41 V +1 08/15/24, 12:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63475

  15. Frank Nägler uploaded patch set 42 V +1 08/15/24, 12:23

  16. core-ci voted on patch set 42 V +1 08/15/24, 12:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63478

  17. Frank Nägler uploaded patch set 43 V +1 08/15/24, 14:24

  18. Andreas Kienast voted on patch set 43 CR +1 V +1 08/15/24, 14:25

  19. core-ci voted on patch set 43 V +1 08/15/24, 14:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63498

  20. Benjamin Franzke uploaded patch set 44 CR +1 V +1 08/15/24, 15:07

  21. core-ci voted on patch set 44 V +1 08/15/24, 15:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63510

  22. core-ci commented on patch set 44 08/15/24, 15:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63513

  23. Benjamin Franzke voted on patch set 44 CR +1 V +1 08/15/24, 16:00

  24. Andreas Kienast voted on patch set 44 CR +2 V +2 08/16/24, 07:37

  25. Andreas Kienast merged patch set 45 08/16/24, 07:37