[test] Add axe-core tests for mui-material #48341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

siriwatknp wants to merge 49 commits into mui:master from siriwatknp:test/a11y-demos

.circleci/config.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -267,6 +267,9 @@ jobs: @@
           - run:
               name: Run visual regression tests
               command: xvfb-run pnpm test:regressions
+          - run:
+              name: A11y results committed?
+              command: git add -A && git diff --exit-code --staged
           - run:
               name: Build packages for fixtures
               command: pnpm release:build
@@ Expand Down @@

AGENTS.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -161,6 +161,46 @@ describe('Button', () => { @@
     });
     ```
+    ### Accessibility Testing
+    axe-core runs inside the visual-regression Playwright loop (`test/regressions/index.test.js`) — no separate browser session. Screenshots and a11y are independent: a demo can opt out of one and still run the other.
+    Key files:
+    - `test/regressions/demoMeta.ts` — `SCREENSHOT_RULES` and `A11Y_RULES` arrays, matched last-wins (no inheritance: overrides restate every field) against `docs/data/material/components/{slug}/{Demo}` (minimatch globs).
+    - `test/regressions/a11y/axe.ts` — asserts `color-contrast` and `link-in-text-block` unless listed in `skipAssertions`.
+    - `test/regressions/a11y/a11yReporter.ts` — writes one file per slug at `docs/data/material/components/{slug}/{slug}.a11y.json`. Each file is keyed by demo name, then by axe rule ID. Each rule records a `status` (`pass`, `fail`, or `incomplete`) and WCAG tags.
+    Enroll a component (slug-wide, or narrow with brace-glob):
+    ```ts
+    // test/regressions/demoMeta.ts
+    { test: 'docs/data/material/components/alert/*', enabled: true, skipAssertions: ['color-contrast'] },
+    { test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true },
+    ```
+    Override a specific demo: append a per-demo rule _after_ the slug-wide rule (last-match-wins; the override must restate every field it wants):
+    ```ts
+    { test: 'docs/data/material/components/popover/AnchorPlayground', enabled: false }, // Redux isolation
+    ```
+    Run `pnpm test:regressions` to refresh the `*.a11y.json` files. CI fails if any are stale.
+    For local iteration, scope the run with vitest's `-t` test-name filter (matched against the `it()` strings, which contain the route). Non-matching tests are skipped — their bodies don't execute, so the browser never navigates to those routes.
+    ```bash
+    # in one terminal
+    pnpm test:regressions:server
+    # in another — note no `--`, pnpm forwards args directly
+    pnpm test:regressions:run -t '/docs-components-buttons/'              # one slug
+    pnpm test:regressions:run -t '/docs-components-buttons/BasicButtons$' # one demo
+    pnpm test:regressions:run -t '/docs-components-(buttons|chips)/'      # multiple slugs
+    ```
+    Filtered runs only refresh the matched slugs' `*.a11y.json`. Run the unfiltered `pnpm test:regressions` before pushing.
     ### Imports
     Use one-level deep imports to avoid bundling entire packages:
@@ Expand Down @@

CONTRIBUTING.md

-Original file line number
+Diff line change
@@ Expand Up @@
     If Chrome failed then `pnpm test:browser` should<sup>[1](test/README.md#accessibility-tree-exclusion)</sup> fail locally as well.
     If other browsers failed, you can debug using `VITEST_BROWSERS=firefox,webkit pnpm test:browser`.
-    #### ci/circleci: test_regression
+    #### ci/circleci: test_regressions
-    This renders tests in `test/regressions/tests` and takes screenshots.
-    This step shouldn't fail if the others pass.
-    Otherwise, a maintainer will take a look.
-    The screenshots are evaluated in another step.
+    This builds the regression fixture app and runs Playwright in a real browser.
+    It checks for visual regressions with screenshots and accessibility regressions with axe-core.
+    If it fails, check the log for the exact error.
+    If generated accessibility results are stale, run `pnpm test:regressions` locally and commit the updated files.
+    Screenshots are uploaded and compared by Argos in a separate step.
     #### ci/circleci: test_types
@@ Expand Down @@

docs/data/material/components/buttons/buttons.a11y.json

-Original file line number
+Diff line change
@@ -0,0 +1,82 @@
+    {
+      "BasicButtons": {
+        "rules": {
+          "aria-allowed-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-conditional-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-prohibited-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-valid-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-valid-attr-value": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "button-name": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "color-contrast": {
+            "status": "pass",
+            "tags": ["wcag2aa"]
+          },
+          "nested-interactive": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "target-size": {
+            "status": "pass",
+            "tags": ["wcag22aa"]
+          }
+        }
+      },
+      "ColorButtons": {
+        "rules": {
+          "aria-allowed-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-conditional-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-prohibited-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-valid-attr": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "aria-valid-attr-value": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "button-name": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "color-contrast": {
+            "status": "pass",
+            "tags": ["wcag2aa"]
+          },
+          "nested-interactive": {
+            "status": "pass",
+            "tags": ["wcag2a"]
+          },
+          "target-size": {
+            "status": "pass",
+            "tags": ["wcag22aa"]
+          }
+        }
+      }
+    }

package.json

-Original file line number
+Diff line change
@@ Expand Up / @@ -57,7 +57,7 @@ @@
         "test:e2e:dev": "pnpm -F ./test/e2e dev",
         "test:e2e-website": "playwright test test/e2e-website --config test/e2e-website/playwright.config.ts",
         "test:e2e-website:dev": "cross-env PLAYWRIGHT_TEST_BASE_URL=http://localhost:3000 playwright test test/e2e-website --config test/e2e-website/playwright.config.ts",
-        "test:regressions": "cross-env NODE_ENV=production pnpm test:regressions:build && concurrently --success first --kill-others \"pnpm test:regressions:run\" \"pnpm test:regressions:server\"",
+        "test:regressions": "cross-env NODE_ENV=production pnpm test:regressions:build && concurrently --success first --kill-others \"pnpm test:regressions:run\" \"pnpm test:regressions:server\" && prettier --write \"docs/data/material/components/**/*.a11y.json\"",
         "test:regressions:build": "vite build test/regressions",
         "test:regressions:dev": "vite test/regressions --port 5001",
         "test:regressions:run": "vitest run -r ./test/regressions/",
@@ Expand Down Expand Up / @@ -112,6 +112,7 @@ @@
         "@vitejs/plugin-react": "^5.1.1",
         "@vitest/browser-playwright": "^4.0.13",
         "@vitest/coverage-v8": "^4.0.13",
+        "axe-core": "4.11.1",
         "babel-plugin-module-resolver": "5.0.3",
         "chalk": "5.6.2",
         "concurrently": "9.2.1",
@@ Expand All / @@ -127,6 +128,7 @@ @@
         "jsdom": "26.1.0",
         "lerna": "9.0.7",
         "markdownlint-cli2": "0.22.1",
+        "minimatch": "10.2.4",
         "nx": "20.8.4",
         "pkg-pr-new": "0.0.66",
         "playwright": "1.59.1",
@@ Expand Down @@

pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

test/regressions/README.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -16,9 +16,11 @@ @@
 . a demo from `docs/data`
-       By default all demos are included.
-       We exclude demos if they are redundant or flaky etc.
-       The logic for this exclusion is handled (like the composition) in `./index.js`
+       Most import-safe demos are included.
+       Non-demo paths and interactive or flaky components are excluded in `./index.jsx`.
+       Per-demo screenshot and accessibility rules live in `./demoMeta.ts`.
+       This keeps screenshot exclusions separate from accessibility coverage.
     If you introduce new behavior, prefer adding a demo to the documentation to solve documentation and testing with one file.
     If you're adding a new test prefer a new component instead of editing existing files since that might unknowingly alter existing tests.
@@ Expand All / @@ -33,16 +35,33 @@ @@
     ### Automatic
-    We're using [`playwright`](https://playwright.dev) to iterate over each fixture and take a screenshot.
+    We're using [`playwright`](https://playwright.dev) to iterate over each fixture in a real browser.
+    For screenshot-enabled fixtures, Playwright saves a screenshot in `./screenshots/$BROWSER_NAME/`.
+    Some demos also run axe-core accessibility checks.
+    Their results are saved next to the related component docs.
     It allows catching regressions like this one:
     ![before](/test/docs-regressions-before.png)
     ![diff](/test/docs-regressions-diff.png)
-    Screenshots are saved in `./screenshots/$BROWSER_NAME/`.
     Each test tests only a single fixture.
     A fixture can be loaded with `await renderFixture(fixturePath)`, for example `renderFixture('FocusTrap/OpenFocusTrap')`.
+    Accessibility checks are opt-in.
+    Add rules in `./demoMeta.ts` under `A11Y_RULES`.
+    Use a slug-wide rule for many demos, or a brace-glob for specific demos:
+    ```ts
+    { test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true }
+    ```
+    Filtered runs with `-t` only refresh matched slugs.
+    Use an unfiltered run when you need to refresh all generated results.
     ## Commands
     For development `pnpm test:regressions:dev` and `pnpm test:regressions:run --watch` in separate terminals is recommended.
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[test] Add axe-core tests for mui-material #48341

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

GitHub Actions / test-dev (ubuntu-latest)

Uh oh!

Uh oh!

Uh oh!

[test] Add axe-core tests for mui-material #48341

Are you sure you want to change the base?

Uh oh!

[test] Add axe-core tests for mui-material #48341

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

GitHub Actions / test-dev (ubuntu-latest)

Uh oh!

Uh oh!