Skip to content

Add seeded test order randomization - #3458

Open
q36zhd46w17o wants to merge 1 commit into
avajs:mainfrom
q36zhd46w17o:codex/ava-595-seeded-test-order
Open

q36zhd46w17o wants to merge 1 commit into
avajs:mainfrom
q36zhd46w17o:codex/ava-595-seeded-test-order

Conversation

@q36zhd46w17o

@q36zhd46w17o q36zhd46w17o commented Jun 16, 2026 •

Copy link
Copy Markdown

Summary

This adds opt-in randomized test ordering with reproducible seeds:

  • --randomize shuffles test file order and concurrent tests within each file, generating and reporting a seed for the run.
  • --seed=<value> enables the same randomization using a reproducible seed.
  • .serial tests and --serial execution continue to run in declaration order.
  • The default and TAP reporters include the seed so failures can be reproduced.
  • CLI and configuration docs describe the new options.

Fixes #595.

Validation

  • npx tap test-tap/runner.js
  • npx tap test-tap/integration/assorted.js
  • npx xo
  • npx tsc --noEmit
  • env AVA_FAKE_SCM_ROOT=.fake-root AVA_FORCE_CI=ci node ../../entrypoints/cli.js --tap --seed=ava-seed randomize-tests/test.js
  • npm test

npm test completed with the self-hosted AVA suite passing 210 tests with 4 skips, and the TAP suite passing 1633 tests.

Bounty

This is for the IssueHunt-funded #595 bounty. The implementation covers the requested random ordering, reproducible seed, and reported seed behavior.


IssueHunt Summary

Referenced issues

This pull request has been submitted to:


@woahwhattheheck

Copy link
Copy Markdown

Prepared with Codex a small portability follow-up against this exact head 54bcdd19: commit fd71b4e (five-file comparison).

The current per-file salt includes the absolute filename. This follow-up uses the project-relative path and forwards Runner.projectDir, retaining the shuffle algorithm and serial ordering.

The new actual CLI regression fails on the unchanged head and passes across checkout roots in both worker-thread and child-process modes. Patched portability, runner and assorted CLI tests pass all 131 assertions on Node 24.19.0/Linux. XO and TypeScript checks pass, with 10 existing lint warnings.

The patch is +78/-4 across five files and also applies cleanly to #3463 at ba7877be. Windows and the full suite were not run. Offered for author review and cherry-picking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Randomize test runs

3 participants