Skip to content

CI: run all of test262 on every tested lane and fail unless the shell writes the checked-in record - #933

Draft
robobun wants to merge 1 commit into
mainfrom
robobun/9b175542/ci-test262
Draft

robobun wants to merge 1 commit into
mainfrom
robobun/9b175542/ci-test262

Conversation

@robobun

@robobun robobun commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Draft for @dylan-conway and @sosukesuzuki: nobody asked for this. Your decisions are first in the notes.

Problem

Fix

  • .github/scripts/test262.sh runs the whole suite with upstream's runner and --save into a copy of the record. It fails unless the copy is the checked-in file, byte for byte.
  • skipped-linux.txt, the tests that config.yaml skips, is held the same way. The test job runs the script on every tested lane.
  • Verified on CI's runners with the published shells of d6e9f85: 90 rounds, six lanes, 0 red.
  • Self-reviewed: 19 concerns raised, 17 addressed. Not taken: a time limit in the script, a skip of the slow test.

Background

  • test262 is tc39's conformance suite: 101,982 runs of jsc, one per test and mode.
  • A record entry is a test, a mode and an exit code. --save from a whole run writes it from nothing.
  • Considered test/intl402 alone with the runner's exit status: 22 of 126 rows, none of the wrong 23.

Downsides

  • Each test leg takes 2 to 5 minutes longer. Most runs end 3 minutes later.
  • In six weeks of main the failing set changed at three upstream syncs and at no commit of the fork.
  • A head that predates the script fails the step: merge main.
Notes

For the owner of ci.yml: four decisions

  1. The two-sided rule. The step fails for a run that is in the record and passes. Upstream's bots fail only for a run that fails and is not in it (RunTest262Tests in Tools/CISupport/build-webkit-org/steps.py counts ! NEW FAIL lines). That is how 15 runs that pass stayed in the record. With the rule, a pull request that fixes a test commits the record too, and a sync is red each time upstream has fixed a test and left the run in its Linux record. The one-sided rule is the runner's own exit status.
  2. A step of test, or a job of its own. It is a step: the leg has the shell and the tree. Upstream has both shapes (TestJSFactory runs test262 behind the JSC tests on the same worker, Test262Factory runs it alone). A step waits behind run-javascriptcore-tests. An amd64 Test leg is often the last job of a run (at 191c6a4 it ended at +81.9 minutes and release at +79.6), so most runs end about 3 minutes later. A job (needs: [plan, build-tested], three sparse paths, the jsc-shell-<label> artifact) ran on the harness with 17 to 29 s outside the script. It runs beside test, has its own name when it is red, and waits for six more runners. The script is the same in both.
  3. CI: nothing is published when a test fails #659. If release waits for test, this step is in the release gate. A job of its own is not, unless release names it.
  4. One slow test. test/staging/sm/regress/regress-1507322-deep-weakmap.js is a chain of 99,999 WeakMap entries, the case [JSC] Mark a chain of WeakMap entries in a bounded number of rounds #883 makes fast. It passes. It takes 38 to 92 s per mode, and both modes run in one worker: about two thirds of the step. One line under skip.files of config.yaml until [JSC] Mark a chain of WeakMap entries in a bounded number of rounds #883 brings the step to about 1 minute on x64 and 2 on arm64.

Nobody asked

The 23 runs, by cause

Before this change the record was upstream's file, byte for byte.

  • 11 runs in 6 files pass. Upstream fixed them and changed expectations.yaml alone: WebKit/WebKit e73985a (2026-08-30), a848db6 (2026-09-17), da75c48 (2026-09-19). They are still in upstream's Linux record. Upstream removes such rows by hand ([GLIB][JSC] Remove stale test262 expectations that now pass on WPE WebKit/WebKit#68724, [Tools][JSC][Test262] Update expectations for Linux after 318774@main WebKit/WebKit#71981).
  • 4 runs pass here and fail upstream: test/staging/sm/RegExp/replace-sticky.js and replace-sticky-lastIndex.js, both modes. They are in both of upstream's records.
  • 7 runs fail on purpose: the six tests of the feature import-text. ScriptFetchParameters::parseType makes "text" a host-defined type under USE(BUN_JSC_ADDITIONS), and the jsc shell loads such a file as JavaScript. config.yaml now skips the feature.
  • 1 run fails for a gap that JSTests/BUN-TEST-DIFFERENCES.md has under C: test/language/expressions/dynamic-import/import-defer/import-defer-transitive-async-module/promise-prototype-then-not-called.js (module). import() and import.defer() call a replaced Promise.prototype.then once. It is in the record now.

The record has 112 runs in 67 files (126 in 74 before). 25 lines differ from upstream's expectations-linux.yaml (0 before), 6 from its config.yaml (2 before).

Six weeks of main, run again

62 of the 65 commits of main from c775a5d (15 September) to d6e9f85 have a release. A workflow that is not in this pull request ran all of test262 with the bun-webkit-linux-amd64-lto shell of each (run 40714016980), with the tests and config.yaml of this branch.

from runs that fail what changed
c775a5d (15 September) 135
7465044, sync #725 128 7 runs start to pass
b1e30d0, sync #814 119 9 runs start to pass
8deea35, sync #889 112 7 runs start to pass
  • The failing set changed at three commits. Each is an upstream sync, and each change is runs that start to pass.
  • No commit of the fork changed one of the 101,982 results. In these six weeks the step would have caught no regression.
  • Its work would have been the record: each of the three syncs left runs in it that pass.

Measured on CI's runners

The first run of this pull request through ci.yml itself (run 40731447608, a582ab7 on d6e9f85): the test262 step is green on all six Test legs, in 152 to 215 s. Each leg wrote the two checked-in files. The step that installs what the runner needs took 0 to 1 s: both images have python3-yaml. The artifact of a leg has 6 files, 722,885 bytes on amd64 -lto-aot (721,192 before).

The branch was then rebased onto f495db1 (#930). The run of that head (run 40749995010, 442dd63) is green on all six Test legs too, the step in 193 to 222 s.

Before the pull request, a harness ran .github/scripts/test262.sh on the shells that CI published, on the runners of the test job, with nothing built.

the 90 rounds x64 (32 cores, Ubuntu 24.04, Python 3.12.3, PyYAML 6.0.1) arm64 (16 cores, Ubuntu 22.04, Python 3.10.12, PyYAML 5.4.1)
wall per round, min / median / max 208 / 215 / 220 s 185 / 189 / 270 s
CPU per round, user + system 1,545 / 1,673 / 1,716 s 880 / 938 / 1,279 s
regress-1507322-deep-weakmap.js, per mode 72 / 76 / 83 s 57 / 61 / 88 s
  • Over all 174 rounds the wall time is 156 to 220 s on x64 and 130 to 279 s on arm64. The processor differs from runner to runner.
  • A test leg takes 9 to 17 minutes today and starts about 125,000 jsc processes in run-javascriptcore-tests. The step adds 101,982.
  • The step writes 19 MB to RUNNER_TEMP. The artifact gets test262/test262.log, the two files the leg wrote and, when the leg is red, test262/test262.diff.
  • Two lanes that test does not run, bun-webkit-linux-amd64-assertions-aot and -arm64-assertions-aot, wrote the same two files in 4 rounds of 4 (246 to 271 s). No lane with assertions runs test262 in CI.

The script's verdicts

Eight stand-in shells, each a run of the whole suite. They run the real jsc for the tests in the record and for test/intl402/Locale, and pass every other test at once.

stand-in exit what the script prints
changes nothing 0
a run in the record passes 1 3 lines with -
a run that is not in the record fails 1 3 lines with +
a run dies of SIGSEGV 1 + strict mode: 139
both of the two above 1 the same count of failures, other lines
a run in the record fails with 134 in place of 3 1 - default: 3, + default: 134
the list lacks a test that config.yaml skips 1 +test/... in skipped-linux.txt
a run never ends 1 the run is named at 154, 214 and 274 s, then + strict mode: 1 and "did not end in the 300 s"

A runner that ends before it saves, a jsc that is not there and a python3 without PyYAML: exit 1, each with an ::error:: line. Off Linux: one line, exit 0. actionlint 1.7.12 with shellcheck 0.11.0 has no finding on ci.yml or on the script.

Nothing is downloaded

The runner imports PyYAML and, through webkitcorepy's TaskPool, tblib. Upstream's autoinstaller gets them and seven more packages from pypi.org by version, with no hash. The script turns it off. PyYAML is the runner image's python3-yaml: 6.0.1 and 5.4.1, each with libyaml, and the runner writes the same bytes with both. tblib 1.7.0 is four files, 511 lines, BSD 2-Clause, in .github/scripts/python/tblib (ionelmc/python-tblib at 6f769cf6). If you do not want them in the tree, the other way is a git fetch of that commit in the step.

What nothing holds

  • JSTests/test262/expectations.yaml (97 runs), which the runner reads on macOS and Windows. No lane is tested there. The docs now say that this fork does not edit it.
  • That a skip in config.yaml is still needed. A test that is skipped is not run.
  • jsc with bun's options. Bun turns on useV8DateParser, and test/staging/sm/Date/two-digit-years.js is in the record for that reason alone.
  • A run of Tools/Scripts/test262-runner by hand still exits 0 for a run that is in the record and passes.

Other pull requests

Self-review

19 concerns. Addressed: no download (it was two archives with a SHA-256), the list of skipped tests, what a red leg says about main and about the 300 s limit, the name of the artifact as an argument, four wrong or unmeasured sentences in the docs, the rule for expectations.yaml, the options of jsc against bun's, and this body (no request, the 23 runs by cause, the six weeks, both shapes with their numbers). Not taken: a time limit inside the script (the step has 30 minutes, and the script now names every jsc that runs for more than 120 s), and the skip of regress-1507322-deep-weakmap.js, which is decision 4.

@robobun

robobun commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: draft. It waits for the decisions at the top of the notes in the description (the two-sided rule, a step or a job).

How it was checked

  • Two runs of this pull request through ci.yml itself. On d6e9f85 (run 40731447608) and, after a rebase, on f495db1 (run 40749995010): the test262 step is green on all six Test legs, in 152 to 222 s, and each leg wrote the two checked-in files.
  • Before that, .github/scripts/test262.sh ran on the six published shells of d6e9f85 on the same runners: 90 rounds, 0 red (run 40714016936).
  • Eight stand-in shells give the verdicts in the description: one green, seven red.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Preview build of 442dd63: autobuild-preview-pr-933-442dd63d

… say what the shell does with it

The `test` job has a `test262` step. .github/scripts/test262.sh runs all of JSTests/test262 with upstream's
Tools/Scripts/test262-runner and has it write the record of the runs that fail again (--save, into a copy). The
step fails unless that copy is JSTests/test262/expectations-linux.yaml, byte for byte: a run that fails and is not
in the record fails the leg, and so does a run that is in the record and passes. In the same way it writes
JSTests/test262/skipped-linux.txt, the list of the tests that config.yaml takes out of the run.

The record is now the one the tested shells write. 15 runs that pass leave it. One run enters it: the import-defer
test that counts the calls of a replaced Promise.prototype.then, a known gap. The six tests of import-text are
skipped in config.yaml, because `with { type: "text" }` is a host-defined type in this fork.

Nothing is downloaded for it. webkitcorepy's autoinstaller is off, PyYAML is the runner image's, and tblib 1.7.0,
which webkitcorepy's TaskPool imports, is in .github/scripts/python (BSD 2-Clause, ionelmc/python-tblib at 6f769cf6).
@robobun
robobun force-pushed the robobun/9b175542/ci-test262 branch from a582ab7 to 442dd63 Compare October 28, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant