Repository navigation
CI: run all of test262 on every tested lane and fail unless the shell writes the checked-in record - #933
Draft
robobun wants to merge 1 commit into
Draft
CI: run all of test262 on every tested lane and fail unless the shell writes the checked-in record#933robobun wants to merge 1 commit into
robobun wants to merge 1 commit into
Conversation
Collaborator
Author
|
Status: draft. It waits for the decisions at the top of the notes in the description (the two-sided rule, a step or a job). How it was checked
|
|
Preview build of 442dd63: |
… say what the shell does with it
The `test` job has a `test262` step. .github/scripts/test262.sh runs all of JSTests/test262 with upstream's
Tools/Scripts/test262-runner and has it write the record of the runs that fail again (--save, into a copy). The
step fails unless that copy is JSTests/test262/expectations-linux.yaml, byte for byte: a run that fails and is not
in the record fails the leg, and so does a run that is in the record and passes. In the same way it writes
JSTests/test262/skipped-linux.txt, the list of the tests that config.yaml takes out of the run.
The record is now the one the tested shells write. 15 runs that pass leave it. One run enters it: the import-defer
test that counts the calls of a replaced Promise.prototype.then, a known gap. The six tests of import-text are
skipped in config.yaml, because `with { type: "text" }` is a host-defined type in this fork.
Nothing is downloaded for it. webkitcorepy's autoinstaller is off, PyYAML is the runner image's, and tblib 1.7.0,
which webkitcorepy's TaskPool imports, is in .github/scripts/python (BSD 2-Clause, ionelmc/python-tblib at 6f769cf6).
robobun
force-pushed
the
robobun/9b175542/ci-test262
branch
from
October 28, 2026 18:58
a582ab7 to
442dd63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for @dylan-conway and @sosukesuzuki: nobody asked for this. Your decisions are first in the notes.
Problem
JSTests/test262/expectations-linux.yaml, the record of the runs that fail, was wrong in 23 runs for the tested shells.test262-runnerexits 1 only for a failure outside it (Tools/Scripts/test262/runner.py:1566).Fix
.github/scripts/test262.shruns the whole suite with upstream's runner and--saveinto a copy of the record. It fails unless the copy is the checked-in file, byte for byte.skipped-linux.txt, the tests thatconfig.yamlskips, is held the same way. Thetestjob runs the script on every tested lane.Background
jsc, one per test and mode.--savefrom a whole run writes it from nothing.test/intl402alone with the runner's exit status: 22 of 126 rows, none of the wrong 23.Downsides
testleg takes 2 to 5 minutes longer. Most runs end 3 minutes later.mainthe failing set changed at three upstream syncs and at no commit of the fork.main.Notes
For the owner of
ci.yml: four decisionsRunTest262TestsinTools/CISupport/build-webkit-org/steps.pycounts! NEW FAILlines). That is how 15 runs that pass stayed in the record. With the rule, a pull request that fixes a test commits the record too, and a sync is red each time upstream has fixed a test and left the run in its Linux record. The one-sided rule is the runner's own exit status.test, or a job of its own. It is a step: the leg has the shell and the tree. Upstream has both shapes (TestJSFactoryruns test262 behind the JSC tests on the same worker,Test262Factoryruns it alone). A step waits behindrun-javascriptcore-tests. An amd64Testleg is often the last job of a run (at 191c6a4 it ended at +81.9 minutes andreleaseat +79.6), so most runs end about 3 minutes later. A job (needs: [plan, build-tested], three sparse paths, thejsc-shell-<label>artifact) ran on the harness with 17 to 29 s outside the script. It runs besidetest, has its own name when it is red, and waits for six more runners. The script is the same in both.releasewaits fortest, this step is in the release gate. A job of its own is not, unlessreleasenames it.test/staging/sm/regress/regress-1507322-deep-weakmap.jsis a chain of 99,999 WeakMap entries, the case [JSC] Mark a chain of WeakMap entries in a bounded number of rounds #883 makes fast. It passes. It takes 38 to 92 s per mode, and both modes run in one worker: about two thirds of the step. One line underskip.filesofconfig.yamluntil [JSC] Mark a chain of WeakMap entries in a bounded number of rounds #883 brings the step to about 1 minute on x64 and 2 on arm64.Nobody asked
testapi,testmasm,testb3,testair,testdfg), which the release Dockerfiles do not build, and test262."testb3,testairandtestmasmrun intestnow (testProgramsinlanes.mjs).test/intl402, after [JSC] Intl.DateTimeFormat: remove the era override for dates before an islamic or coptic epoch #922 added a row to the record from a hand-run. This is larger:test/intl402has 22 of the 126 rows.built-ins/ShadowRealmpass". [JSC] Add live module exports for module mocks, plus fifteen smaller changes for bun test #781: "Not tested: test262, workers, the inspector."The 23 runs, by cause
Before this change the record was upstream's file, byte for byte.
expectations.yamlalone: WebKit/WebKit e73985a (2026-08-30), a848db6 (2026-09-17), da75c48 (2026-09-19). They are still in upstream's Linux record. Upstream removes such rows by hand ([GLIB][JSC] Remove stale test262 expectations that now pass on WPE WebKit/WebKit#68724, [Tools][JSC][Test262] Update expectations for Linux after 318774@main WebKit/WebKit#71981).test/staging/sm/RegExp/replace-sticky.jsandreplace-sticky-lastIndex.js, both modes. They are in both of upstream's records.import-text.ScriptFetchParameters::parseTypemakes"text"a host-defined type underUSE(BUN_JSC_ADDITIONS), and thejscshell loads such a file as JavaScript.config.yamlnow skips the feature.JSTests/BUN-TEST-DIFFERENCES.mdhas under C:test/language/expressions/dynamic-import/import-defer/import-defer-transitive-async-module/promise-prototype-then-not-called.js(module).import()andimport.defer()call a replacedPromise.prototype.thenonce. It is in the record now.The record has 112 runs in 67 files (126 in 74 before). 25 lines differ from upstream's
expectations-linux.yaml(0 before), 6 from itsconfig.yaml(2 before).Six weeks of
main, run again62 of the 65 commits of
mainfrom c775a5d (15 September) to d6e9f85 have a release. A workflow that is not in this pull request ran all of test262 with thebun-webkit-linux-amd64-ltoshell of each (run 40714016980), with the tests andconfig.yamlof this branch.Measured on CI's runners
The first run of this pull request through
ci.ymlitself (run 40731447608, a582ab7 on d6e9f85): thetest262step is green on all sixTestlegs, in 152 to 215 s. Each leg wrote the two checked-in files. The step that installs what the runner needs took 0 to 1 s: both images havepython3-yaml. The artifact of a leg has 6 files, 722,885 bytes on amd64-lto-aot(721,192 before).The branch was then rebased onto f495db1 (#930). The run of that head (run 40749995010, 442dd63) is green on all six
Testlegs too, the step in 193 to 222 s.Before the pull request, a harness ran
.github/scripts/test262.shon the shells that CI published, on the runners of thetestjob, with nothing built.regress-1507322-deep-weakmap.js, per modetestleg takes 9 to 17 minutes today and starts about 125,000jscprocesses inrun-javascriptcore-tests. The step adds 101,982.RUNNER_TEMP. The artifact getstest262/test262.log, the two files the leg wrote and, when the leg is red,test262/test262.diff.testdoes not run,bun-webkit-linux-amd64-assertions-aotand-arm64-assertions-aot, wrote the same two files in 4 rounds of 4 (246 to 271 s). No lane with assertions runs test262 in CI.The script's verdicts
Eight stand-in shells, each a run of the whole suite. They run the real
jscfor the tests in the record and fortest/intl402/Locale, and pass every other test at once.-++ strict mode: 139- default: 3,+ default: 134config.yamlskips+test/...inskipped-linux.txt+ strict mode: 1and "did not end in the 300 s"A runner that ends before it saves, a
jscthat is not there and apython3without PyYAML: exit 1, each with an::error::line. Off Linux: one line, exit 0.actionlint1.7.12 withshellcheck0.11.0 has no finding onci.ymlor on the script.Nothing is downloaded
The runner imports PyYAML and, through webkitcorepy's
TaskPool, tblib. Upstream's autoinstaller gets them and seven more packages from pypi.org by version, with no hash. The script turns it off. PyYAML is the runner image'spython3-yaml: 6.0.1 and 5.4.1, each with libyaml, and the runner writes the same bytes with both. tblib 1.7.0 is four files, 511 lines, BSD 2-Clause, in.github/scripts/python/tblib(ionelmc/python-tblib at 6f769cf6). If you do not want them in the tree, the other way is agit fetchof that commit in the step.What nothing holds
JSTests/test262/expectations.yaml(97 runs), which the runner reads on macOS and Windows. No lane is tested there. The docs now say that this fork does not edit it.config.yamlis still needed. A test that is skipped is not run.jscwith bun's options. Bun turns onuseV8DateParser, andtest/staging/sm/Date/two-digit-years.jsis in the record for that reason alone.Tools/Scripts/test262-runnerby hand still exits 0 for a run that is in the record and passes.Other pull requests
ci.ymlas merged withmain, on the tree of its own head. A head without the script fails thetest262step at once with a message to mergemain, asplandoes for a head withoutlanes.mjs. On 27 October about 25 open heads were new enough for theirtestlegs to pass.mainbehind it. A change that is merged with a redtest262step leaves the step red in every branch that mergesmain, untilmainhas the files its own leg wrote. The script and the docs say to look atmain's run first.expectations.yamlalone. [JSC] Implement Promise.allKeyed and Promise.allSettledKeyed behind usePromiseAllKeyed #359 changesconfig.yaml. ICU: patch the Chinese calendar to the HKO month starts and leap months for 1901-2100 #610 patches the Chinese calendar in ICU, which 18 runs of the record are about. The step asks each of them for the files its legs write.Self-review
19 concerns. Addressed: no download (it was two archives with a SHA-256), the list of skipped tests, what a red leg says about
mainand about the 300 s limit, the name of the artifact as an argument, four wrong or unmeasured sentences in the docs, the rule forexpectations.yaml, the options ofjscagainst bun's, and this body (no request, the 23 runs by cause, the six weeks, both shapes with their numbers). Not taken: a time limit inside the script (the step has 30 minutes, and the script now names everyjscthat runs for more than 120 s), and the skip ofregress-1507322-deep-weakmap.js, which is decision 4.