Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/scripts/lanes.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,11 @@ if (command === "build") {
const build = ({ label, runner, image, package_os, package_cpu, test }) => ({ label, runner, image, package_os, package_cpu, test });
console.log(`build=${matrix(lanes.filter(lane => !lane.test).map(build))}`);
console.log(`build_tested=${matrix(lanes.filter(lane => lane.test).map(build))}`);
console.log(`test=${matrix(lanes.filter(lane => lane.test).map(lane => ({ label: lane.label, runner: lane.test_runner, quick: lane.test_quick })))}`);
// TEMPORARY, not for merging: every tested lane is tested TEST_REPEATS times over, side by side, against the one build, to
// count the tests that fail only some of the time.
const TEST_REPEATS = 15;
const repeats = Array.from({ length: TEST_REPEATS }, (_, i) => i + 1);
console.log(`test=${matrix(lanes.filter(lane => lane.test).flatMap(lane => repeats.map(repeat => ({ label: lane.label, runner: lane.test_runner, quick: lane.test_quick, repeat }))))}`);
Comment on lines +323 to +327

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Every push to main and every PR will now run 30 test legs instead of 2, and any one of them failing deletes the release. The block at lanes.mjs:323-327 is commented "TEMPORARY, not for merging" but is still here: TEST_REPEATS = 15 fans each tested lane out into 15 matrix entries. Since release now needs test, one flaky leg out of 30 blocks autobuild-{sha}. Fix: revert the TEST_REPEATS scaffolding so plan emits one test entry per tested lane, and drop the matching matrix.repeat bits at ci.yml:315 and ci.yml:494. The PR description only mentions reverting the failing-test commit, not this one.

Extended reasoning...

lanes.mjs:325 sets TEST_REPEATS = 15; line 327 flatMaps each tested lane over repeats, so the test matrix has 2 lanes × 15 = 30 entries. ci.yml:514 adds test to release.needs, and ci.yml:528 marks the run failed if any needed job did not succeed, so a single flaky leg out of 30 deletes the draft (ci.yml:551) and nothing is published. The PR itself cites stress/sampling-profiler-bound-function-name.js failing ~3/334 on Linux x64; across 15 x64 runs that is roughly a 13 % chance per push, before counting arm64. On the base branch there were 2 legs and they did not gate the release at all. CI cost also rises ~15×: 30 jobs of ~11 minutes each on every run. The comment on line 323 explicitly says "TEMPORARY, not for merging". Related scaffolding that should go with it: ci.yml:315 appends #${{ matrix.repeat }} to the job name (which YAML actually treats as a comment because of the space before #, so the repeat number never shows anyway) and ci.yml:494 appends -${{ matrix.repeat }} to the artifact name.

Verification: normal — acknowledged in diff: the comment "TEMPORARY, not for merging" (lanes.mjs:323) is accurate and the note's claim holds, but the scaffolding is still in the change being merged; removing it leaves the PR's stated purpose (tests gate the release) met in full. .github/scripts/lanes.mjs:325 sets const TEST_REPEATS = 15; and :327 emits `test=${matrix(lanes.filter(lane =>… | normal — the…

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is deliberate and temporary: the 15x fan-out is there to count intermittently failing tests before the gate goes in, and it is reverted before this is merged. The pull request description did not say so and now does ("Not for merging yet"). Leaving this thread open until the revert is pushed.

console.log(`labels=${JSON.stringify(labels)}`);
// Only the toolchain images that are not in the registry yet get an `image` job: normally none.
const missing = images.filter(({ image }) => {
Expand Down
26 changes: 16 additions & 10 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
name: CI

# Builds every lane of bun-webkit, publishes them as one GitHub release, and tests the lanes that are marked as tested.
# Builds every lane of bun-webkit, tests the lanes that are marked as tested, and publishes them all as one GitHub
# release when every lane built and every test passed.
# A push to main and a pull request run exactly the same jobs; they differ in the commit that is built (env.REF), in
# the name of the release (the `plan` job's `tag`), and in that a pull request gets a comment linking its release.
#
# plan ──> image ──> build, build-tested ──> release
# └────────> test
# plan ──> image ──> build ───────────────────> release
# └──> build-tested ──> test ──┘
# ──> prune (main only)
#
# A lane is one <label>.tar.gz on the release. The lanes (which runner, which Dockerfile, which settings, tested or not)
Expand Down Expand Up @@ -307,11 +308,11 @@ jobs:
# Tools/Scripts/run-javascriptcore-tests with the jsc shell that a tested lane ships in its tarball's bin/, on a
# runner of the architecture it was built for. Which lanes are tested, and why those, is in lanes.mjs.
#
# `release` does not wait for this job, so a test failure never holds back or deletes a release: the run goes red,
# the release is still published. --no-fail-fast below means every test runs and every failure is reported before
# the job fails.
# `release` waits for this job: when any test fails, on any tested lane, the draft is deleted and nothing is
# published, on main as on a pull request. A test that fails once in a while is a failing test: nothing is run a
# second time. --no-fail-fast below means every test runs and every failure is reported before the job fails.
test:
name: Test ${{ matrix.label }}
name: Test ${{ matrix.label }} #${{ matrix.repeat }}
# It runs the branch's code, and only needs to check it out: the jsc shell is an artifact of this run.
permissions:
contents: read
Expand Down Expand Up @@ -490,7 +491,7 @@ jobs:
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: jsc-test-results-${{ matrix.label }}
name: jsc-test-results-${{ matrix.label }}-${{ matrix.repeat }}
path: |
${{ runner.temp }}/jsc-results.json
${{ runner.temp }}/jsc-failures.log
Expand All @@ -500,15 +501,17 @@ jobs:

release:
name: release
# Runs whatever happened to the builds (also when the run was cancelled), so the draft never outlives the run: it
# is published when every build job succeeded and every listed asset is on it, and deleted otherwise.
# Runs whatever happened to the builds and the tests (also when the run was cancelled), so the draft never outlives
# the run: it is published when every build job succeeded, every test leg passed and every listed asset is on it,
# and deleted otherwise.
if: always() && needs.plan.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 15
needs:
- plan
- build
- build-tested
- test
Comment thread
dylan-conway marked this conversation as resolved.
Resolved
steps:
# Every lane uploads <label>.tar.gz. The lanes are the release: a label missing from it fails the job.
- name: Publish or delete the draft
Expand All @@ -526,6 +529,9 @@ jobs:
if [ -n "$failed" ]; then
echo "jobs that did not succeed:"; echo "$failed"; ok=false
else
# `plan` made the draft. It is gone if this is a re-run of only the jobs that failed, after this job deleted
# it in the earlier attempt (a failed test leaves every lane built, so nothing before here notices).
gh api "repos/$GH_REPO/releases/$RELEASE_ID" --jq .id >/dev/null 2>&1 || { echo "::error::no release for $TAG (id $RELEASE_ID): an earlier attempt of this run deleted the draft. Use 'Re-run all jobs' so that plan creates it again."; exit 1; }
jq -r '.[]' <<< "$LABELS" | sort > expected.txt
gh api "repos/$GH_REPO/releases/$RELEASE_ID" --jq '.assets[].name' | sed 's/\.tar\.gz$//' | sort > uploaded.txt
missing=$(comm -23 expected.txt uploaded.txt)
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,9 @@ The lanes are defined in `.github/scripts/lanes.mjs` and nowhere else: label, ru

Each lane is built by a Dockerfile (`Dockerfile`, `Dockerfile.musl`, `Dockerfile.macos`, `Dockerfile.windows`, `Dockerfile.freebsd`, `Dockerfile.android`). In every one of them the `base` stage is the toolchain and nothing else (compilers, SDKs, sysroots): it must not take any lane setting, so that it is identical for every lane of a platform and architecture; lane settings go in the stages on top (`lane`, `build_icu`, `build_webkit`). Everything a lane needs from the network is in `base` too (packages, SDKs, sysroots, ICU's sources and host tools, `mig`, zstd, node): the stages on top must not download or install anything, so a lane's only network use is pulling its image. CI keeps those `base` stages as images in `ghcr.io/oven-sh/bun-webkit-build-env`, tagged `<toolchain>-<hash>`, the hash being of the Dockerfile up to the end of `base` (less comments and lane-only ARGs), the files `base` copies in and the build arguments it takes. The `image` job builds and pushes the ones whose tag is missing (so only after a change to a `base` stage), failing if it cannot, and each lane starts from its image (`lanes.mjs build --base-image`, which is `--build-context base=docker-image://...`). Without `--base-image` the toolchain is built as part of the build. Old images are not kept around: a build of `main` runs `prune`, which deletes every image that `main`'s lanes do not use, that is more than 14 days old and that is not one of the two newest of its toolchain: images a pull request pushed recently survive while a toolchain change is being worked on, and so does the image `main` used until the last change.

The release starts as a draft, each lane uploads its tarball onto it, and the `release` job publishes it once every lane succeeded (or deletes the draft when one failed).
The release starts as a draft, each lane uploads its tarball onto it, and the `release` job publishes it once every lane succeeded and every test leg passed (or deletes the draft when a lane failed to build or a test failed).

The `test` job runs `Tools/Scripts/run-javascriptcore-tests` (JSTests, LayoutTests/js, the PerformanceTests collections) and `testFFI` against the `bin/jsc` built by the lanes marked `tested` in `lanes.mjs`, as soon as those lanes are built, each on a runner of its own platform and architecture: the Linux x64 and arm64 lto lanes, the builds Bun ships there (about 11 minutes each). Nothing else is tested. The asan lanes are built and published but not tested: the tests take 65 to 100 minutes against them on Linux and macOS and do not finish in 275 on Windows; no tested lane has assertions on. The macOS and Windows lanes are not tested either. The job can run them, on GitHub's standard runners (3 or 4 cores, so with `--quick`, which also skips the `//@ slow!` tests; on Windows with Windows' own perl and `--test-writer=ruby --ruby-runner`, as upstream's Windows bots did): the comment at the top of `lanes.mjs` has the `tested` entries. macOS took 16 minutes; Windows x64 took 145, because `--ruby-runner` runs the tests one at a time. It runs every test before failing (`--no-fail-fast`) and turns the run red when any fail, but does not gate the release, which is still published; the step's own log shows each `FAIL:` as it happens, a progress line and the runner's results (the full output is over 100,000 lines, more than the log viewer shows); the failing tests are listed in the job summary (whether a failing test is one to change is what `JSTests/BUN-TEST-DIFFERENCES.md` is for), and the workflow artifact has `jsc-failures.log` (what each failing run printed, with its repro command, grouped under its name), the full `jsc-tests.log` and the results JSON. To run them against an asan build locally (`--no-slow`: the tests marked `//@ slow!` pass the 300 second hard timeout under ASan; `--asan` sets `$asan` in `run-jsc-stress-tests`, for `//@ skip if $asan`): extract `bun-webkit/bin` from the asan tarball and run `ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=1:allow_user_segv_handler=1 JSCTEST_memoryLimit=4294967296 Tools/Scripts/run-javascriptcore-tests --no-build --root=<path>/bun-webkit --release --jsc-only --no-testmasm --no-testair --no-testb3 --no-testdfg --no-testapi --no-testwasmdebugger --no-testlibjsctools --no-fail-fast --memory-limited --no-slow --asan` (`--quick` instead of `--no-slow` for what the macOS and Windows legs run; add `--filter <regex>` for a subset).
The `test` job runs `Tools/Scripts/run-javascriptcore-tests` (JSTests, LayoutTests/js, the PerformanceTests collections) and `testFFI` against the `bin/jsc` built by the lanes marked `tested` in `lanes.mjs`, as soon as those lanes are built, each on a runner of its own platform and architecture: the Linux x64 and arm64 lto lanes, the builds Bun ships there (about 11 minutes each). Nothing else is tested. The asan lanes are built and published but not tested: the tests take 65 to 100 minutes against them on Linux and macOS and do not finish in 275 on Windows; no tested lane has assertions on. The macOS and Windows lanes are not tested either. The job can run them, on GitHub's standard runners (3 or 4 cores, so with `--quick`, which also skips the `//@ slow!` tests; on Windows with Windows' own perl and `--test-writer=ruby --ruby-runner`, as upstream's Windows bots did): the comment at the top of `lanes.mjs` has the `tested` entries. macOS took 16 minutes; Windows x64 took 145, because `--ruby-runner` runs the tests one at a time. It runs every test before failing (`--no-fail-fast`) and fails when any do, and a failed test leg on any tested lane stops the release: the draft is deleted and nothing is published, on `main` (no `autobuild-{sha}`, so Bun cannot move to that commit) as on a pull request (no preview). A test that fails intermittently is a failing test: nothing is rerun, and the fix is to the engine or to the test (see `JSTests/BUN-TEST-DIFFERENCES.md`); the step's own log shows each `FAIL:` as it happens, a progress line and the runner's results (the full output is over 100,000 lines, more than the log viewer shows); the failing tests are listed in the job summary (whether a failing test is one to change is what `JSTests/BUN-TEST-DIFFERENCES.md` is for), and the workflow artifact has `jsc-failures.log` (what each failing run printed, with its repro command, grouped under its name), the full `jsc-tests.log` and the results JSON. To run them against an asan build locally (`--no-slow`: the tests marked `//@ slow!` pass the 300 second hard timeout under ASan; `--asan` sets `$asan` in `run-jsc-stress-tests`, for `//@ skip if $asan`): extract `bun-webkit/bin` from the asan tarball and run `ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=1:allow_user_segv_handler=1 JSCTEST_memoryLimit=4294967296 Tools/Scripts/run-javascriptcore-tests --no-build --root=<path>/bun-webkit --release --jsc-only --no-testmasm --no-testair --no-testb3 --no-testdfg --no-testapi --no-testwasmdebugger --no-testlibjsctools --no-fail-fast --memory-limited --no-slow --asan` (`--quick` instead of `--no-slow` for what the macOS and Windows legs run; add `--filter <regex>` for a subset).

## Architecture Notes

Expand Down
10 changes: 8 additions & 2 deletions JSTests/BUN-TEST-DIFFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ be sorted into one of three piles:
2. **The test cannot pass in a mode or build we run.** Skip it in that mode only, with the reason. Section B.
3. **A known gap or bug in the fork.** Skipped with `TODO(bun)`. Section C. Remove the skip when the gap is closed.

Anything that fits none of these is a regression from the sync: fix the engine, not the test.
Anything that fits none of these is a regression: fix the engine, not the test.

Rules for an edit to an upstream test:

Expand Down Expand Up @@ -152,7 +152,13 @@ Rules for an edit to an upstream test:

---

## E. After a sync: a failing test, step by step
## E. A failing test, step by step

A failing test on a tested lane stops the release (`.github/workflows/ci.yml`: `release` needs `test`): no
`autobuild-{sha}` from `main`, no preview from a pull request. That includes a test that fails once in a hundred runs.
Nothing is run a second time and there is no list of tests that are allowed to fail sometimes: an intermittent failure
has a cause, in the engine or in the test, and it goes through the same steps as any other. `jsc-failures.log` in the
run's artifact has what the failing run printed and the command that reproduces it; run that command in a loop.

1. Does the difference match A1–A9 exactly (same text, same shape)? Check that the fork code named there is still in the
tree, then update the expectation with a `// Bun:` comment and add the file to its list here.
Expand Down
Loading