Skip to content

runner: keep docker-build and test-buffer-constants out of the N-wide phase - #35944

Closed
robobun wants to merge 3 commits into
mainfrom
farm/b68213f4/buffer-constants-oom-shard19
Closed

robobun wants to merge 3 commits into
mainfrom
farm/b68213f4/buffer-constants-oom-shard19

Conversation

@robobun

@robobun robobun commented Jul 26, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes test/js/node/test/parallel/test-buffer-constants.js going red with SIGKILL on shard 19/20 of the linux x64 test lanes.

Repro

Builds 82231, 82241 and 82296, all on unrelated feature branches with base 44f6469e0d, shard 19/20 of ubuntu 25.04 x64 / debian 13 x64:

Mem: total 7780  used 643  free 6504  ...  Swap: 0
--- Running 179 parallel-safe tests (2-wide)
[104/282] test/js/bun/test/parallel/test-docker-build-alpine.ts
[105/282] test/js/bun/test/parallel/test-docker-build-debian.ts
[106/282] test/js/bun/test/parallel/test-unref-timer-under-socket-wait.ts
[107/282] test/js/node/test/parallel/test-async-hooks-asyncresource-constructor.js
[108/282] test/js/node/test/parallel/test-async-wrap-constructor.js
[109/282] test/js/node/test/parallel/test-buffer-badhex.js
[110/282] test/js/node/test/parallel/test-buffer-constants.js
[110/282] test/js/node/test/parallel/test-buffer-constants.js - SIGKILL
main process killed by SIGKILL but no core file found

One slot is still running test-docker-build-debian.ts (10-20 s docker build --no-cache) when the other slot reaches test-buffer-constants.js.

Cause

test-buffer-constants.js does ' '.repeat(MAX_STRING_LENGTH) to prove the constant is not off by one. Under V8 that limit is 536 870 888 (~512 MiB), but JSC's WTF::StringImpl::MaxLength is 2^31 - 1, so the single-char repeat path (repeatCharacter in JSStringInlines.h) allocates and memsets a 2 GiB Latin-1 buffer:

$ bun -e 'const b=process.memoryUsage().rss; " ".repeat(require("buffer").constants.MAX_STRING_LENGTH); console.log(((process.memoryUsage().rss-b)/1024/1024).toFixed(1))'
2049.0

The x64 test runners are 7.8 GiB with no swap. With a concurrent docker build --no-cache pulling debian:trixie-slim and running apt-get install, there is not always 2 GiB free for the memset, and Linux overcommit means the tryCreateUninitialized malloc succeeds, so the process is OOM-killed during the page-fault loop instead of JSC throwing a catchable RangeError. The same shard/ordering passed on build 80215, so whether the 2 GiB fits depends on where the docker build is at that instant.

The overlap exists since #33622 (c464531) made js/*/test/parallel/ a 2-4-wide phase. The test-docker-build-* files were placed there in #25210, before the directory name was a concurrency signal, and LPT bin-packing currently puts both on shard 19.

Fix

Add a parallelSafeMemoryHeavy exclusion to isParallelSafeTest so test-docker-build-*.ts and test-buffer-constants.js run in the serial phase. They still run exactly as before (same bun run --config=bunfig.node-test.toml path), just one at a time with the machine to themselves. Shard assignment is unchanged; the four docker builds already bin-pack across shards, so wall-time impact is one extra ~12 s serial slot on the shard that carries them.

The test file itself is left verbatim (modulo the existing JSC-error-message patch); it only needs to not share the box with a docker build.

Verification

This is a runner-scheduling change with no src/** diff; the failure only reproduces when the two processes overlap on an 8 GiB no-swap host, so there is no fail-before test the gate can re-derive locally. Evidence that the fix removes the overlap:

  • isParallelSafeTest now returns false for all four test-docker-build-*.ts and for test-buffer-constants.js (Windows-separator paths included), true for the neighbouring test-buffer-badhex.js / test-http-* files (inline probe in the commit log of this branch).
  • bun test/js/node/test/parallel/test-buffer-constants.js passes standalone.
  • node --check scripts/runner.node.mjs is clean.
  • Three independent failing CI job logs (links above) all show the same slot-B test-docker-build-debian.ts still running when slot A reaches test-buffer-constants.js.

Side effect: docker-build output now reaches the log

In the parallel-safe phase these tests ran with concurrent=true, so spawnBun's stdout/stderr callbacks were () => {} and the ~350 lines of docker build --progress=plain per image were silently dropped. In the serial phase they run inside a startGroup, so that output is now written to the job log (folded under the test's group header, so not visible unless expanded). This is intentional: a docker build failure is currently invisible in CI beyond the exit code, and the extra lines are bounded (four files across all shards, folded).


[stamp-90s] gate passed · iteration 0 · 2 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/internal/expected-durations.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/internal/expected-durations.test.ts
bun test v1.4.0 (ca250f681)

test/internal/expected-durations.test.ts:
(pass) test/expected-durations.json > every lane the runner selects is declared and populated [151.32ms]
(pass) test/expected-durations.json > keys are relative test paths, not runner retry/error labels [199.60ms]
(pass) test/expected-durations.json > covers the parallel-safe phase and clamps its spans [355.04ms]
(pass) test/expected-durations.json > every entry is {lane: non-negative ms} [428.34ms]

 4 pass
 0 fail
 15 expect() calls
Ran 4 tests across 1 file. [3.47s]
Exit: 0
diff hotspot
scripts/runner.node.mjs                  | 15 +++++++++++++++
 test/internal/expected-durations.test.ts |  8 +++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                      reads  edits  tests
scripts/runner.node.mjs                       8      2      0
test/internal/expected-durations.test.ts      1      1      0

… phase

The parallel-safe phase is sized for ~50 ms startup-bound files, but two
entries on shard 19/20 are not that:

  * test/js/bun/test/parallel/test-docker-build-*.ts run a full
    `docker build --no-cache` (10-20 s, GB-scale RSS while layers are
    pulled and dpkg/apk runs). They were placed in that directory before
    #33622 made parallel/ a concurrency signal.
  * test/js/node/test/parallel/test-buffer-constants.js calls
    `' '.repeat(MAX_STRING_LENGTH)`. Under JSC that limit is 2^31-1
    (V8's is ~512 MiB), so the single-char repeat materializes a 2 GiB
    Latin-1 buffer and touches every page.

On the 7.8 GiB no-swap x64 test runners those two overlapped 2-wide and
the OOM killer SIGKILLed the buffer test (builds 82231, 82241, 82296;
shard 19 on ubuntu 25.04 x64 / debian 13 x64). Linux overcommit lets the
malloc succeed, so there is no catchable RangeError to try/catch around.

Route both back through the serial phase so each gets the machine to
itself; invocation (bun run / bun test) and shard assignment are
otherwise unchanged.
@robobun

robobun commented Jul 26, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 11:34 AM PT - Jul 26th, 2026

❌ @robobun, your commit ca250f6 has 1 failures in Build #82486 (All Failures):

  • 📦 Binary size — 12 over 0.50 MB
  • targetthis build canary: main #79916
    sizeΔ
    ❌ bun-darwin-aarch6458.13 MB57.58 MB+564.9 KB
    ❌ bun-darwin-x6463.48 MB62.95 MB+544.5 KB
    ❌ bun-linux-aarch6470.98 MB70.42 MB+576.0 KB
    ❌ bun-linux-x6472.47 MB71.95 MB+528.0 KB
    ❌ bun-linux-aarch64-musl64.88 MB64.32 MB+576.0 KB
    ❌ bun-linux-x64-musl66.98 MB66.45 MB+544.0 KB
    ❌ bun-linux-aarch64-android78.47 MB77.97 MB+512.0 KB
    ❌ bun-linux-x64-android80.62 MB80.10 MB+529.2 KB
    ❌ bun-freebsd-x6483.07 MB82.56 MB+528.0 KB
    ❌ bun-freebsd-aarch6484.84 MB84.31 MB+544.0 KB
    ❌ bun-windows-x6480.26 MB79.70 MB+570.5 KB
    ❌ bun-windows-aarch6470.86 MB70.34 MB+533.0 KB

    Add [skip size check] to the commit message if this increase is intentional.


🧪   To try this PR locally:

bunx bun-pr 35944

That installs a local version of the PR into your bun-35944 executable, so you can run:

bun-35944 --bun

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

Parallel-safe test filtering

Layer / File(s) Summary
Memory-heavy test exclusions
scripts/runner.node.mjs
Adds exclusions for Docker build and buffer constants tests, normalizes paths, and rejects those tests before existing parallel-safe classification.

Suggested reviewers: cirospaciari, dylan-conway

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly names the scheduling change and the affected tests.
Description check ✅ Passed The description covers the bug, root cause, fix, verification, and side effect details.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/runner.node.mjs`:
- Around line 614-620: Update the expected-duration classification used by
test/internal/expected-durations.test.ts to match isParallelSafeTest and
parallelSafeMemoryHeavy in scripts/runner.node.mjs, excluding Docker-build tests
and test-buffer-constants.js from concurrent treatment. Regenerate
test/expected-durations.json so these serial tests retain unclamped duration
spans.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3e407142-56a3-416d-bdd9-1a529c4c60b4

📥 Commits

Reviewing files that changed from the base of the PR and between 44f6469 and bfd3eda.

📒 Files selected for processing (1)
  • scripts/runner.node.mjs

Comment thread scripts/runner.node.mjs Resolved
The parallel-safe clamp assertion keys off the js/{node,bun}/test/parallel/
path prefix, but runner.node.mjs now routes the test-docker-build-* files
and test-buffer-constants.js through the serial phase, where
update-test-durations.mjs records real wall-clock spans instead of the
500 ms inter-dispatch clamp. Mirror the exclusion here so the next
regeneration of test/expected-durations.json does not trip the unclamped
check on those five entries.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — targeted CI-runner scheduling fix that moves five memory-heavy tests to the serial phase.

What was reviewed:

  • isParallelSafeTest is used only to partition into phase 1 (serial) vs phase 2 (N-wide) at runner.node.mjs:907-908; command dispatch is via isNodeTest (path-based), so the excluded files still run under bunfig.node-test.toml exactly as before.
  • Predicate operates on the backslash-normalized p, so Windows paths match; verified the four test-docker-build-*.ts files exist at the matched path.
  • expected-durations.json keys (js/bun/test/parallel/test-docker-build-*, js/node/test/parallel/test-buffer-constants.js) match the mirrored memoryHeavy predicate in the test exactly, and the CodeRabbit sync concern is addressed in f0c36b6.
Extended reasoning...

Overview

Two-file change to CI test-scheduling infrastructure only. scripts/runner.node.mjs gains a parallelSafeMemoryHeavy predicate that returns false from isParallelSafeTest for four test-docker-build-*.ts files and test-buffer-constants.js, moving them from the N-wide phase 2 into the serial phase 1. test/internal/expected-durations.test.ts mirrors the same exclusion so the 500 ms clamp assertion on concurrent-phase spans won't fail once the next duration-table regeneration records real wall-clock spans for these now-serial files. No src/** changes.

Security risks

None. This is CI orchestration only — a path-based predicate over an internally enumerated test list, with no user-controlled input, no auth, no crypto, no network surface.

Level of scrutiny

Low. The only consumer of isParallelSafeTest is the phase-1/phase-2 partition at lines 907–908; returning false simply routes a test into serialTests, which runs under limit (width 1 by default) with concurrent = parallelism > 1. Command selection for these files goes through isNodeTest(testPath) at line 778, which matches on directory path independently of the parallel-safe predicate — so the docker-build and buffer-constants tests still hit the same spawnBun + bunfig.node-test.toml code path they did before, just serialized. The side effect (docker-build stdout now reaching the folded log group because concurrent is false) is called out and bounded.

Other factors

  • The predicate is applied to p = testPath.replaceAll("\\\\", "/"), so includes/endsWith work on Windows paths.
  • Verified via glob that all four test-docker-build-{alpine,debian,debian-slim,distroless}.ts exist at test/js/bun/test/parallel/ and via grep that expected-durations.json keys are exactly js/bun/test/parallel/test-docker-build-*.ts and js/node/test/parallel/test-buffer-constants.js, so both the runner predicate and the test-file mirror match them.
  • CodeRabbit's one concern (keeping expected-durations.test.ts in sync) was addressed in f0c36b6 and the thread is resolved. The current JSON entries are already ≤500 ms so nothing regresses immediately; the mirror pre-empts the next regeneration.
  • The root-cause analysis (JSC MAX_STRING_LENGTH = 2³¹−1 → 2 GiB memset colliding with a concurrent docker build on an 8 GiB no-swap host) is well-evidenced with three build logs and matches the fix scope.

@robobun

robobun commented Jul 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fix verified on build 82472 and build 82486, shard 19/20 on both :ubuntu: 25.04 x64 and :debian: 13 x64: test-docker-build-alpine.ts / test-docker-build-debian.ts / test-buffer-constants.js now run in the serial phase (--- group prefix) and test-buffer-constants.js passes; the parallel-safe phase starts afterwards with 176 files instead of 179.

Remaining red on both builds is unrelated to this diff:

  • binary-size: pre-existing delta vs canary #79916 on main (no src/** change here).
  • Flaky tests that passed on retry: bun-install-registry, bun-security-scanner-matrix-with-node-modules, spawn-pipe-leak, webview-chrome, spawn-stdin-pipe-fd-leak, hot.test, bun-upgrade, jsc-stress, no-orphans.

Ready for review.

…comment

Shard placement comes from LPT bin-packing over expected-durations.json
(regenerated weekly) and maxShards is a tunable, so the hardcoded
"Shard 19/20" and bare build numbers would go stale. The predicate is
path-based and shard-agnostic anyway; the full repro with build links
lives in the PR body per REVIEW.md.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — targeted CI-runner scheduling tweak, no product code touched.

What was reviewed:

  • isParallelSafeTest still partitions the full test list (excluded files fall through to the serial phase via serialTests.filter, not dropped); execution path is governed by isNodeTest, which is unchanged, so both docker-build and buffer-constants still run via bunfig.node-test.toml.
  • Path-separator normalization happens before parallelSafeMemoryHeavy is called, so Windows paths match.
  • Mirrored opt-out in expected-durations.test.ts keeps the ≤500ms clamp assertion aligned with the runner predicate; CodeRabbit's concern is addressed and resolved.
  • Build 82472 confirms shard 19/20 now runs the three files serially and test-buffer-constants.js passes.
Extended reasoning...

Overview

Two files touched, both CI infrastructure:

  • scripts/runner.node.mjs: adds a parallelSafeMemoryHeavy predicate that excludes js/bun/test/parallel/test-docker-build-* and js/node/test/parallel/test-buffer-constants.js from the N-wide parallel-safe phase, so they run in the serial phase instead. The predicate is applied after backslash→forward-slash normalization inside isParallelSafeTest.
  • test/internal/expected-durations.test.ts: mirrors the same exclusion so the ≤500ms clamp assertion (which only applies to concurrent-phase inter-dispatch gaps) skips these files once their real wall-clock spans are recorded.

No src/** changes; the affected test files themselves are untouched.

Security risks

None. This is a test-scheduling change in the CI runner script; no auth, crypto, network, or user-facing surface is involved.

Level of scrutiny

Low. The change is a narrow path-match predicate inserted into an existing serial/parallel partition. I confirmed:

  • The partition at runner.node.mjs:906-908 covers the full tests list, so excluded files move to serialTests rather than being dropped.
  • Execution routing (isNodeTest at line 1915) is path-based and independent of isParallelSafeTest, so the docker-build tests and buffer-constants still go through the same bun run --config=bunfig.node-test.toml invocation with the same env, just with concurrent=false (which restores their stdout/stderr to the log inside a startGroup, as the PR description notes).
  • The four test-docker-build-*.ts files exist on disk and match the includes pattern.

Other factors

  • The one review comment (CodeRabbit, on keeping expected-durations.test.ts in sync) was addressed in f0c36b6 and marked resolved.
  • The author verified on build 82472 that the previously-SIGKILLed shard now runs the three files serially and passes; remaining red on that build is unrelated flakes / pre-existing binary-size delta.
  • Worst-case failure mode of this change is that five test files run serially instead of 2-4-wide, adding ~10-20s to one shard — no correctness risk to the shipped binary.

@robobun

robobun commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as part of a cleanup of stale pull requests. This PR has had no new commits since 2026-07-26, it conflicts with main, and its last CI run failed. This is not a judgment on the fix itself. If the problem still reproduces on a current build, reopen this PR after a rebase or open a new one against main.

@robobun robobun closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant