Repository navigation
Conversation
WalkthroughRegression tests now use local Verdaccio registries, shared asynchronous helpers, concurrent execution, repository-local fixtures, and explicit subprocess assertions. External package installation and manual cleanup were removed. ChangesHermetic regression coverage
Possibly related PRs
Suggested reviewers: Pre-merge checks:
|
|
Status: ready for review. CI is green on On 2026-10-19 the ten files ran again on this branch merged with main How the network dependency reproduces: run the ten files with no route to the registries. Two setups show it.
In both, main fails 13 of the 16 tests, for example On this branch the 16 tests pass in both setups with This PR no longer changes PR: #37401 |
There was a problem hiding this comment.
I reviewed this and didn't find any bugs. It's a test-only change that follows the harness patterns (VerdaccioRegistry, tempDir, file: deps), and I verified the referenced fixture packages (no-deps 1.0.0/1.0.1/1.1.0/2.0.0 with latest=2.0.0, basic-1, a-dep) and packages/@types/bun all exist with the expected shape. Given the scope — ten regression test files rewritten, with strengthened/changed assertions in 26657, 28193, 11806, and a semantic shift in what 24364 validates (in-tree types instead of published @types/bun) — a human look to confirm the assertion changes preserve each regression's original intent would still be worthwhile.
What was reviewed:
- Fixture availability in
test/cli/install/registry/packages/matches every version the new tests depend on. using dir = project(...)pattern in 26657 / patch-bounds-check correctly disposes at the call site; concurrent tests use distinct tempDir prefixes and read-only registry access.nodeExe() ?? bunExe()in 24364 matches the harness signature; the ctrl-cfile:bin still createsnode_modules/.binfor all six launch variants.
Extended reasoning...
Overview
This PR converts ten test/regression/issue/ files that previously ran bun install/add/update against registry.npmjs.org into hermetic tests. Seven files now start a VerdaccioRegistry and point bunfig.toml at it, swapping real npm packages (is-even, left-pad, lodash, typescript) for local-registry fixtures (no-deps, basic-1, a-dep). Three files use different approaches: ctrl-c.test.ts replaces the vite install with a local file: package that has a bin; 26225.test.ts drops the install entirely and requires form-data from test/node_modules by absolute path (added to test/package.json); 24364.test.ts runs tsc from test/node_modules against in-repo packages/@types/bun with a stubbed bun-plugin-tailwind declaration instead of installing four packages from npm. Only test files, test/package.json, and one line of test/bun.lock are touched.
Security risks
None. Test-only change with no runtime code modified. The change reduces external network dependencies rather than adding any.
Level of scrutiny
Medium. This is test infrastructure, not shipped code, and each individual conversion follows the pattern REVIEW.md and CLAUDE.md prescribe (VerdaccioRegistry from harness, tempDir, no live registries). However, the aggregate is large (~500 changed lines across 12 files), and several tests had their assertions rewritten rather than just their package names swapped: 26657's second case went from a vacuous conditional assertion to concrete 2.0.0 checks; 28193 tightened toContain("security scanner") to the specific error names; 11806 added post-rewrite package.json parsing assertions; 24364 now type-checks against in-tree packages/bun-types instead of the last published @types/bun. These are all improvements on paper, but they change what each regression test is pinning, which is the kind of thing a maintainer should glance at.
Other factors
I verified the local registry fixtures the new tests depend on: no-deps has exactly 1.0.0/1.0.1/1.1.0/2.0.0 with dist-tags.latest = 2.0.0 (so 26657's ^1.0.0 → 1.1.0 and 24131's latest → 2.0.0 assumptions hold), basic-1@1.0.0 and a-dep exist, and packages/@types/bun exists for 24364's --typeRoots. The using dir = helper() pattern where the helper returns tempDir(...) correctly defers disposal to the caller's scope. The new test.concurrent/describe.concurrent usages all use distinct tempDir prefixes and only read from the shared registry, so there's no shared-state hazard. nodeExe() returns string | null and the test correctly falls back to bunExe(). The PR description's verification (all ten pass with npm proxied to a closed port) is credible and matches the code. No prior human reviews to consider; the bug-hunting system found nothing. Deferring only because of the breadth of assertion changes across ten regression tests, not because of any specific concern.
|
For the human review of the assertion changes, this is how each changed assertion maps to its original regression.
07740 and 00631 keep their assertions. Only the package name and the version strings changed. 07740 also asserts exit code 0 now. Updated on 2026-10-17 for the rework: patch-bounds-check, 15276 and ctrl-c are no longer part of this PR. Main already runs the first two on local registries, and ctrl-c is in #45542. |
|
Updated 12:09 PM PT - Oct 17th, 2026
✅ @robobun, your commit d6a1b4d71cc91d889cf99a58c1760b86a9d30ebc passed in 🧪 To try this PR locally: bunx bun-pr 37401That installs a local version of the PR into your bun-37401 --bun |
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/package.json`:
- Line 52: Update the form-data dependency from 4.0.0 to 4.0.6 in
test/package.json and synchronize the corresponding entry in test/bun.lock, then
rerun bun bd test test/regression/issue/26225.test.ts to verify the change.
In `@test/regression/issue/026039.test.ts`:
- Around line 9-22: Update frozenInstall to bind Bun.serve explicitly to
127.0.0.1 and construct the registry URL with 127.0.0.1 instead of localhost,
matching VerdaccioRegistry.start() and ensuring both server and client use the
same address family.
- Around line 17-22: Update the mock registry server setup to bind explicitly to
IPv4 with hostname "127.0.0.1", and construct the registry URL using
http://127.0.0.1:${server.port} instead of localhost. Leave npm_config_registry
unchanged.
In `@test/regression/issue/15276.test.ts`:
- Around line 5-7: Replace the reproduction-history paragraph above the
regression test with the standard single issue-URL comment required for
test/regression/issue files, using the issue URL for regression 15276 and
preserving the existing test body unchanged.
In `@test/regression/issue/24364.test.ts`:
- Around line 35-39: Update the typeRoots initialization to use ESM resolution
for both package paths: resolve each package.json with import.meta.resolve and
convert the resulting URLs via fileURLToPath, removing the require.resolve usage
while preserving the existing repository and test/node_modules locations.
In `@test/regression/issue/26225.test.ts`:
- Line 8: Replace the prose comment identifying issue `#26225` with the standard
GitHub issue URL comment, using the exact URL format required for regression
tests and keeping this as the single issue-reference comment.
- Around line 199-202: Update the large-file test around runClient to assert the
returned exitCode after validating stderr and parsed stdout, ensuring the child
exits successfully while preserving the existing response assertions.
In `@test/regression/issue/ctrl-c.test.ts`:
- Around line 130-148: Update the SIGINT launch-matrix test around the existing
it(...) declaration to use it.concurrent(...), allowing each independent mode in
the matrix to run concurrently while preserving the current setup and
assertions.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7e67a339-a607-4fb6-8b5f-33c240899b13
⛔ Files ignored due to path filters (1)
test/bun.lockis excluded by!**/*.lock
📒 Files selected for processing (13)
test/package.jsontest/regression/issue/00631.test.tstest/regression/issue/026039.test.tstest/regression/issue/07740.test.tstest/regression/issue/11806.test.tstest/regression/issue/15276.test.tstest/regression/issue/24131.test.tstest/regression/issue/24364.test.tstest/regression/issue/26225.test.tstest/regression/issue/26657.test.tstest/regression/issue/28193.test.tstest/regression/issue/ctrl-c.test.tstest/regression/issue/patch-bounds-check.test.ts
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it rewrites twelve separate regression tests — several with tightened or restructured assertions (26657 case 2, 24364's switch to in-tree types, 026039's request-path recording) — a human look at the assertion mapping the author already summarised in the thread would still be worthwhile.
What was reviewed
- Verdaccio fixture substitutions: confirmed
no-depshas 1.0.0/1.0.1/1.1.0/2.0.0 withlatest=2.0.0 (matches 24131/26657's version assumptions),basic-1@1.0.0's index.js is exactly the one line the patch-bounds-check hunks reference, anda-depexists. packages/@types/bunexists for 24364's--typeRoots;nodeExe()is exported from harness.- ctrl-c: the
long-running-srcfolder-name workaround is sound — the bin resolution order it protects against is real, and the fixture has no SIGINT handler so default termination behaviour is what's tested. - 026039/15276: the in-test 404 server plus recorded-path assertions encode the same property the old hostname checks did, and
await usingdisposes both server and tempDir after the spawn awaits complete.
Extended reasoning...
Overview
Test-only change across twelve files in test/regression/issue/ plus test/package.json/test/bun.lock. Each file previously contacted registry.npmjs.org (or npm.pkg.github.com) during CI; they now use VerdaccioRegistry from harness, an in-test Bun.serve 404 registry, a local file: package, or a dependency added to test/package.json. No runtime code is touched.
Security risks
None. All changes are to test files and test dependencies. form-data@4.0.0 was already present in test/bun.lock as a transitive dependency, so adding it as a direct dependency introduces nothing new to the tree.
Level of scrutiny
Medium. Test-only, so no user-facing risk, but each of the twelve files pins a real past regression and the substitution has to preserve what each test was catching. I spot-checked the fixture data the new assertions depend on (no-deps version set and dist-tags, basic-1 index.js contents, a-dep existence, packages/@types/bun) and found them to match what the tests assume. The individual transformations follow the repo's documented patterns (VerdaccioRegistry, tempDir, await using, concurrent pipe drains).
Other factors
- The author left a per-file mapping of assertion changes in the PR thread and explicitly anticipated a human look at it. Several files gained materially different assertions: 26657's second case went from vacuous to concrete; 24364 now checks against in-tree
packages/bun-typesrather than the published@types/bun; 026039 asserts on recorded request paths instead of hostname substrings in stderr; 28193 pins exact error names. These are improvements on my read, but they are judgment calls worth a maintainer's glance. - Breadth: twelve independent regression pins rewritten in one PR. Each looks correct in isolation, but the aggregate is more than I'd approve without a human confirming the coverage-preservation argument for the less mechanical ones (24364 and 26657 in particular).
- CI build #92170 was still running at review time.
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. The fixture packages referenced (no-deps 1.0.0/1.0.1/1.1.0/2.0.0 with latest=2.0.0, basic-1@1.0.0's one-line index.js, a-dep, packages/@types/bun) all check out against what the new assertions expect. Because roughly half the files rewrite assertions rather than just swapping package names — 26657 case 2 goes from vacuous to concrete, 24364 shifts to in-tree types, 026039/15276 now assert recorded request paths — a human look at those coverage changes would still be worthwhile.
Checked that the unresolved CodeRabbit items don't block: 26225 already asserts exitCode on all three tests; ctrl-c staying serial is deliberate (300ms grace comment at ctrl-c.test.ts:138); form-data 4.0.0 in a test fixture with fixed field names is not a real exposure; require.resolve in 24364 is fine in Bun tests.
Extended reasoning...
Overview
Test-only change across 12 files under test/regression/issue/ plus test/package.json/test/bun.lock. Each file previously contacted registry.npmjs.org (or npm.pkg.github.com); they now use VerdaccioRegistry from harness (7 files), an in-test 404 Bun.serve registry (026039, 15276), a local file: package (ctrl-c), test/node_modules (26225, 24364), or in-tree types (24364). No runtime code is touched.
I verified the fixture packages the new assertions depend on: test/cli/install/registry/packages/no-deps has exactly 1.0.0/1.0.1/1.1.0/2.0.0 with dist-tags.latest = 2.0.0 (so 26657's ^1.0.0 → 1.1.0, latest 2.0.0 and 24131's 1.0.0 → 2.0.0 hold); basic-1@1.0.0's index.js is the single line console.log("basic-1 1.0.0"); (matches all three patch-bounds-check hunks and the final inline snapshot); a-dep and packages/@types/bun exist.
Security risks
None. Test-only; the only dependency addition is form-data@4.0.0 to test/package.json, which was already resolved in test/bun.lock as a transitive dep. The CVE CodeRabbit flagged is CRLF injection via attacker-controlled field names — the fixture uses hard-coded "file" field names and never runs outside the test suite.
Level of scrutiny
Moderate. The mechanical half (00631, 07740, patch-bounds-check, ctrl-c, 26225's install removal) is straightforward substitution. The other half rewrites assertions: 26657 case 2 was previously vacuous and now pins concrete versions; 24131 and 28193 tightened from not.toBe/toContain("security scanner") to exact values; 026039 and 15276 replaced hostname substring checks with recorded-request-path equality; 24364 moved from published @types/bun to in-tree types with a stubbed bun-plugin-tailwind. Each is well-argued in the description and the second robobun comment, but collectively this is enough non-mechanical coverage change that a maintainer should confirm the mapping back to the original regressions.
Other factors
The author's own status comment invites a human look at the assertion changes. CodeRabbit's actionable feedback (127.0.0.1 binding, issue-URL comments) was addressed in 926d27a; the remaining open items are either already satisfied (26225 exitCode is asserted on all three tests), explicitly rebutted in-code (ctrl-c serial comment), or non-blocking style/test-fixture-CVE nits. CI build #92180 is referenced for the head commit. Given the scope — twelve files with several substantive assertion rewrites — deferring rather than auto-approving.
|
The ctrl-c.test.ts change from this PR is now on its own in #45542, because that test made CI red on 2026-10-13 (build 125085). This PR can drop that file at its next rebase. |
926d27a to
e6dacd1
Compare
packages/@types/bun is gone from the repository. The test now writes that one-line alias into its temporary directory, as a reference to packages/bun-types/index.d.ts of the checkout.
tsc reads 250 files. With a cold page cache that takes 4.7 seconds under node, which is at the default timeout of 5 seconds. Under a debug+ASAN bun the same run takes 160 seconds, so the comment now says that.
e6dacd1 to
6a9d3b1
Compare
The test imports @helpers/x and expects that bun does not find it. With auto-install on, bun asks the npm registry for that package first: 6 requests to registry.npmjs.org in each run. The error that the test asserts is the same with --no-install.
A timeout of its own caps the test below the --timeout that the CI runner passes: 90 seconds, and 270 on ASAN. Without node on PATH the test runs tsc under bun, which needs that time. 28193 lost its timeouts in this branch for the same reason.
There was a problem hiding this comment.
LGTM. Test-only change, and the latest commit drops the per-test timeout in 24364 that I flagged earlier.
What was reviewed:
- Every spawned install/add/update goes through a bunfig.toml pointing at
TestRegistry(or--no-installfor 25622);cache = falseis parsed at src/bunfig/bunfig.rs:1524-1530 and matches the harness's ownauthBunfig. 026039's in-test server records the tarball path, so its assertion no longer depends on a live host. - Fixture versions match the new exact assertions:
no-depsships 1.0.0/1.0.1/1.1.0/2.0.0 withlatest2.0.0,a-depexists, andform-data@ 4.0.1was already resolved in test/bun.lock so the lockfile delta is one line. - 24364:
typescript,@ types/reactand@ types/nodeare all in test/package.json, so the/// <reference types="node" />in packages/bun-types/index.d.ts resolves through the second typeRoot; the 28193 strings match src/install/PackageManager/security_scanner.rs:283 and install_with_manager.rs:2569-2571. - 26225 keeps a
setDefaultTimeout(30_000)that predates this PR; the tests only got faster, so I did not flag it.
Extended reasoning...
The change touches only test/regression/issue/*.test.ts plus a one-line form-data pin in test/package.json and test/bun.lock, replacing live npm installs with the TestRegistry fixture server, a local Bun.serve, test/node_modules, or --no-install. No production code, no .d.ts file in the repo, and no security-sensitive surface is touched; the CODEOWNERS entries for *.d.ts and bun-types do not cover these paths. The import pattern (from "registry") is already used by 50 other test files, the bunfig key and fixture versions were verified against source, and the only prior inline finding (the 24364 timeout) was removed in the last commit. The one author-resolved coderabbit thread sits on the form-data line and the rework's pinned version plus lockfile entry plausibly addressed it.
Problem
test/regression/issue/contactregistry.npmjs.orgin every CI run. Eight runbun install,bun addorbun updateon real package names.026039expects its requests to fail.25622imports a missing package.ctrl-c.test.tsdid in build 125085:error: GET https://registry.npmjs.org/rollup/-/rollup-4.64.6.tgz - 404(see test: replace the vite install in ctrl-c.test.ts with a local bin #45542).error: ConnectionRefused downloading package manifest is-even.Fix
00631,07740,11806,24131,26657and28193install the fixture packagesno-depsanda-depfrom aTestRegistry.026039uses oneBun.serveas the default registry and the scope registry. It answers 404, and the test asserts the tarball request path.26225requiresform-datafromtest/node_modules.24364runs thetscoftest/node_modulesagainstpackages/bun-types.25622passes--no-install.bun bd test, also in a loopback-only network namespace.Background
TestRegistry(test/packages/registry/test-registry.ts) is an npm registry onBun.serve. It serves the packages intest/cli/install/registry/packages. test: replace verdaccio with an npm registry on Bun.serve #44055 replacedVerdaccioRegistrywith it.no-depshas the versions 1.0.0, 1.0.1, 1.1.0 and 2.0.0. Thebun update -itests need a newerlatest.node-fetchalways resolves to the implementation inside bun. So26225needs onlyform-datafrom npm.node_modules, bun asks the registry for an import that it cannot resolve.--no-installturns that off.Behaviour change: none
Notes
What changed in the rework of 2026-10-17
017b2439a7, and its history is rewritten. 13 of the 15 earlier commits are replayed. Five new commits follow them.VerdaccioRegistry. The six files that used it now useTestRegistry.ctrl-c.test.tsis no longer part of this PR. test: replace the vite install in ctrl-c.test.ts with a local bin #45542 has that change.patch-bounds-check.test.tsand15276.test.tsare no longer part of this PR. Main already runs both on local registries (install: consolidate the open package manager pull requests #44968, install: fail the update request of an npm: alias whose download fails #43168). Both pass on main in the loopback-only namespace.24364: Remove the @types/bun alias, pin the root devDependencies, and add --package-manager=npm #41225 removedpackages/@types/bun, which the earlier version used as a type root. The test now writes that one-line alias into its temporary directory, as a reference topackages/bun-types/index.d.ts. The test has no timeout of its own, so the--timeoutof the CI runner applies.tscreads 250 files: 1.7 s under node, and 4.7 s with a cold page cache.bun bd testpasses no--timeout, so the default of 5 s applies there, and the first run after a build can time out. That happened in 2 of 2 such runs here. Every later run of this test passed.form-datais pinned to 4.0.1, the version thattest/bun.lockon main already resolves. The lockfile change is one line, andbun install --frozen-lockfileaccepts it.25622is new in this PR. The run of every file in the directory (below) found it.Per file
26657(update -i,Aselects all)0.1.0and^1.0.0from npmno-deps1.0.0and^1.0.0(resolves to 1.1.0). Both cases assertSelected 1 package to update, and that package.json and node_modules end on 2.0.0. In the second case the conditional assertions never ran before: is-even^1.0.0is already atlatest, soupdate -ihad nothing to show.24131(update -i,lselects)0.1.0no-deps1.0.0. Asserts the update to2.0.0.11806(bun addin a workspace member)no-depsdeclared, thenbun add --dev a-dep. Also asserts that the rewritten package.json parses, keeps the escapedtestRegex, and has the new dependency.28193(security scanner errors)no-deps. Asserts the exact linessecurity scanner failed: SecurityScannerNotInDependenciesandsecurity scanner failed: ScannerFailed, andSecurity scanner installed successfully.in the second case.07740(duplicate dependency warning)no-deps. Also asserts that the install succeeds.00631(bun addJSON escaping)bun add left-pad,^1.3.0bun add no-deps,^2.0.0. The same byte-for-byte package.json assertion.026039(empty registry field in bun.lock)/scoped/of the local server is the registry of@example, and/default/is[install].registry. The tests assert the recorded paths:/scoped/@example/test-package/-/test-package-1.0.0.tgzand/default/fake-nonexistent-package/-/fake-nonexistent-package-1.0.0.tgz.26225(node-fetch with a form-data body)bun installof form-data and node-fetchform-datais a dependency intest/package.json, and the client requires it by absolute path. The client scripts and the server checks are the same.24364(react-tailwind template type-checks)bun add -d typescript @types/bun @types/react bun-plugin-tailwind, thenbunx tsctsc(6.0.2) and@types/reactcome fromtest/node_modules.@types/bunandbun-plugin-tailwindare stubs of their published declarations, written by the test. The program has 28 files ofpackages/bun-types(--listFiles).25622(child tsconfigpathsreplace the parent's)bun runof a script that imports@helpers/x, which must not resolve. Auto-install asked the registry for it.bun --no-install run. The asserted error is the same:Cannot find module '@helpers/x'.The six
TestRegistryfiles writecache = falseinto thebunfig.tomlof each project. That moves the package cache into the project'snode_modules/.cacheand turns the manifest cache off, so tests that run at the same time share no cache.A comment in this thread maps each changed assertion to its original regression.
Main, with no route to the registries
The ten files at main
017b2439a7, debug build, in a network namespace that has only the loopback interface (unshare -n, thenip link set lo up):With
HTTPS_PROXYandHTTP_PROXYon a closed port andNO_PROXY=localhost,127.0.0.1, the result is the same 3 pass and 13 fail, withConnectionRefused. The three passes are07740and the two tests of25622. They assert only on output that does not depend on the answer of the registry.Requests on main behind a proxy that records each request and then closes it. 6 is one request and its 5 retries:
006310260390774011806241312436425622262252665728193This branch
bun bd test, network as it isexpect()callsHTTPS_PROXYandHTTP_PROXYon a closed port--parallel=4 --timeout=90000batch with one sharedBUN_INSTALL_CACHE_DIR6bb5dc1956(25 commits later): network as it is, and loopback-only namespaceChecks that the tests still detect their regressions
24131and26657: with only Enter sent (nol, noA), all three tests fail atSelected 1 package to update.24364: with thebuild.tsthat fix(init): resolve TypeScript errors in react-tailwind template build.ts #26258 replaced,tscreportsTS2532,TS7053andTS2538in the same setup. A type error appended to the currentbuild.tsfails the test withTS2322.25622: with the path of the parent added to the child tsconfig, the import resolves and the first test fails.The other files in the directory
Each of the 428 test files in
test/regression/issue/ran alone behind the recording proxy, with the release build of 2026-09-18. The 16 files that failed on that build and are not in the list below ran again with the debug build of main. They sent no request to an outside host. Other than the ten files of this PR, three files send a request to an outside host:ctrl-c.test.ts: test: replace the vite install in ctrl-c.test.ts with a local bin #45542.malformed-integrity-base64.test.ts: test: make malformed-integrity-base64.test.ts hermetic #37375.30205.test.ts:bun installof node-gyp for an addon build, astest/napidoes. It is not changed here.The proxy sees HTTP and HTTPS requests only. This list does not cover other traffic, for example DNS.
026039,07740,11806and26657are ontest/parallel-denylist.txt(they failed inside the parallel batch and passed alone). This PR does not change that list.