Skip to content

test: stop ten regression tests from contacting npm - #37401

Open
robobun wants to merge 18 commits into
mainfrom
farm/71320a9a/hermetic-regression-install-tests
Open

robobun wants to merge 18 commits into
mainfrom
farm/71320a9a/hermetic-regression-install-tests

Conversation

@robobun

@robobun robobun commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Ten files in test/regression/issue/ contact registry.npmjs.org in every CI run. Eight run bun install, bun add or bun update on real package names. 026039 expects its requests to fail. 25622 imports a missing package.
  • These tests fail when npm fails, as ctrl-c.test.ts did in build 125085: error: GET https://registry.npmjs.org/rollup/-/rollup-4.64.6.tgz - 404 (see test: replace the vite install in ctrl-c.test.ts with a local bin #45542).
  • With no route to the registries, main fails 13 of the 16 tests: error: ConnectionRefused downloading package manifest is-even.

Fix

  • 00631, 07740, 11806, 24131, 26657 and 28193 install the fixture packages no-deps and a-dep from a TestRegistry.
  • 026039 uses one Bun.serve as the default registry and the scope registry. It answers 404, and the test asserts the tarball request path.
  • 26225 requires form-data from test/node_modules. 24364 runs the tsc of test/node_modules against packages/bun-types. 25622 passes --no-install.
  • Verified: the ten files pass with bun bd test, also in a loopback-only network namespace.

Background

  • TestRegistry (test/packages/registry/test-registry.ts) is an npm registry on Bun.serve. It serves the packages in test/cli/install/registry/packages. test: replace verdaccio with an npm registry on Bun.serve #44055 replaced VerdaccioRegistry with it.
  • no-deps has the versions 1.0.0, 1.0.1, 1.1.0 and 2.0.0. The bun update -i tests need a newer latest.
  • node-fetch always resolves to the implementation inside bun. So 26225 needs only form-data from npm.
  • Auto-install: with no node_modules, bun asks the registry for an import that it cannot resolve. --no-install turns that off.

Behaviour change: none

Notes

What changed in the rework of 2026-10-17

Per file

file before after
26657 (update -i, A selects all) is-even 0.1.0 and ^1.0.0 from npm no-deps 1.0.0 and ^1.0.0 (resolves to 1.1.0). Both cases assert Selected 1 package to update, and that package.json and node_modules end on 2.0.0. In the second case the conditional assertions never ran before: is-even ^1.0.0 is already at latest, so update -i had nothing to show.
24131 (update -i, l selects) is-even 0.1.0 no-deps 1.0.0. Asserts the update to 2.0.0.
11806 (bun add in a workspace member) typescript no-deps declared, then bun add --dev a-dep. Also asserts that the rewritten package.json parses, keeps the escaped testRegex, and has the new dependency.
28193 (security scanner errors) is-even no-deps. Asserts the exact lines security scanner failed: SecurityScannerNotInDependencies and security scanner failed: ScannerFailed, and Security scanner installed successfully. in the second case.
07740 (duplicate dependency warning) empty-package-for-bun-test-runner no-deps. Also asserts that the install succeeds.
00631 (bun add JSON escaping) bun add left-pad, ^1.3.0 bun add no-deps, ^2.0.0. The same byte-for-byte package.json assertion.
026039 (empty registry field in bun.lock) requests to npm.pkg.github.com and registry.npmjs.org /scoped/ of the local server is the registry of @example, and /default/ is [install].registry. The tests assert the recorded paths: /scoped/@example/test-package/-/test-package-1.0.0.tgz and /default/fake-nonexistent-package/-/fake-nonexistent-package-1.0.0.tgz.
26225 (node-fetch with a form-data body) bun install of form-data and node-fetch form-data is a dependency in test/package.json, and the client requires it by absolute path. The client scripts and the server checks are the same.
24364 (react-tailwind template type-checks) bun add -d typescript @types/bun @types/react bun-plugin-tailwind, then bunx tsc tsc (6.0.2) and @types/react come from test/node_modules. @types/bun and bun-plugin-tailwind are stubs of their published declarations, written by the test. The program has 28 files of packages/bun-types (--listFiles).
25622 (child tsconfig paths replace the parent's) bun run of a script that imports @helpers/x, which must not resolve. Auto-install asked the registry for it. bun --no-install run. The asserted error is the same: Cannot find module '@helpers/x'.

The six TestRegistry files write cache = false into the bunfig.toml of each project. That moves the package cache into the project's node_modules/.cache and turns the manifest cache off, so tests that run at the same time share no cache.

A comment in this thread maps each changed assertion to its original regression.

Main, with no route to the registries

The ten files at main 017b2439a7, debug build, in a network namespace that has only the loopback interface (unshare -n, then ip link set lo up):

error: DNSResolveFailed downloading package manifest is-even
error: is-even@0.1.0 failed to resolve
(fail) bun update -i select all with 'A' key > should update packages when 'A' is pressed to select all
...
 3 pass
 13 fail
Ran 16 tests across 10 files.

With HTTPS_PROXY and HTTP_PROXY on a closed port and NO_PROXY=localhost,127.0.0.1, the result is the same 3 pass and 13 fail, with ConnectionRefused. The three passes are 07740 and the two tests of 25622. They assert only on output that does not depend on the answer of the registry.

Requests on main behind a proxy that records each request and then closes it. 6 is one request and its 5 retries:

file registry.npmjs.org npm.pkg.github.com
00631 6
026039 6 6
07740 6
11806 6
24131 6
24364 24
25622 6
26225 30
26657 12
28193 12

This branch

how the ten files ran result
bun bd test, network as it is 16 pass, 65 expect() calls
HTTPS_PROXY and HTTP_PROXY on a closed port 16 pass
the recording proxy 16 pass, no request from the tests
loopback-only network namespace 16 pass, 6 runs of 6
the same as one --parallel=4 --timeout=90000 batch with one shared BUN_INSTALL_CACHE_DIR 16 pass, 3 runs of 3. The shared cache directory stays empty.
release build of 2026-09-18 (1.4.3-canary.1), network as it is 16 pass in 2.5 s
2026-10-19, this branch merged with main 6bb5dc1956 (25 commits later): network as it is, and loopback-only namespace 16 pass, 2 runs of 2 and 3 runs of 3

Checks that the tests still detect their regressions

  • 24131 and 26657: with only Enter sent (no l, no A), all three tests fail at Selected 1 package to update.
  • 24364: with the build.ts that fix(init): resolve TypeScript errors in react-tailwind template build.ts #26258 replaced, tsc reports TS2532, TS7053 and TS2538 in the same setup. A type error appended to the current build.ts fails the test with TS2322.
  • 25622: with the path of the parent added to the child tsconfig, the import resolves and the first test fails.

The other files in the directory

Each of the 428 test files in test/regression/issue/ ran alone behind the recording proxy, with the release build of 2026-09-18. The 16 files that failed on that build and are not in the list below ran again with the debug build of main. They sent no request to an outside host. Other than the ten files of this PR, three files send a request to an outside host:

The proxy sees HTTP and HTTPS requests only. This list does not cover other traffic, for example DNS.

026039, 07740, 11806 and 26657 are on test/parallel-denylist.txt (they failed inside the parallel batch and passed alone). This PR does not change that list.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Regression tests now use local Verdaccio registries, shared asynchronous helpers, concurrent execution, repository-local fixtures, and explicit subprocess assertions. External package installation and manual cleanup were removed.

Changes

Hermetic regression coverage

Layer / File(s) Summary
Registry-backed install workflows
test/regression/issue/00631.test.ts, test/regression/issue/07740.test.ts, test/regression/issue/11806.test.ts, test/regression/issue/28193.test.ts
Tests use local registries, lifecycle hooks, asynchronous installs, shared helpers, and explicit manifest, output, and exit-code assertions.
Package resolution and update scenarios
test/regression/issue/026039.test.ts, test/regression/issue/15276.test.ts, test/regression/issue/24131.test.ts, test/regression/issue/26657.test.ts, test/regression/issue/patch-bounds-check.test.ts
Tests use deterministic local packages and verify registry requests, alias resolution, interactive updates, dependency ranges, installed versions, and patch behavior.
Local execution fixtures
test/regression/issue/24364.test.ts, test/regression/issue/ctrl-c.test.ts
Compiler and SIGINT tests use repository-local templates and package fixtures instead of installing external dependencies.
Shared subprocess dependency harness
test/package.json, test/regression/issue/26225.test.ts
The multipart tests resolve form-data from the test environment, share subprocess execution, run concurrently, and validate parsed output, stderr, and exit status.

Possibly related PRs

  • oven-sh/bun#36194: Both changes update registry-backed setup in 026039.test.ts and 07740.test.ts.
  • oven-sh/bun#36424: Both changes overlap in asynchronous test helpers and 11806.test.ts.
  • oven-sh/bun#36988: Both changes use the shared VerdaccioRegistry and writeBunfig test infrastructure.

Suggested reviewers: jarred-sumner

Pre-merge checks: Passed 3 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Out of Scope Changes check Inconclusive The changes are broadly within the stated test-hermeticity objective, but the supplied summaries and description disagree about whether ctrl-c.test.ts, patch-bounds-check.test.ts, and 15276.test.ts ar… Reconcile the changed-file list, objectives, and description. State clearly whether ctrl-c.test.ts, patch-bounds-check.test.ts, and 15276.test.ts are part of this pull request.
Passed Passed checks: 3 passed
Check name Status Explanation
Description check Passed The description explains the problem, fix, scope, per-file changes, verification, and regression coverage. It does not use the exact template headings, but it provides the required information in equi…
Linked Issues check Passed The description references relevant issue numbers and maps several test changes to their corresponding regressions.
Title check Passed The title clearly identifies the main change: preventing regression tests from contacting npm. The stated count of ten conflicts with the supplied summaries, which list twelve regression test files, b…
Full details: Description check

Explanation

The description explains the problem, fix, scope, per-file changes, verification, and regression coverage. It does not use the exact template headings, but it provides the required information in equivalent sections.

Full details: Title check

Explanation

The title clearly identifies the main change: preventing regression tests from contacting npm. The stated count of ten conflicts with the supplied summaries, which list twelve regression test files, but the primary scope remains clear.

Full details: Out of Scope Changes check

Explanation

The changes are broadly within the stated test-hermeticity objective, but the supplied summaries and description disagree about whether ctrl-c.test.ts, patch-bounds-check.test.ts, and 15276.test.ts are included.

Resolution

Reconcile the changed-file list, objectives, and description. State clearly whether ctrl-c.test.ts, patch-bounds-check.test.ts, and 15276.test.ts are part of this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. CI is green on d6a1b4d71c (build 127557). The branch is based on main 017b2439a7 and merges cleanly.

On 2026-10-19 the ten files ran again on this branch merged with main 6bb5dc1956, which is 25 commits later and has more install changes. The 16 tests pass with bun bd test, also with no network.

How the network dependency reproduces: run the ten files with no route to the registries. Two setups show it.

  • A network namespace that has only the loopback interface (unshare -n, then ip link set lo up).
  • HTTPS_PROXY and HTTP_PROXY on a closed port, with NO_PROXY=localhost,127.0.0.1.

In both, main fails 13 of the 16 tests, for example error: ConnectionRefused downloading package manifest is-even. Behind a proxy that records each request, the ten files on main send 114 requests to registry.npmjs.org and 6 to npm.pkg.github.com.

On this branch the 16 tests pass in both setups with bun bd test, and the recording proxy gets no request from them.

This PR no longer changes ctrl-c.test.ts (see #45542), patch-bounds-check.test.ts or 15276.test.ts. Main already runs the last two on local registries. 25622.test.ts is new in this PR: a run of every file in the directory behind the recording proxy found it.

PR: #37401

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this and didn't find any bugs. It's a test-only change that follows the harness patterns (VerdaccioRegistry, tempDir, file: deps), and I verified the referenced fixture packages (no-deps 1.0.0/1.0.1/1.1.0/2.0.0 with latest=2.0.0, basic-1, a-dep) and packages/@types/bun all exist with the expected shape. Given the scope — ten regression test files rewritten, with strengthened/changed assertions in 26657, 28193, 11806, and a semantic shift in what 24364 validates (in-tree types instead of published @types/bun) — a human look to confirm the assertion changes preserve each regression's original intent would still be worthwhile.

What was reviewed:

  • Fixture availability in test/cli/install/registry/packages/ matches every version the new tests depend on.
  • using dir = project(...) pattern in 26657 / patch-bounds-check correctly disposes at the call site; concurrent tests use distinct tempDir prefixes and read-only registry access.
  • nodeExe() ?? bunExe() in 24364 matches the harness signature; the ctrl-c file: bin still creates node_modules/.bin for all six launch variants.
Extended reasoning...

Overview

This PR converts ten test/regression/issue/ files that previously ran bun install/add/update against registry.npmjs.org into hermetic tests. Seven files now start a VerdaccioRegistry and point bunfig.toml at it, swapping real npm packages (is-even, left-pad, lodash, typescript) for local-registry fixtures (no-deps, basic-1, a-dep). Three files use different approaches: ctrl-c.test.ts replaces the vite install with a local file: package that has a bin; 26225.test.ts drops the install entirely and requires form-data from test/node_modules by absolute path (added to test/package.json); 24364.test.ts runs tsc from test/node_modules against in-repo packages/@types/bun with a stubbed bun-plugin-tailwind declaration instead of installing four packages from npm. Only test files, test/package.json, and one line of test/bun.lock are touched.

Security risks

None. Test-only change with no runtime code modified. The change reduces external network dependencies rather than adding any.

Level of scrutiny

Medium. This is test infrastructure, not shipped code, and each individual conversion follows the pattern REVIEW.md and CLAUDE.md prescribe (VerdaccioRegistry from harness, tempDir, no live registries). However, the aggregate is large (~500 changed lines across 12 files), and several tests had their assertions rewritten rather than just their package names swapped: 26657's second case went from a vacuous conditional assertion to concrete 2.0.0 checks; 28193 tightened toContain("security scanner") to the specific error names; 11806 added post-rewrite package.json parsing assertions; 24364 now type-checks against in-tree packages/bun-types instead of the last published @types/bun. These are all improvements on paper, but they change what each regression test is pinning, which is the kind of thing a maintainer should glance at.

Other factors

I verified the local registry fixtures the new tests depend on: no-deps has exactly 1.0.0/1.0.1/1.1.0/2.0.0 with dist-tags.latest = 2.0.0 (so 26657's ^1.0.0 → 1.1.0 and 24131's latest → 2.0.0 assumptions hold), basic-1@1.0.0 and a-dep exist, and packages/@types/bun exists for 24364's --typeRoots. The using dir = helper() pattern where the helper returns tempDir(...) correctly defers disposal to the caller's scope. The new test.concurrent/describe.concurrent usages all use distinct tempDir prefixes and only read from the shared registry, so there's no shared-state hazard. nodeExe() returns string | null and the test correctly falls back to bunExe(). The PR description's verification (all ten pass with npm proxied to a closed port) is credible and matches the code. No prior human reviews to consider; the bug-hunting system found nothing. Deferring only because of the breadth of assertion changes across ten regression tests, not because of any specific concern.

@robobun

robobun commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

For the human review of the assertion changes, this is how each changed assertion maps to its original regression.

  • 26657: The first case asserted !== "0.1.0" on package.json and > 0.1.0 on node_modules after A on an exact pin. It now asserts the concrete value, 2.0.0 in both places, for the same scenario. That is the branch that the A handler gained in fix(cli): bun update -i select all ('A') now updates packages correctly #26658: a package with current == update goes to latest. In the second case the assertions were conditional and never ran: is-even ^1.0.0 is already at latest, so update -i had no package to show. With no-deps ^1.0.0 (resolves to 1.1.0, latest is 2.0.0) the test now covers the situation that its own comment describes.
  • 24131: The shape is the same as before (the version must change, exit code 0), with the new value pinned. With only Enter sent, the test fails at Selected 1 package to update. So the assertion depends on the l key.
  • 28193: toContain("security scanner") matched the line error: security scanner failed: <name> that fix(install): print error message when security scanner fails #28196 added. The original is-even setup prints SecurityScannerNotInDependencies in the first test and ScannerFailed in the second. The no-deps setup prints the same two names. So the tests match the same line more exactly. The check for Security scanner installed successfully. shows that the second test still installs the scanner before it fails.
  • 11806: Both exit code assertions are still there. The added checks are the property that the issue reported: the rewritten package.json parses, jest.testRegex is unchanged, and the new dependency is present. The issue added a new dependency, and the test now does that too (a-dep). Before, it added typescript a second time.
  • 24364: The move to the types of the checkout is deliberate. The issue was a build.ts that did not type-check. The template ships in the same binary as packages/bun-types, so a check against the checkout finds a break before the next @types/bun release. test/cli/init/init.test.ts still type-checks the templates with the packages that bun init installs. With the build.ts from before the fix (fix(init): resolve TypeScript errors in react-tailwind template build.ts #26258), tsc reports TS2532, TS7053 and TS2538 in the new setup.
  • 25622: The assertions are unchanged. The spawn now passes --no-install, and the error that the test asserts is the same with it. With the path of the parent added to the child tsconfig, the import resolves and the first test fails.

07740 and 00631 keep their assertions. Only the package name and the version strings changed. 07740 also asserts exit code 0 now.

Updated on 2026-10-17 for the rework: patch-bounds-check, 15276 and ctrl-c are no longer part of this PR. Main already runs the first two on local registries, and ctrl-c is in #45542.

@robobun

robobun commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:09 PM PT - Oct 17th, 2026

✅ @robobun, your commit d6a1b4d71cc91d889cf99a58c1760b86a9d30ebc passed in Build #127557! 🎉


🧪   To try this PR locally:

bunx bun-pr 37401

That installs a local version of the PR into your bun-37401 executable, so you can run:

bun-37401 --bun

@robobun robobun changed the title test: stop the install-based regression tests in test/regression/issue from hitting npm test: stop the regression tests in test/regression/issue from contacting npm Aug 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/package.json`:
- Line 52: Update the form-data dependency from 4.0.0 to 4.0.6 in
test/package.json and synchronize the corresponding entry in test/bun.lock, then
rerun bun bd test test/regression/issue/26225.test.ts to verify the change.

In `@test/regression/issue/026039.test.ts`:
- Around line 9-22: Update frozenInstall to bind Bun.serve explicitly to
127.0.0.1 and construct the registry URL with 127.0.0.1 instead of localhost,
matching VerdaccioRegistry.start() and ensuring both server and client use the
same address family.
- Around line 17-22: Update the mock registry server setup to bind explicitly to
IPv4 with hostname "127.0.0.1", and construct the registry URL using
http://127.0.0.1:${server.port} instead of localhost. Leave npm_config_registry
unchanged.

In `@test/regression/issue/15276.test.ts`:
- Around line 5-7: Replace the reproduction-history paragraph above the
regression test with the standard single issue-URL comment required for
test/regression/issue files, using the issue URL for regression 15276 and
preserving the existing test body unchanged.

In `@test/regression/issue/24364.test.ts`:
- Around line 35-39: Update the typeRoots initialization to use ESM resolution
for both package paths: resolve each package.json with import.meta.resolve and
convert the resulting URLs via fileURLToPath, removing the require.resolve usage
while preserving the existing repository and test/node_modules locations.

In `@test/regression/issue/26225.test.ts`:
- Line 8: Replace the prose comment identifying issue `#26225` with the standard
GitHub issue URL comment, using the exact URL format required for regression
tests and keeping this as the single issue-reference comment.
- Around line 199-202: Update the large-file test around runClient to assert the
returned exitCode after validating stderr and parsed stdout, ensuring the child
exits successfully while preserving the existing response assertions.

In `@test/regression/issue/ctrl-c.test.ts`:
- Around line 130-148: Update the SIGINT launch-matrix test around the existing
it(...) declaration to use it.concurrent(...), allowing each independent mode in
the matrix to run concurrently while preserving the current setup and
assertions.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7e67a339-a607-4fb6-8b5f-33c240899b13

📥 Commits

Reviewing files that changed from the base of the PR and between 9fcdea8 and b5b22f9.

⛔ Files ignored due to path filters (1)
  • test/bun.lock is excluded by !**/*.lock
📒 Files selected for processing (13)
  • test/package.json
  • test/regression/issue/00631.test.ts
  • test/regression/issue/026039.test.ts
  • test/regression/issue/07740.test.ts
  • test/regression/issue/11806.test.ts
  • test/regression/issue/15276.test.ts
  • test/regression/issue/24131.test.ts
  • test/regression/issue/24364.test.ts
  • test/regression/issue/26225.test.ts
  • test/regression/issue/26657.test.ts
  • test/regression/issue/28193.test.ts
  • test/regression/issue/ctrl-c.test.ts
  • test/regression/issue/patch-bounds-check.test.ts

Comment thread test/package.json Resolved
Comment thread test/regression/issue/026039.test.ts Resolved
Comment thread test/regression/issue/026039.test.ts Resolved
Comment thread test/regression/issue/15276.test.ts Resolved
Comment thread test/regression/issue/24364.test.ts Resolved
Comment thread test/regression/issue/26225.test.ts Resolved
Comment thread test/regression/issue/26225.test.ts Resolved
Comment thread test/regression/issue/ctrl-c.test.ts Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it rewrites twelve separate regression tests — several with tightened or restructured assertions (26657 case 2, 24364's switch to in-tree types, 026039's request-path recording) — a human look at the assertion mapping the author already summarised in the thread would still be worthwhile.

What was reviewed

  • Verdaccio fixture substitutions: confirmed no-deps has 1.0.0/1.0.1/1.1.0/2.0.0 with latest=2.0.0 (matches 24131/26657's version assumptions), basic-1@1.0.0's index.js is exactly the one line the patch-bounds-check hunks reference, and a-dep exists.
  • packages/@types/bun exists for 24364's --typeRoots; nodeExe() is exported from harness.
  • ctrl-c: the long-running-src folder-name workaround is sound — the bin resolution order it protects against is real, and the fixture has no SIGINT handler so default termination behaviour is what's tested.
  • 026039/15276: the in-test 404 server plus recorded-path assertions encode the same property the old hostname checks did, and await using disposes both server and tempDir after the spawn awaits complete.
Extended reasoning...

Overview

Test-only change across twelve files in test/regression/issue/ plus test/package.json/test/bun.lock. Each file previously contacted registry.npmjs.org (or npm.pkg.github.com) during CI; they now use VerdaccioRegistry from harness, an in-test Bun.serve 404 registry, a local file: package, or a dependency added to test/package.json. No runtime code is touched.

Security risks

None. All changes are to test files and test dependencies. form-data@4.0.0 was already present in test/bun.lock as a transitive dependency, so adding it as a direct dependency introduces nothing new to the tree.

Level of scrutiny

Medium. Test-only, so no user-facing risk, but each of the twelve files pins a real past regression and the substitution has to preserve what each test was catching. I spot-checked the fixture data the new assertions depend on (no-deps version set and dist-tags, basic-1 index.js contents, a-dep existence, packages/@types/bun) and found them to match what the tests assume. The individual transformations follow the repo's documented patterns (VerdaccioRegistry, tempDir, await using, concurrent pipe drains).

Other factors

  • The author left a per-file mapping of assertion changes in the PR thread and explicitly anticipated a human look at it. Several files gained materially different assertions: 26657's second case went from vacuous to concrete; 24364 now checks against in-tree packages/bun-types rather than the published @types/bun; 026039 asserts on recorded request paths instead of hostname substrings in stderr; 28193 pins exact error names. These are improvements on my read, but they are judgment calls worth a maintainer's glance.
  • Breadth: twelve independent regression pins rewritten in one PR. Each looks correct in isolation, but the aggregate is more than I'd approve without a human confirming the coverage-preservation argument for the less mechanical ones (24364 and 26657 in particular).
  • CI build #92170 was still running at review time.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. The fixture packages referenced (no-deps 1.0.0/1.0.1/1.1.0/2.0.0 with latest=2.0.0, basic-1@1.0.0's one-line index.js, a-dep, packages/@types/bun) all check out against what the new assertions expect. Because roughly half the files rewrite assertions rather than just swapping package names — 26657 case 2 goes from vacuous to concrete, 24364 shifts to in-tree types, 026039/15276 now assert recorded request paths — a human look at those coverage changes would still be worthwhile.

Checked that the unresolved CodeRabbit items don't block: 26225 already asserts exitCode on all three tests; ctrl-c staying serial is deliberate (300ms grace comment at ctrl-c.test.ts:138); form-data 4.0.0 in a test fixture with fixed field names is not a real exposure; require.resolve in 24364 is fine in Bun tests.

Extended reasoning...

Overview

Test-only change across 12 files under test/regression/issue/ plus test/package.json/test/bun.lock. Each file previously contacted registry.npmjs.org (or npm.pkg.github.com); they now use VerdaccioRegistry from harness (7 files), an in-test 404 Bun.serve registry (026039, 15276), a local file: package (ctrl-c), test/node_modules (26225, 24364), or in-tree types (24364). No runtime code is touched.

I verified the fixture packages the new assertions depend on: test/cli/install/registry/packages/no-deps has exactly 1.0.0/1.0.1/1.1.0/2.0.0 with dist-tags.latest = 2.0.0 (so 26657's ^1.0.0 → 1.1.0, latest 2.0.0 and 24131's 1.0.0 → 2.0.0 hold); basic-1@1.0.0's index.js is the single line console.log("basic-1 1.0.0"); (matches all three patch-bounds-check hunks and the final inline snapshot); a-dep and packages/@types/bun exist.

Security risks

None. Test-only; the only dependency addition is form-data@4.0.0 to test/package.json, which was already resolved in test/bun.lock as a transitive dep. The CVE CodeRabbit flagged is CRLF injection via attacker-controlled field names — the fixture uses hard-coded "file" field names and never runs outside the test suite.

Level of scrutiny

Moderate. The mechanical half (00631, 07740, patch-bounds-check, ctrl-c, 26225's install removal) is straightforward substitution. The other half rewrites assertions: 26657 case 2 was previously vacuous and now pins concrete versions; 24131 and 28193 tightened from not.toBe/toContain("security scanner") to exact values; 026039 and 15276 replaced hostname substring checks with recorded-request-path equality; 24364 moved from published @types/bun to in-tree types with a stubbed bun-plugin-tailwind. Each is well-argued in the description and the second robobun comment, but collectively this is enough non-mechanical coverage change that a maintainer should confirm the mapping back to the original regressions.

Other factors

The author's own status comment invites a human look at the assertion changes. CodeRabbit's actionable feedback (127.0.0.1 binding, issue-URL comments) was addressed in 926d27a; the remaining open items are either already satisfied (26225 exitCode is asserted on all three tests), explicitly rebutted in-code (ctrl-c serial comment), or non-blocking style/test-fixture-CVE nits. CI build #92180 is referenced for the head commit. Given the scope — twelve files with several substantive assertion rewrites — deferring rather than auto-approving.

@robobun

robobun commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

The ctrl-c.test.ts change from this PR is now on its own in #45542, because that test made CI red on 2026-10-13 (build 125085). This PR can drop that file at its next rebase.

@robobun
robobun force-pushed the farm/71320a9a/hermetic-regression-install-tests branch from 926d27a to e6dacd1 Compare October 17, 2026 13:01
packages/@types/bun is gone from the repository. The test now writes that
one-line alias into its temporary directory, as a reference to
packages/bun-types/index.d.ts of the checkout.
tsc reads 250 files. With a cold page cache that takes 4.7 seconds under
node, which is at the default timeout of 5 seconds. Under a debug+ASAN bun
the same run takes 160 seconds, so the comment now says that.
@robobun robobun changed the title test: stop the regression tests in test/regression/issue from contacting npm test: stop nine regression tests from contacting npm Oct 17, 2026
@robobun
robobun force-pushed the farm/71320a9a/hermetic-regression-install-tests branch from e6dacd1 to 6a9d3b1 Compare October 17, 2026 13:02
The test imports @helpers/x and expects that bun does not find it. With
auto-install on, bun asks the npm registry for that package first: 6
requests to registry.npmjs.org in each run. The error that the test asserts
is the same with --no-install.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread test/regression/issue/24364.test.ts Resolved
@robobun robobun changed the title test: stop nine regression tests from contacting npm test: stop ten regression tests from contacting npm Oct 17, 2026
A timeout of its own caps the test below the --timeout that the CI runner
passes: 90 seconds, and 270 on ASAN. Without node on PATH the test runs tsc
under bun, which needs that time. 28193 lost its timeouts in this branch for
the same reason.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Test-only change, and the latest commit drops the per-test timeout in 24364 that I flagged earlier.

What was reviewed:

  • Every spawned install/add/update goes through a bunfig.toml pointing at TestRegistry (or --no-install for 25622); cache = false is parsed at src/bunfig/bunfig.rs:1524-1530 and matches the harness's own authBunfig. 026039's in-test server records the tarball path, so its assertion no longer depends on a live host.
  • Fixture versions match the new exact assertions: no-deps ships 1.0.0/1.0.1/1.1.0/2.0.0 with latest 2.0.0, a-dep exists, and form-data@ 4.0.1 was already resolved in test/bun.lock so the lockfile delta is one line.
  • 24364: typescript, @ types/react and @ types/node are all in test/package.json, so the /// <reference types="node" /> in packages/bun-types/index.d.ts resolves through the second typeRoot; the 28193 strings match src/install/PackageManager/security_scanner.rs:283 and install_with_manager.rs:2569-2571.
  • 26225 keeps a setDefaultTimeout(30_000) that predates this PR; the tests only got faster, so I did not flag it.
Extended reasoning...

The change touches only test/regression/issue/*.test.ts plus a one-line form-data pin in test/package.json and test/bun.lock, replacing live npm installs with the TestRegistry fixture server, a local Bun.serve, test/node_modules, or --no-install. No production code, no .d.ts file in the repo, and no security-sensitive surface is touched; the CODEOWNERS entries for *.d.ts and bun-types do not cover these paths. The import pattern (from "registry") is already used by 50 other test files, the bunfig key and fixture versions were verified against source, and the only prior inline finding (the 24364 timeout) was removed in the last commit. The one author-resolved coderabbit thread sits on the form-data line and the rework's pinned version plus lockfile entry plausibly addressed it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant