Skip to content

test(sql): share one mock server per file in postgres fault-injection tests - #33962

Merged
dylan-conway merged 3 commits into
mainfrom
farm/d2652c68/sql-mock-server-shared
Jul 11, 2026
Merged

dylan-conway merged 3 commits into
mainfrom
farm/d2652c68/sql-mock-server-shared

Conversation

@robobun

@robobun robobun commented Jul 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

postgres-binary-array-bounds.test.ts and postgres-invalid-message-length.test.ts have been flaking on Windows CI lanes (2019 x64, 2019 x64-baseline, 11 aarch64) with ERR_POSTGRES_CONNECTION_REFUSED. The first 1-2 cases in the file pass, then every remaining case fails, and the failure persists across every retry of the test process for ~30-50s before recovering. Seen in builds 70081 onward (e.g. 71781, 71702, 71632, 71614, 71607, 71604, 71587, 71582, 71563).

error: expect(received).toMatch(expected)
Expected substring or pattern: /ERR_POSTGRES_INVALID_BINARY_DATA|InvalidBinaryData/
Received: "ERR_POSTGRES_CONNECTION_REFUSED"

Cause

Both files create a fresh net.Server per test case (7 and 10 respectively) and close it in finally. Since #33622 switched the runner to LPT sharding, on Windows both files land in the same shard a few positions after node-http-uaf.test.ts, whose fixture issues ~22000 localhost connections and leaves tens of thousands of TIME_WAIT sockets (44k measured locally).

Bun disables SYN retransmission for loopback connects on Windows (SIO_TCP_INITIAL_RTO with TCP_INITIAL_RTO_NO_SYN_RETRANSMISSIONS in packages/bun-usockets/src/bsd.c), so a lost SYN under that load becomes an immediate connect failure, and the SQL client surfaces it as ERR_POSTGRES_CONNECTION_REFUSED without retry (by design: "nothing is listening" fails fast).

I could not reproduce the dropped SYN locally on an idle Windows Server 2019 box even with 44k TIME_WAIT sockets in place, but the CI pattern (Windows-only, first-N-then-all-fail, cross-process, ~30-50s recovery, exclusively in the shard that runs after the 22k-connection fixture) is consistent with this mechanism and nothing else in the process carries state across a fresh bun test invocation.

Fix

Start one long-lived mock listener per file at module scope and latch the per-test payload into a module-level variable that the connection handler reads at accept time. Each file now needs one listen port and a handful of client ports for the whole run, removing the per-test listen()+close() churn. Assertions are unchanged; the combined run also drops from ~395ms to ~150ms on Windows release.

Verification

bun bd test test/js/sql/postgres-binary-array-bounds.test.ts test/js/sql/postgres-invalid-message-length.test.ts
  17 pass  0 fail

On Windows, 7 consecutive runs immediately after node-http-uaf.test.ts (with thousands of TIME_WAIT sockets in place) all pass.

Refs #33622.


[stamp-90s] gate passed · iteration 1 · 2 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/sql/postgres-binary-array-bounds.test.ts' 'test/js/sql/postgres-invalid-message-length.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/sql/postgres-binary-array-bounds.test.ts test/js/sql/postgres-invalid-message-length.test.ts
info: syncing channel updates for nightly-2026-05-06-x86_64-unknown-linux-gnu
info: latest update on 2026-05-06 for version 1.97.0-nightly (e95e73209 2026-05-05)
info: component rust-src is up to date
info: checking for self-update (current version: 1.29.0)
bun test v1.4.0 (8c1de5de2)

test/js/sql/postgres-binary-array-bounds.test.ts:
(pass) binary int4[] with len exceeding column bytes is rejected [454.64ms]
(pass) binary int4[] with len exceeding column bytes (partial data) is rejected [133.45ms]
(pass) binary int4[] with negative len is rejected [63.86ms]
(pass) binary int4[] with ndim=1 but truncated header is rejected [67.45ms]
(pass) binary int4[] with len = INT32_MAX is rejected [111.00ms]
(pass) binary float4[] with len exceeding column bytes is rejected [95.72ms]
(pass) well-formed binary int4[] still parses [66.09ms]

test/js/sql/postgres-invalid-message-length.test.ts:
(pass) postgres: ReadyForQuery declaring length 3 is a protocol error [70.61ms]
(pass) postgres: ReadyForQuery declaring length 0 is a protocol error [31.84ms]
(pass) postgres: ReadyForQuery declaring length -1 is a protocol error [33.82ms]
(pass) postgres: ParameterStatus declaring length 3 is a protocol error [38.84ms]
(pass) postgres: NotificationResponse declaring length 3 is a protocol error [42.64ms]
(pass) postgres: NoticeResponse declaring length 3 is a protocol error [51.42ms]
(pass) postgres: ErrorResponse declaring length 3 is a protocol error [53.33ms]
(pass) postgres: unknown message type declaring length 3 is a protocol error [53.54ms]
(pass) postgres: CommandComplete declaring length 3 fails the in-flight query [82.24ms]
(pass) postgres: an empty NoticeResponse (length exactly 4) is accepted [58.24ms]

 17 pass
 0 fail
 23 expect() calls
Ran 17 tests across 2 files. [4.22s]
Exit: 0
diff hotspot
test/js/sql/postgres-binary-array-bounds.test.ts   | 58 ++++++++++++----------
 .../js/sql/postgres-invalid-message-length.test.ts | 50 +++++++++----------
 2 files changed, 55 insertions(+), 53 deletions(-)

gate history · 3 passed · 0 rejected · iteration 1

evidence per changed file
file                                                 reads  edits  tests
test/js/sql/postgres-binary-array-bounds.test.ts         2      2      0
test/js/sql/postgres-invalid-message-length.test.ts      2      2      0

… tests

postgres-binary-array-bounds.test.ts and postgres-invalid-message-length.test.ts
were creating a fresh net.Server per test case (7 and 10 respectively) and
closing it in finally. On Windows CI lanes both files started intermittently
failing with ERR_POSTGRES_CONNECTION_REFUSED from the 2nd or 3rd case on,
across every retry of the test process, then recovering after ~30-50s.

In the failing shards both files run a few positions after
node-http-uaf.test.ts, whose fixture issues ~22000 localhost connections and
leaves tens of thousands of TIME_WAIT sockets. bun disables SYN
retransmission for loopback connects on Windows (SIO_TCP_INITIAL_RTO in
packages/bun-usockets/src/bsd.c), so a lost SYN in that state becomes an
immediate connect failure, which the SQL client surfaces as
ERR_POSTGRES_CONNECTION_REFUSED without retry.

Both files now start a single long-lived listener at module scope and latch
the per-test payload into a module-level variable that the connection handler
reads at accept time. The assertions are unchanged. Each file needs one
listen port and a handful of client ports for the whole run, removing the
per-test listen()+close() churn that tripped the loopback path. The combined
run also drops from ~395ms to ~150ms on Windows release.

Refs #33622 (LPT sharding placed these after node-http-uaf on Windows).
@robobun

robobun commented Jul 11, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 9:06 PM PT - Jul 10th, 2026

⏳ @robobun, your commit 8c1de5d is still building in Build #71822, but has 1 failures so far (All Failures):

@coderabbitai

coderabbitai Bot commented Jul 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

Postgres mock lifecycle

Layer / File(s) Summary
Binary array mock server lifecycle
test/js/sql/postgres-binary-array-bounds.test.ts
The test uses one file-scoped mock server, shared connection payload state, and afterAll shutdown while preserving its existing fault-injection cases.
Invalid message response lifecycle
test/js/sql/postgres-invalid-message-length.test.ts
Startup and query response frames are latched through shared state, with one server reused across tests and closed in afterAll.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: sharing one mock server per file in the PostgreSQL fault-injection tests.
Description check ✅ Passed The description covers the problem, cause, fix, and verification, though it uses custom headings instead of the template labels.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/sql/postgres-binary-array-bounds.test.ts`:
- Around line 43-51: The rationale comment above the shared `current` state is
overly detailed. Trim it to three lines: retain that one mock server is shared
for the file and that each connection reads the `current` payload set before
connecting, while removing the Windows, SYN retransmission, and historical
flakiness explanation.

In `@test/js/sql/postgres-invalid-message-length.test.ts`:
- Around line 21-28: Trim the comment above the shared `current` mock-server
state to three lines, retaining only that one server is reused for the file and
each test sets `current` before connecting to avoid Windows CI
connection-refused flakes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 42bec14b-34fc-45f3-a0f4-19da07efe1f9

📥 Commits

Reviewing files that changed from the base of the PR and between 18cfc1a and df1d850.

📒 Files selected for processing (2)
  • test/js/sql/postgres-binary-array-bounds.test.ts
  • test/js/sql/postgres-invalid-message-length.test.ts

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that hoists the mock server to module scope; assertions unchanged.

What was reviewed:

  • current is destructured into locals at accept time, so each connection latches its own payload; tests run serially so the module-level mutation is safe.
  • connectError's switch from socket.once to the shared startup-flag handler is behaviorally equivalent (no atQuery → later data is ignored).
  • afterAll(server.close) mirrors the prior per-test close ordering (after db.close()), so no new hang risk.
  • Comment blocks are long but document a non-obvious Windows-specific mechanism (SIO_TCP_INITIAL_RTO / no SYN retransmit) — treated as acceptable rationale, not bug-history noise.
Extended reasoning...

Overview

This PR refactors two postgres fault-injection test files to share a single long-lived net.Server per file instead of creating/closing one per test case. Per-test payload is stored in a module-level current variable that the connection handler destructures at accept time. No runtime code is touched; all test assertions are byte-for-byte unchanged.

Security risks

None. Test-only change to mock-server lifecycle in test/js/sql/. No production code, no auth/crypto/permissions paths.

Level of scrutiny

Low. This is a mechanical test-infrastructure refactor to reduce listen()/close() churn that was causing Windows CI flakes under ephemeral-port pressure. The key correctness question is whether sharing state via a module-level let current is race-safe: it is, because (a) bun:test runs test() cases serially by default and neither file uses test.concurrent, and (b) the handler captures current into connection-local consts at accept time, so a lingering socket from a prior test can't observe a later test's payload. I also verified listeningServer in wire-frames.ts uses port: 0 and that top-level await at module scope is the established pattern.

Other factors

The PR description gives a detailed and plausible root-cause analysis (LPT sharding + TIME_WAIT exhaustion + TCP_INITIAL_RTO_NO_SYN_RETRANSMISSIONS on Windows loopback). Even if the exact mechanism isn't proven locally, the fix is independently good: fewer ephemeral ports, faster tests, simpler cleanup. The new explanatory comments run longer than the repo's 3-line guideline, but a verifier already considered and dismissed this — the content is the kind of non-obvious platform rationale the guidelines say belongs in a comment. Not worth blocking on.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that hoists the mock server to module scope; assertions unchanged.

What was reviewed:

  • Verified current is set before each new SQL() and latched per-connection at accept time, so serial tests can't cross-contaminate.
  • Checked the merged handler in invalid-message-length still matches old once/on semantics (atQuery undefined → ignore post-startup data for connect-error cases).
  • Confirmed listeningServer returns a plain net.Server and top-level await + afterAll close is sound; 17/17 tests pass in the PR evidence.
Extended reasoning...

Overview

Refactors two Postgres fault-injection test files to share one long-lived net.Server per file instead of creating/closing a fresh server per test case. Per-test payload is latched into a module-level current variable that the connection handler destructures at accept time. Server is closed in afterAll. No production code touched; all existing assertions and test cases preserved verbatim.

Security risks

None. Test-only change to mock server lifecycle in test/js/sql/. No auth, crypto, or user-facing code paths involved.

Level of scrutiny

Low. This is a mechanical test-infrastructure refactor to reduce Windows CI flakiness caused by per-test listen()/close() churn under TIME_WAIT pressure. The connection-handler logic is byte-identical to before except for reading col/oid/atStartup/atQuery from a latched local instead of a closure parameter. Tests run serially (no test.concurrent), so the module-level current mutation is race-free by construction.

Other factors

  • PR evidence shows 17/17 pass on debug/ASAN.
  • CodeRabbit's two comment-length nits were addressed in 70cb959 and threads are resolved.
  • Checked wire-frames.ts:listeningServer — simple net.createServer + listen(0), compatible with top-level await and afterAll close.
  • The afterAll close relies on each test's sql.close()/db.close({timeout:0}) having dropped client connections so server.close() fires its callback; verified this holds for every test path (all wrap the SQL instance in try/finally with close).

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only refactor hoisting the mock server to file scope; assertions are unchanged.

What was reviewed:

  • Confirmed current is set before each new SQL() connect and latched into locals at accept time, so serial tests don't bleed state.
  • Checked afterAll cleanup pairs with the module-scope listeningServer; per-test sql.close() still runs in finally.
  • Verified the connectError/queryError/last-test paths in the invalid-message-length file all route through the shared handler with equivalent frame sequences.
  • CodeRabbit's comment-length nits were addressed in 70cb959 and threads are resolved.
Extended reasoning...

Overview

This PR refactors two Postgres fault-injection test files (test/js/sql/postgres-binary-array-bounds.test.ts and test/js/sql/postgres-invalid-message-length.test.ts) to create one net.Server per file at module scope instead of one per test case. Each test writes its payload into a module-level current variable before connecting; the accept handler destructures current into connection-local variables. Cleanup moves from per-test finally to afterAll. No production code is touched, and no test assertions are changed.

Security risks

None. This is test infrastructure only — the mock servers bind to 127.0.0.1:0, the wire-frame helpers are unchanged, and no runtime/src code paths are modified.

Level of scrutiny

Low. Test-only, mechanical lifecycle refactor with a well-documented rationale (Windows CI ERR_POSTGRES_CONNECTION_REFUSED flakes from per-test listen/close churn under TIME_WAIT pressure). The 17 assertions are byte-identical to before; only where the server is created and closed moved.

Other factors

  • Tests run serially (no test.concurrent), so the shared current mutable is safe. The handler latches it into locals at accept time, so a late data event on a prior connection can't observe the next test's payload.
  • Each test still closes its SQL client in finally, so server.close() in afterAll won't hang on live connections.
  • All 17 tests pass in the PR's debug/ASAN evidence block.
  • CodeRabbit's two minor comment-length requests were addressed in commit 70cb959 and confirmed resolved.
  • No prior claude[bot] reviews on this PR.

@robobun

robobun commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator Author

Both modified tests pass on all lanes in builds 71813 and 71822. The remaining red is test/js/node/test/parallel/test-worker-message-port-transfer-terminate.js SIGABRT on 13 x64-asan, a pre-existing worker_threads assertion failure also seen in unrelated builds 71706 and 71693 and already being tracked separately; the other annotations are retry-flakes. This diff is green and ready.

@dylan-conway
dylan-conway merged commit 6c88e5a into main Jul 11, 2026
74 of 75 checks passed
@dylan-conway
dylan-conway deleted the farm/d2652c68/sql-mock-server-shared branch July 11, 2026 04:47
robobun added a commit that referenced this pull request Jul 12, 2026
main added its own pgParseComplete/pgBindComplete/pgParameterDescription and a
two-arg pgReadFrontendMessages in #33962; drop the duplicates this PR had
introduced (keeping pgCloseComplete and pgReadCString, which main does not
have) and rewrite the mock server to handle StartupMessage inline like the
other fault-injection fixtures now do.

Switch the real-server test to describeWithContainer (harness), matching the
shape #33986 moved sql-prepare-false to so the test skips correctly on agents
where docker is unavailable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants