feat(bench/http): Bier vs PostgREST HTTP benchmark harness#68
Merged
Conversation
k6-driven native macOS benchmark (run.sh) comparing Bier and PostgREST v14.12 across read/write scenarios, with parity checks, closed-loop ceiling calibration, and open-loop latency measurement; plus the design spec, plan, and generated REPORT.md. Two stability fixes over the initial harness: - run.sh raises `ulimit -n` to 65536 before launching either server; the open-loop latency burst otherwise hit macOS's 256-fd default and crashed the Bier BEAM with :emfile. - Split the arrival-rate fraction per scenario class (reads 0.7/0.4, writes 0.3/0.2): the closed-loop ceiling over-predicts open-loop write throughput, driving PostgREST past its knee and tripping the dropped-iterations guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four independent fixes, each verified by re-running the harness; together they produced the first fully completed run (1 round, all 8 latency stages, 0 dropped iterations, 0 failed requests): - run.sh: bounded server teardown. stop_server sent one SIGTERM and waited unbounded — an orphaned PostgREST once hung a run for ~5h. Now SIGTERM -> 10s liveness poll -> SIGKILL -> reap any orphaned port listener; the EXIT trap sweeps both bench ports. - scenarios.js: VU burst headroom (preAllocatedVUs 1000 -> 2000, maxVUs 1000 -> 8000) so a server stall (PostgREST's GHC stop-the-world GC) is measured as open-loop queueing tail latency instead of shedding load and voiding the stage. - run.sh: ephemeral-port drain guard before each open-loop stage. k6's arrival-rate executors round-robin iterations across the whole VU pool, opening ~2x preAllocatedVUs keep-alive connections per stage regardless of concurrency; on exit they all sit in client-side TIME_WAIT (30s), and back-to-back short stages exhausted macOS's 16,384-port ephemeral range (EADDRNOTAVAIL at the 5th consecutive stage). Wait for TIME_WAIT toward the bench ports to drop below 1000 before each latency stage. - run.sh: full-run RATE_FRACTION_READ 0.7 -> 0.4, calibrated by probing PostgREST's open-loop knee (r1, 60s windows): 0.5-0.7 of the closed-loop ceiling queue-spiral into its 10s pool-acquisition timeout 100% of the time; 0.4 sustains with zero drops while still capturing GC pauses as p90/p99 tail. WARMUP/DURATION/ROUNDS/VUS are now env-overridable around the per-mode defaults (e.g. ROUNDS=1 for a single-round full run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZEyhWcdyTwk4e7M2qQrPo
…67 rerun report The 0.4 read fraction is only sustainable against a cold-probed ceiling: over the ~40-min run the rig drifts and PostgREST queue-spiraled in round 3 at the same arrival rate (p99 74ms -> 556ms -> 2.7s). 0.3 keeps the late-run effective fraction at or below the cold 0.4 knee. Also pin k6's name tag to URL templates so id-randomizing scenarios stop minting >100k time series per stage. REPORT.md regenerated post-#67 (3 rounds, clean): R2 flips from 0.80x to 3.85x throughput; Bier now leads max throughput in all four scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tight M2 tails ~80k updates per M2 window tripped autovacuum every write stage; when the pass landed inside a measurement window it inflated whichever server was being measured (p99 5ms -> ~50-134ms, p50/p90 flat) — run 2 hit Bier, run 3 hit PostgREST. TRUNCATE + post-reset ANALYZE already restore identical state, so autovacuum only added nondeterminism. Run 4 with the fix: M2 p99 tight across all rounds for both servers (Bier 2.38ms 2.36-3.47, PostgREST 6.22ms 5.90-12.19). Throughput and p50/p90 reproduce runs 2-4; read-side p99 remains rig-noisy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he bench harness Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bench/http/) comparing Bier against PostgREST v14.12 across read (single-row PK, filtered page) and write (insert, update) scenariosbench/http/REPORT.md) and the design spec/plan underdocs/superpowers/Test plan
bench/http/run.shcompletes a full run end-to-end (see REPORT.md results)mix test/ CI — standalone harness, nolib/changes🤖 Generated with Claude Code