Repository navigation
feat: update bench serving - #685
Merged
Merged
Conversation
Contributor
Author
|
test with python3 -m sglang.bench_serving --backend sglang --multi --num-prompts 64 --request-rate-range 1,2result |
merrymercy
approved these changes
Jul 21, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…y pays THE ~47k WAS THE ARENA TAIL, CHARGED TWICE. `arming_floor_subtrahend_bytes` has warned about this since sgl-project#678 -- "Reserving it here as well would be the third payment for one requirement" -- and this is that sentence finally made true for the one term it was not. THE TAIL IS REAL, AND MEASURED AT THE OPERATING POINT. Not another remembered number; it reconciles three independent ways on the 2026-08-16 boot -- the boot's own `rung 3 released ...` lines, the record's `arena_fixed_bytes`, and the documented definition `max(0, pp_bytes - tp_bytes)`: PP0 pp 15790.5 tp 16329.9 -> max(0, -539) = 0 MiB PP1 pp 9792.8 tp 8977.8 = 815 MiB PP2 pp 10434.0 tp 8977.8 = 1456 MiB WHAT IT PHYSICALLY IS: the weights the PP layout needs beyond what the TP layout needs on that rank. Entering PP re-commits it, entering TP releases it. It is largest on the SMALLEST card because uneven TP shrinks that card's TP shard (both 3080s at 8977.8 MiB) while PP still hands it a full stage (9792.8 / 10434.0). The 5090's TP shard is LARGER than its PP share, so it has no tail at all. Geometry of uneven-TP crossed with layer-split PP, not a defect. WHAT WAS WRONG WAS THE TREATMENT. It was charged to the arming floor, a PERMANENT free-VRAM reservation, while the same record already showed the KV rung able to pay it at flip time: PP1 954 MiB of rung_fund against an 815 MiB tail, PP2 1595 against 1456. Reserved permanently it took the binding rank's entire headroom -- 2474 MiB free at rest against a 2467 MiB floor, an equality solve with 7 MiB to spare. as shipped POOL = 435696 (boot: 435319) arena tail funded by the rung POOL = 483723 demonstrated safe 482490 demonstrated unsafe 537076 The binding constraint also moves off the floor: all three ranks now sit near their SEAM solves (505307 / 483723 / 503138), which is the measured cost of the flip itself and the right thing to bind on. CONDITIONAL ON THE RECORDED FUNDING, never unconditional removal. A rank whose rung cannot cover its own tail keeps paying for it in the floor, because for that rank the ladder genuinely cannot find it. `rung_fund_bytes` is 0 on a record that never measured one, and 0 means "no funding known", never "funded". And the result never falls below the leg the rung does NOT fund -- the drafter's restore floors it, so relieving the arena cannot relieve the other leg by arithmetic accident. GATED ON METAL, NOT ON ARITHMETIC. The rung that must pay was latched off until 38c1161, whose recovery aimed above the arena's immutable reservation and failed 59 times out of 59. This relief was deliberately not built until that fix was running: on the boot carrying it the log shows zero recovery failures and the arena tail released 22 times at flip cadence, plus 522/770 MiB rung-3 releases and KV-BACKING releases to 2268 MiB. The provider pays, repeatedly, before anything was sized against it. TESTS, red-first. Five new cases on the draw itself: the two relief cases failed before and pass after, while the three that keep the charge -- an uncovered tail, a rank with no tail, a record that never measured a rung -- passed from the start, so the fix cannot have been "remove the term". THE ACCEPTANCE INVERTED EXACTLY AS DESIGNED. The fifth ship-pin case was committed in 952fc4b asserting `pool < demonstrated_safe` and saying in its own docstring that reaching it would mean the follow-up could close. It now asserts `pool >= demonstrated_safe`, against the artifact's number rather than a literal, so a re-derived pin moves the bar with it. Its fixture floors are now DERIVED from each record's own draw instead of pinned, which is what makes it sensitive to this change at all; the non-draw residue is held fixed so rank 0's corridor-shortfall term is not frozen into the test. Mutation-proven: disabling the relief drops the solve back to exactly 435696 and the acceptance goes red naming it. AND THE OVERSHOOT IS A PROPERTY, NOT A CONSTANT. A new case checks the solve against `SHIP_PIN.demonstrated_unsafe_tokens` and asserts a positive margin, so a future relief that buys too much pool is caught by the same test that certified this one, with no literal for anyone to forget to update. FOLLOW-UP, REGISTERED HERE AND NOT A TASK YET: rank 0's seam slope is 2360.8 B/token against 424.1 and 550.7 on the 3080s -- 5.6x. Mechanism located (`per_row = wave_slack / id_space`; rank 0's staging is genuinely 980 MiB against 176/229) but the ratio is not explained by its token share (1.40x) or its TP-shard ratio (1.82x). It binds nothing after this change -- rank 0 sits at 505307, above the group minimum -- so it is an open question about a number, not a constraint on the pool. Suites: 66 in this file, 2442 across unit/managers + unit/mem_ledger + the resident-carry file. The 4 failures in that sweep are pre-existing and unrelated, parity-proven earlier against the untouched tree.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… anomaly
THE OBSERVATION. Measured per-token seam slopes on the reference rig are
2360.1 / 424.1 / 547.6 B/token. Rank 0 is 5.6x its peers and nothing in the
seam code explains it: `phase_flip_seam_reserve.py:217` carries the triple as
a FROZEN MEASURED CONSTANT (`basis_per_row_bytes`), derived from nothing.
THE MECHANISM, from the code. `phase_flip_runtime._staging_bytes` (:4403) peaks
at `incoming + max(outgoing, local)`, and the incoming leg (:4487-4497) is
`dst.row_nbytes(layer) * rows` summed over `tr.recv_layers[peer]`. So a rank's
per-ROW slope is exactly the number of full-attention layers it must RECEIVE at
the cutover. Nothing else in the formula scales with rows per layer.
HOW MANY LAYERS THAT IS, is decided by the mismatch between the two layouts:
received_r = max(0, tp_share_r * n_attention_total - attention_held_r)
With `--phase-flip-tp-vector 32,16,16` and the cut 28,20,16 (attention [7,5,4]):
rank 0: 0.500 * 16 - 7 = +1 layer -> 1 * 2326.7 = 2326.7 B/token
rank 1: 0.250 * 16 - 5 = -1 layer -> sheds, receives nothing
rank 2: 0.250 * 16 - 4 = 0 layers -> neutral, receives nothing
against measured 2360.1 / 424.1 / 547.6. Rank 0 lands within 1.4 %; the other
two predict ZERO, and their whole measured slope is the baseline every rank
pays regardless (checksums, the one-layer streaming window, allocator grain).
SO RANK 0 IS NOT ANOMALOUS. It is the only rank the flip asks to ACQUIRE KV:
the flip hands it half the token axis while the PP cut gave it only 7 of 16
attention layers. The 5.6x is the ratio between one received layer and a
baseline, not a per-rank pathology, and there is nothing rank-0-specific to
fix -- the asymmetry is the flip vector and the cut disagreeing by one layer.
WHY THIS MATTERS BEYOND THE EXPLANATION, and it invalidates an input of my own
work: THE SLOPE IS A FUNCTION OF THE CUT. `basis_per_row_bytes` is valid only
for the cut and flip vector it was measured at. `received_r` moves in WHOLE
LAYER steps, so any solver that carries the frozen triple across cuts is
pricing the seam of a layout it is not proposing. Moving one attention layer
onto rank 0 raises its slope by another ~2327 B/token; moving one off rank 1
starts charging rank 1 a slope it currently does not pay at all. The KV-floor
fixed point in this branch takes the slopes as parameters, which is why it was
not wrong for `29,19,16` -- that cut leaves the attention split at [7,5,4] --
but it would have been wrong for any cut that moved an attention layer, and
that is now stated rather than latent.
`seam_slope_bytes_per_token` predicts the vector from the flip weights, the
attention counts and the KV cell, so a solve can price the seam of the cut it
is actually considering.
TESTS. Red-first: 5 of 5 fail before the function exists. Green now: rank 0
predicted within 2 % of measured; ranks 1 and 2 predicting exactly zero (the
point of the decomposition -- their slope is all baseline); the measured triple
reproduced when the baselines are supplied; the slope moving by exactly one KV
cell when one attention layer moves onto rank 0 (the property that makes the
frozen constant unsafe); and a mis-sized flip vector refused.
NO REGRESSIONS: test/registered/unit/planner 2561 passed (2556 before, +5),
2 failed -- the same pre-existing test_rejected_evidence_pins pair. Hermetic
(CUDA_VISIBLE_DEVICES=""), no boot, no live flag touched.
FOR F4-r4's FUNDING WORK: the seam demand he is funding is not a per-rank
property to be measured once. It is `received_r` layers of KV per row, and it
re-derives from the layer cut and the flip vector. A rank funding a seam it
does not receive is over-reserving; a cut that changes the attention split
changes who pays.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ead of freezing it
A dependency-free module: it imports nothing from sglang, so the funding path
can read and cherry-pick it without pulling in a solver or a cost model.
THE SLOPE IS THE INCOMING LEG. `phase_flip_runtime._staging_bytes` (:4403)
peaks at `incoming + max(outgoing, local)`, and the incoming term (:4487-4497)
is `dst.row_nbytes(layer) * rows` summed over `tr.recv_layers[peer]`. Nothing
else there scales with rows-per-layer, so a rank's per-ROW slope is exactly the
number of full-attention layers it must RECEIVE at a pp->tp cutover:
received_r = max(0, tp_share_r * n_attention_total - attention_held_r)
clamped at zero because a rank whose PP holding EXCEEDS its TP share SHEDS KV,
and shedding costs no incoming staging.
WHAT THAT EXPLAINS. With --phase-flip-tp-vector 32,16,16 and the cut 28,20,16
(attention [7,5,4]): +1 / -1 / 0 layers, i.e. 2326.7 / 0 / 0 B/row of transfer
against measured 2360.1 / 424.1 / 547.6. Rank 0 lands within 1.4 %; the other
two receive NOTHING and their entire measured slope is the baseline every rank
pays regardless. "Rank 0 is 5.6x its peers" is one received layer against a
baseline -- not a rank-0 pathology. Rank 0 is the only rank this flip asks to
ACQUIRE KV, because it gets half the token axis while holding 7 of 16
attention layers.
FOR THE FUNDING PATH: a rank that receives nothing is being taxed for bytes
that never move. On the shipping cut that is ranks 1 and 2.
WHY A FROZEN VECTOR CANNOT BE CARRIED ACROSS CUTS. `received_r` moves in WHOLE
LAYER steps. Move one attention layer off rank 2 and it starts paying a full
cell it never paid; move one off rank 0 and its slope RISES, because it still
needs half the token axis and now holds less of it. The planted test
`TheFrozenTripleMispricesAMovedCut` is that failure: at attention [7,6,3] the
frozen triple underprices rank 2 by a whole KV cell while the derivation
tracks.
WHAT THIS DELIBERATELY DOES NOT DO, and the brief asked for the opposite, so
the reasoning is here rather than in a reply that scrolls away:
`SHIP_PIN.basis_per_row_bytes` (phase_flip_seam_reserve.py:217) is NOT the live
sizing input and is not touched. It is a frozen PROVENANCE record whose job is
to stay fixed so `test_arming_floor_funding_662.py:1198` can assert the live
values still equal it and go red when they drift. The sizing path reads the
per-boot MEASURED record instead (`per_row_bytes=float(rec["per_row_bytes"])`,
:687, logged as "MEASURED BY A PREVIOUS BOOT"). Overwriting :217 with a derived
vector would have deleted a drift detector while changing no live behaviour at
all. A test pins that the frozen triple is still the measured one.
WHERE THE DERIVED VECTOR DOES BELONG, named as the boundary rather than forced:
the COLD path (:670) returns a SeamReserve with `per_row_bytes = 0.0`, and the
consumers (:927, :1008) then take the floor-bound branch because `a * t_floor
<= F` holds trivially at a = 0 -- a cold boot prices the per-token seam at
NOTHING, which is the cold-overshoot that has OOMed the 3080s. Deriving there
is the real fix, and it is not a small change: `read_seam_reserve(server_args,
world_rank)` receives neither the KV cell size nor the per-stage attention
counts, so wiring it needs those threaded in. Left undone deliberately while
F4-r4 is mid-deploy rather than shipped unvalidated behind a flag nobody would
turn on.
TESTS. 10 green. Red-first is not the module-missing collection error: a
cut-INDEPENDENT implementation returning the frozen triple was substituted and
4 cases go red, including all three planted-failure cases. Covered:
behaviour-neutrality at the current cut and vector (rank 0 within 2 % of the
frozen basis, tolerance named from the 1.4 % measurement error; the full vector
matching once the baselines are supplied); the two ranks that receive nothing
predicting exactly zero; the moved cut making a previously free rank pay a full
cell; the frozen triple underpricing it; moving a layer onto rank 0 raising its
slope by exactly one cell; three refusals; and the pin left untouched.
test/registered/unit/managers: 2029 passed, 4 failed -- all four pre-existing
on this base (this commit adds two NEW files and modifies nothing, per git
status). Two of them, test_collective_family_siblings_610.py
PrefillAdmissionBudgetTest, are in the admission-budget family and are flagged
separately. Hermetic (CUDA_VISIBLE_DEVICES=""), no boot, no live flag touched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…the shared module `pp_cut.seam_slope_bytes_per_token` now delegates to `managers.seam_slope.derive_seam_slope_bytes_per_token` instead of carrying a second implementation of the same formula. Kept as its own commit so the previous one stays cherry-pickable into a deploy line on its own: that commit adds a dependency-free module and its tests and touches nothing else, and this one is a pure de-duplication that a cherry-pick does not need. WHY DE-DUPLICATING THIS PARTICULAR FORMULA IS NOT COSMETIC. The finding is that the slope is NOT a constant -- it moves in whole-layer steps with the cut. Two copies of a quantity whose whole point is that it re-derives are two things to forget to re-derive. planner 2561 passed, 2 failed (the same pre-existing test_rejected_evidence_pins pair). Hermetic, no boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…chor it does not have THE DEFECT. A cold ``SeamReserve`` has every measured field at zero, ``per_row_bytes`` included, so the per-token seam term is 0 on a first boot and the pool is sized floor-only -- a pool whose cutover it cannot fund. The arming FLOOR has been charged on a cold record since sgl-project#662-F4/A0; the SLOPE never was. WHAT IS NOW DERIVED. sgl-project#685 established that a rank's per-row seam cost is the full-attention layers it must RECEIVE at the cutover, ``max(0, share*n_total - held)``. Every term is rank-LOCAL and known at boot, which is what makes it usable where nothing has been measured or exchanged: the share from ``--phase-flip-tp-vector``, ``held`` from the attention modules this runner actually built (``_lane_kv_bearing_layer_count``, family-neutral by construction), the total from the model config, and the per-layer cell from the configurator's own ``_cell_size`` divided by this rank's attention count. No collective, no measurement, nothing transferred from another rig. ``derive_seam_slope_for_rank`` is the single-rank form; it agrees with the whole-vector one, asserted. A FALLBACK, NEVER AN OVERRIDE. Any non-cold provenance is returned untouched -- identity-asserted for both STORED and OVERRIDE. A measured record is a fact about this rig and outranks a model of it. ABSTAIN RATHER THAN APPROXIMATE. Missing flip vector, unparsable vector, rank outside it, no attention layer on this runner, no single cell (the hybrid-SWA / MiniMax case the existing cell read already refuses to invent for), no full-attention layer in the config -- each leaves the slope at zero and says which one. A rank that receives NOTHING is also left at zero: charging it a per-token seam would reserve against a transfer that does not happen, which is the over-reservation sgl-project#685 named. AND HERE IS WHERE IT STOPS, WHICH IS THE HALF THAT MATTERS. The slope is derived and announced, but the reserve stays INACTIVE and the budget path still returns floor-only. ``SeamReserve.active`` requires ``id_space > 0``, and the downstream solve anchors on that measurement point together with ``have_bytes`` (``t_floor = t_m + (have_m - A - F) // cell``). A derived slope has no measurement point. Setting those two to make it active would fabricate the anchor -- precisely the approximation this work is under orders not to make, and the same watchdog-versus-live-input confusion as the ``basis_per_row_bytes`` catch one level down. Consuming it needs an anchor-free branch. ``solve_pool_tokens`` IS that form and takes no anchor, but it has NO live caller anywhere in the tree, so which budget it is solved against (``budget_bytes`` net of the floor charge, or something the sizer has not yet subtracted) is a design decision on the boot path rather than a wiring detail. On a change that cannot be deployed or measured from here, picking it would be guessing with the boot as the test. SO WHAT THIS COMMIT ACTUALLY BUYS: the cold per-token seam is no longer silently zero. It is computed, and a boot that would previously have been sized floor-only in silence now prints the number, the layer count it came from and the share it was derived against. That is the difference between an unfunded cutover nobody can see coming and one named at boot. It is NOT yet the sizing fix, and the tests say so rather than implying otherwise: a case pins that the derived reserve does NOT become active, with the reason. TESTS. Red-first is a genuine ImportError on the missing single-rank API, then 14 green. Covered: rank 0 receiving exactly one layer; the shedding and neutral ranks receiving nothing; agreement with the whole-vector form; an out-of-range rank refused; the cold reserve gaining the slope with a log naming it derived; the consumption boundary pinned; STORED and OVERRIDE returned by identity; all five abstentions; and two sizing cases showing a priced slope grants strictly fewer tokens than a zero one -- so the derivation is not decoration once it can be consumed. test/registered/unit/managers: 2052 passed, 4 pre-existing failures (one further ZMQ round-trip failure in the batch run passes 22/22 in isolation -- flaky, not this change). Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY: this touches the boot sizing path and F4-r4 decides when it enters his line.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…e two commits to hold MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test evidence, whether it touches runtime, and whether an equivalent patch is already on the serving line (verified with `git cherry`, not by message matching -- four are: c41645c, ce60358, 658ea3a, 84b0171). DRY RUN: clean. Merged into `integration/r2` -- the live line, since the serving tree descends from its tip a73a0d8 -- in a throwaway worktree, `--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts, zero unmerged paths, so nothing was pre-resolved because nothing needed it. Verified semantically as well as textually: on the MERGED tree, managers 2093 passed / 0 failed and planner 2574 passed / 2 failed, the two being the same pre-existing test_rejected_evidence_pins pair that is already red on the base. THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is NOT an ancestor of integration/r2, so merging this branch drags in its whole base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677 work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files, +22731/-762. Approving this merge is approving that lineage, most of which is not mine to vouch for. If only this work is wanted it must be cherry-picked rather than merged. NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four already on serving -- merging them only reconciles integration with what is already running; (2) the three sgl-project#624 test-only drift guards, which take managers from 4 failures to 0 and should land early so the line stays green during review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by no serving path; (4) hold. HOLD, two commits, both runtime and neither on the serving line: * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE format. Already queued to land on deploy WITH the W=8/W=4 probe after the sgl-project#694 soak verdict; merging it into integration first puts it in front of the soak meant to measure it. * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and abstention-guarded, but unsoaked, and the R' decision it waits on is not made. Nothing else in the chain can move serving behaviour. Docs only; no merge performed, no deploy, scratch worktree removed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… evaporates
THE DEFECT, MEASURED under lane load 2026-08-16: pp_to_tp abandoned on PP1
every ~3 s -- 18 abandons against 9 cutovers in 8 minutes, three ARM-UNFUNDED
in nine seconds -- while decode sat at 187.5 s against a 180 s budget. The
shortfall was identical every time, which is what gave it away:
staging 733 MiB needed but only 691 MiB is spendable (x8)
A runtime fluctuation does not repeat to the megabyte. That is a SIZING-time
constant.
WHERE IT CAME FROM. arming_draw_bytes excused the weights-arena tail from the
arming floor whenever the KV rung was RECORDED as able to pay it. On PP1: arena
814.9 MiB, rung_fund_bytes 953.8 MiB, so the draw fell 815 -> 138.9 and 815 MiB
stopped being permanently reserved. The pool was sized that much larger, and
the enlarged pool is what left 691 MiB spendable against a 733 MiB need.
WHY THE EXCUSE IS UNSOUND -- and it is not that the measurement was wrong.
rung_fund_bytes is a SINGLE-POINT reading. The rung's deliverable is
(current_rows - floor_rows) * bytes_per_row, and floor_rows tracks the LIVE
SET. At the abandon:
current=473088 rows, floor=471983, slack=1105 -> 12.3 MiB deliverable
954 MiB promised, 12.3 MiB available. The funding collapses precisely when
occupancy is high, which is precisely when a seam is hard to fund. An excuse
granted at low fill is spent at high fill.
THE TWO CANDIDATE REPAIRS ARE ONE REPAIR. "Price the excuse against the worst
case at max fill" and "drop the excuse" give the same answer, because at max
fill current approaches floor_rows and the guaranteed deliverable is ZERO. So
the predicate is written in the general form -- excuse only what the rung is
GUARANTEED to deliver, via a new rung_guaranteed_bytes -- and a record that has
never measured a guarantee excuses nothing. The mechanism survives for a future
measurement instead of being deleted, and sgl-project#685's own condition ("conditional on
the recorded funding, never an unconditional removal") finally holds, because
the condition is now a bound the rung cannot fall below rather than a number it
was once observed at.
ON THE "THIRD PAYMENT" OBJECTION, which is the real tension and is answered
rather than ignored. TheArenaTailIsNotChargedTwice quotes
arming_floor_subtrahend_bytes: "Reserving it here as well would be the third
payment for one requirement." That reasoning holds only if payments one and two
actually discharge the requirement at flip time. Measured, they do not: the
rung covers 12.3 MiB of an 815 MiB tail at the fill where the seam runs. The
third payment is therefore not redundant -- it is the only one that holds when
it matters.
CORROBORATION I DID NOT CONSTRUCT. The restored floors are
rank0 1728 (unchanged), rank1 1728 -> 1825, rank2 1728 -> 2467 MiB
which is exactly SHIP_PIN.basis_arming_floor_mib = (1728, 1825, 2467). The pin
already encodes the UN-EXCUSED floors, so the excuse had silently undercut the
pin its own regime was derived at; this returns the live floors to it.
COST, stated plainly: ~8.7k tokens of pool on rank1 and ~83k on rank2. rank0 is
untouched. That is the price of a seam that can arm, against a pool that is
larger and cannot.
TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
test_arming_draw_excuse_696.py 6 passed (new, red-first: pre-fix it fails
with "138.9 not >= 814.0", reproducing the
PP1 arithmetic that produced the 42 MiB
shortfall)
test_arming_floor_funding_662 66 passed -- the four sgl-project#685 fixtures now state
the condition as a GUARANTEE. Their intent is
the conditional-relief mechanism, so only the
field carrying "the rung can pay" changed; no
expected value was flipped.
managers + mem_cache 64 failed / 1908 passed, the same 64 as the
pre-change baseline.
NOT DEPLOYED. Rides the next window with 8fb86ef, e21e87f, c738ef5
and the W-probe.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… dry run, and the vouch The notes were written at 3c984ad and had gone stale in three ways. Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697 (8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime despite its ticket -- it adds a handover script and one test guard. Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was re-run rather than carried over. Still zero conflicts. Scope shrank with the target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196. Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in test_rejected_evidence_pins.py and were re-verified as pre-existing by aborting the merge and running that file on clean a157bf1 -- same 2 failed, 2 passed. The chain introduces no new failure. Because the target tip moved, this was re-checked instead of quoting revision 1's finding. Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list -- desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays held on the unmade R' decision. Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the originals by git patch-id --stable -- f630947/c41645c8c9, 7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so preferring the originals is free and the only real risk is double attribution. 1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is diagnostic scaffolding and not mine to vouch for. Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's line, not this branch, and are listed only because they lift holds; sgl-project#696 is what was DoSing the lanes. Docs only. No merge performed; the dry-run worktree was removed.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…ribution, rank-2 root 1. RESTORE, NEVER REBUILD -- the pin (test_restore_never_rebuild_677.py). A fence patches the BUILD entry points (weights_arena.allocate_arena, weights_arena.pack_into_arena, torch.cuda.CUDAGraph) to raise, and the REAL production mover PhaseFlipStacks.refill runs under it on both legs, including the checksum-mismatch restore arm -- the one branch that touches the arena twice and is likeliest to reach for a rebuild. arena_refill is deliberately NOT fenced: it is the copy the flip exists to perform, and a pin that fenced it would pass by forbidding the work. Can-fail arms: every entry point is shown to actually raise, and a planted mover calling allocate_arena is caught. Pinning production rather than a re-implemented loop is the point -- a re-implementation keeps passing while production drifts (sgl-project#624). 2. PEAK-INSTANT ATTRIBUTION -- PhaseFlipRuntime._record_seam_peak, emitted on the sgl-project#605 flight-recorder channel (not a new one: it already carries the torch view, the NVML view and the boot id, and is append-only). Placed at _staging_affordable, the instant the flip's demand is weighed against free VRAM -- earlier the buffers do not exist, later the decision is already taken. Carries staging_bytes, refill_destination_bytes, graph_workspace_bytes, the reserve, driver free, allocator cached free, and a SIGNED unattributed_bytes. Two choices, both pinned and both mutation-proven: unmeasured components are None and never 0 (a zero reads as "costs nothing" -- the sgl-project#606 defaulted- measurement defect), and the residual is signed, because a negative one means the named terms OVER-count, a different defect that max(0, ...) would hide. Guarded end to end: an instrument on the seam path may cost a line, never a cutover. 3. THE RANK-2 ANOMALY -- the sgl-project#685 candidate is REFUTED. sgl-project#685 (0e50e48, f1774d7) is an UnboundLocalError use-before-bind on a variable named 'cell' in the cold seam-pricing branch; it has no arena-tail content and no 1456 MiB figure exists anywhere in source or records. The live candidate is arena GROWTH, from the sgl-project#690 image sizes: rank 0 PP 12619.6 TP 9614.9 growth 0.0 seam draw 909 rank 1 PP 9014.0 TP 9614.9 growth 600.9 seam draw 1006 rank 2 PP 7211.2 TP 9614.9 growth 2403.7 seam draw 1648 Rank 0's PP layout is the larger, so it never grows; rank 2 grows four times what rank 1 does. The ordering matches the draws exactly and no other per-rank term does. Stated as a CANDIDATE, not a finding: the excess over rank 0 is 16% of the growth on rank 1 and 31% on rank 2, so growth explains the order and not the size. Instrument 2 settles it without a dedicated experiment -- it emits exactly this quantity at the peak instant. FILED, NOT BUILT: TICKET_718_contiguous_destination_extent.md -- making a peer's destination extent contiguous so a BAR1 write lands in place and the receive buffer stops existing. The largest structural reduction available, and the only one that removes a buffer rather than relocating it; it touches the reshard wire format and the pool allocator together, so it belongs to a deliberate design pass. The ticket names what must be decided, what must not be assumed (it trades allocator freedom for staging bytes, and that freedom is what keeps admission working under fragmentation), and makes a live reading from instrument 2 its prerequisite. Tests, hermetic (CUDA_VISIBLE_DEVICES="", no GPU, no serving contact): test_restore_never_rebuild_677.py 7 passed + 3 subtests test_seam_peak_attribution_677.py 8 passed MUTATION PROOF: null->0 and signed->floored fails 3 of 8 FULL SUITE compared by failing NODE ID against base b786858, ANSI stripped (counts alone hid a real regression earlier in this branch): base 13 failed / 2181 passed; now 13 failed / 2208 passed regressions: NONE. The 13 are pre-existing. ruff + codespell clean on all changed files. Desk only. Live readings ride the later review boot.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…aim: the arena tails are measured and recorded The sgl-project#702 repricing under the cold-spill doctrine found the number I said did not exist. NOTE_677 section 8 stated "no 1456 MiB figure appears anywhere in the records or source". WRONG: managers/phase_flip_seam_reserve.py, record_path docstring, records the measured per-rank arena tails -- "1436 MiB on rank2 against 466 MiB on rank1 and 0 on rank0". 1436 rather than 1456, but plainly the figure the earlier candidate meant. I missed it by grepping the docs tree and the ticket number instead of the module that owns the quantity. The sgl-project#685 ticket attribution was still wrong (it is an UnboundLocalError); the NUMBER was real. It also settles, without a boot, what NOTE_677 left as a candidate. Against the standing pool reduction per rank -- 704 / 801 / 1443 MiB, i.e. arming_floor_subtrahend_bytes = floor - max(corridor law, already_reserved) -- the measured tail is 99.5% of rank 2's, 58% of rank 1's, and 0% of rank 0's. Rank 2's floor excess IS its arena tail. The repricing itself lands in the evidence tree (not a git repo): /spinning/evidence-665-f1/NOTE_702_CUT_TABLE.md sections D1-D6, referenced from an append-only block in PLAN_PERF_PIPELINE_2026-08-16.md (head-298 sha256 verified byte-identical, 81560e94b7628e6c). Headline: crediting the standing reduction back, [31,17,16] recovers to 530,381 -- EXACTLY [31,18,15]'s spill-funded figure, because once floors stop binding both bind PP0 on the same bytes. The pool advantage that made [31,18,15] interesting was an artefact of VRAM-resident floors, as the user said; with it gone [31,17,16] wins outright on speed and is the only one of the two above the 14.1% noise floor. Build NOT undertaken and NOT small: nothing credits ON-DEMAND capacity against the standing floor, and crediting the evict rung's PRICED capacity would be pricing on intention rather than completion -- sgl-project#717's defect one layer up, with an OOM at arm time as its failure mode. Ticket filed at D6 with one live _record_seam_peak reading as prerequisite. Desk only, no boot, no GPU, no serving contact.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…ke the verdict an observation WINDOW-946FIX-0828 MEASURED A COMPENSATOR THAT WAS, FOR THE FIRST TIME IN THIS FAMILY, PROVABLY ON THE DEFECT'S PATH AND STILL DID NOTHING. The sgl-project#946 escape fired 8885 times (act_rid_hit) while PREFETCH RE-ISSUED stayed 0, cached>0 stayed 0, and the rid stayed frozen at told=8192. Root: `_prefetch_kvcache` returned None on every one of its paths and the escape read that as success -- `prefetch(req); return "refetch"`. ITS SUCCESS VALUE WAS NOT EVIDENCE THAT IT ACTED. Five earlier instances were compensators placed off the path the failure takes; this one is a compensator whose own report cannot distinguish acting from not acting. SAME CLASS, NAMED SIBLINGS: sgl-project#872 (silent falsy paths), sgl-project#501 (the contract promises what the code does not deliver), sgl-project#685 ("a silent precondition is indistinguishable from dead code"), sgl-project#656/HANDOFF_678 4.0 (four early ABSTAIN returns above the trace), sgl-project#767 (name the eaten branch). THE VERDICT IS AN OBSERVATION, NOT A PROMISE. There are SIX silent exits between the escape and registration -- two in `_prefetch_kvcache`, four inside `prefetch_from_storage`. Enumerating them one at a time is how this family has failed five times: a seventh added later would be silent again. So the verdict is taken from the EFFECT -- did the operation appear in `ongoing_prefetch` (unified_radix_cache.py:2996)? -- which covers every exit at once, including ones nobody has written yet. Deep reasons come from the sgl-project#915 gate counters as a per-call DELTA (anchor / too_short / rate_limited). CONSEQUENCE, and it is where the old code leaked: the mark was cleared BEFORE the attempt, so a DECLINED re-fetch spent it and the request never saw the escape again -- invisible while every decline reported success. Now an OUTCOME spends the mark, an attempt does not: issued clears it, a decline keeps it and retries, bounded by REFETCH_DECLINE_CAP=3, then the terminator. Kein-Doppel- Prefill unchanged: re-fetch preferred, told=0 the bounded last resort that still names its discard. sgl-project#915 WIRED AT LAST. `note_prefetch_gate` has recorded the decline reason on every attempt since sgl-project#915 landed; `format_prefetch_gate` had ZERO callers, so it was never reported -- PRESENT-BUT-UNWIRED for twelve days, printed by every boot script and read as a note. Now logged on the census cadence. CANDIDATE (iii) REFUTED BY MEASUREMENT, not by argument. The suspicion was the sgl-project#915 host-tier capacity asymmetry (prefetch_capacity_limit = 0.5 * mem_pool_host.size, cache_controller.py:841 -- NOT :729, a stale pointer in sgl-project#915's own comment). The gate line is byte-identical across both stages of the window (attempted=6, anchor/too_short/rate_limited ABSENT), so the escape never reaches the gate at all: it returns at the ANCHOR exit above it. The escape's precondition (`last_host_node.backuped`) is anti-correlated with the situation the escape exists for. TESTS, each module alone in a fresh process, 33-module blast radius derived from the touched symbols, identical script both sides vs the 707f234 baseline: BEFORE 1017-11 = 1006 passed, 9 failed AFTER 1017 passed, 9 failed Exactly ONE module differs: test_chunked_continuation_clamp_946.py 28 -> 39 (+11 new arms). Every other module matches its baseline EXACTLY, including test_unified_radix_cache_unittest.py (642 passed / 860 skipped / 44 subtests, both sides). The 9 failures are PRE-EXISTING at the pin and identical on both sides: test_prefetch_progress_symmetry_580.py (7) and test_collective_family_siblings_610.py (2). Extraction count probe: 9 == 9. CAN-FAIL PROVEN IN BOTH DIRECTIONS, not asserted: a mutant returning "issued" unconditionally is caught by the delivery arm; restoring the old consume- before-attempt is caught by the mark arms. Four contract inversions recorded at their sites -- stubs that returned None used to read as successful re-fetches, which is the defect itself encoded as a test. ruff (F401,F821,UP037) and ruff format clean. The two codespell hits in scheduler.py are pre-existing at the pin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
as titled cc @merrymercy @Ying1123 @hnyls2002
Modification
as titled
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.