Skip to content

feat: update bench serving - #685

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:bench
Jul 21, 2024
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:bench

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 21, 2024

Copy link
Copy Markdown
Contributor

Motivation

as titled cc @merrymercy @Ying1123 @hnyls2002

Modification

as titled

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs

zhyncs commented Jul 21, 2024

Copy link
Copy Markdown
Contributor Author

test with

python3 -m sglang.bench_serving --backend sglang --multi --num-prompts 64 --request-rate-range 1,2

result

{"backend": "sglang", "dataset_name": "sharegpt", "request_rate": 1, "total_input": 14688, "total_output": 12790, "median_ttft": 32.953862100839615, "median_itl": 12.583150564412666, "output_token_throughput": 187.74840245645584, "sharegpt_output_len": null, "random_input_len": 1024, "random_output_len": 128}
{"backend": "sglang", "dataset_name": "sharegpt", "request_rate": 2, "total_input": 14688, "total_output": 12790, "median_ttft": 31.570294871926308, "median_itl": 13.26479745871364, "output_token_throughput": 344.5204016475801, "sharegpt_output_len": null, "random_input_len": 1024, "random_output_len": 128}

@zhyncs
zhyncs merged commit 0fdb312 into sgl-project:main Jul 21, 2024
@zhyncs
zhyncs deleted the bench branch July 21, 2024 06:47
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…y pays

THE ~47k WAS THE ARENA TAIL, CHARGED TWICE. `arming_floor_subtrahend_bytes`
has warned about this since sgl-project#678 -- "Reserving it here as well would be the
third payment for one requirement" -- and this is that sentence finally made
true for the one term it was not.

THE TAIL IS REAL, AND MEASURED AT THE OPERATING POINT. Not another remembered
number; it reconciles three independent ways on the 2026-08-16 boot -- the
boot's own `rung 3 released ...` lines, the record's `arena_fixed_bytes`, and
the documented definition `max(0, pp_bytes - tp_bytes)`:

    PP0  pp 15790.5  tp 16329.9  -> max(0, -539) =    0 MiB
    PP1  pp  9792.8  tp  8977.8                  =  815 MiB
    PP2  pp 10434.0  tp  8977.8                  = 1456 MiB

WHAT IT PHYSICALLY IS: the weights the PP layout needs beyond what the TP
layout needs on that rank. Entering PP re-commits it, entering TP releases it.
It is largest on the SMALLEST card because uneven TP shrinks that card's TP
shard (both 3080s at 8977.8 MiB) while PP still hands it a full stage
(9792.8 / 10434.0). The 5090's TP shard is LARGER than its PP share, so it has
no tail at all. Geometry of uneven-TP crossed with layer-split PP, not a defect.

WHAT WAS WRONG WAS THE TREATMENT. It was charged to the arming floor, a
PERMANENT free-VRAM reservation, while the same record already showed the KV
rung able to pay it at flip time: PP1 954 MiB of rung_fund against an 815 MiB
tail, PP2 1595 against 1456. Reserved permanently it took the binding rank's
entire headroom -- 2474 MiB free at rest against a 2467 MiB floor, an equality
solve with 7 MiB to spare.

    as shipped                     POOL = 435696   (boot: 435319)
    arena tail funded by the rung  POOL = 483723
    demonstrated safe                     482490
    demonstrated unsafe                   537076

The binding constraint also moves off the floor: all three ranks now sit near
their SEAM solves (505307 / 483723 / 503138), which is the measured cost of the
flip itself and the right thing to bind on.

CONDITIONAL ON THE RECORDED FUNDING, never unconditional removal. A rank whose
rung cannot cover its own tail keeps paying for it in the floor, because for
that rank the ladder genuinely cannot find it. `rung_fund_bytes` is 0 on a
record that never measured one, and 0 means "no funding known", never "funded".
And the result never falls below the leg the rung does NOT fund -- the
drafter's restore floors it, so relieving the arena cannot relieve the other
leg by arithmetic accident.

GATED ON METAL, NOT ON ARITHMETIC. The rung that must pay was latched off until
38c1161, whose recovery aimed above the arena's immutable reservation and
failed 59 times out of 59. This relief was deliberately not built until that
fix was running: on the boot carrying it the log shows zero recovery failures
and the arena tail released 22 times at flip cadence, plus 522/770 MiB rung-3
releases and KV-BACKING releases to 2268 MiB. The provider pays, repeatedly,
before anything was sized against it.

TESTS, red-first. Five new cases on the draw itself: the two relief cases
failed before and pass after, while the three that keep the charge -- an
uncovered tail, a rank with no tail, a record that never measured a rung --
passed from the start, so the fix cannot have been "remove the term".

THE ACCEPTANCE INVERTED EXACTLY AS DESIGNED. The fifth ship-pin case was
committed in 952fc4b asserting `pool < demonstrated_safe` and saying in its
own docstring that reaching it would mean the follow-up could close. It now
asserts `pool >= demonstrated_safe`, against the artifact's number rather than
a literal, so a re-derived pin moves the bar with it. Its fixture floors are
now DERIVED from each record's own draw instead of pinned, which is what makes
it sensitive to this change at all; the non-draw residue is held fixed so rank
0's corridor-shortfall term is not frozen into the test. Mutation-proven:
disabling the relief drops the solve back to exactly 435696 and the acceptance
goes red naming it.

AND THE OVERSHOOT IS A PROPERTY, NOT A CONSTANT. A new case checks the solve
against `SHIP_PIN.demonstrated_unsafe_tokens` and asserts a positive margin, so
a future relief that buys too much pool is caught by the same test that
certified this one, with no literal for anyone to forget to update.

FOLLOW-UP, REGISTERED HERE AND NOT A TASK YET: rank 0's seam slope is 2360.8
B/token against 424.1 and 550.7 on the 3080s -- 5.6x. Mechanism located
(`per_row = wave_slack / id_space`; rank 0's staging is genuinely 980 MiB
against 176/229) but the ratio is not explained by its token share (1.40x) or
its TP-shard ratio (1.82x). It binds nothing after this change -- rank 0 sits
at 505307, above the group minimum -- so it is an open question about a number,
not a constraint on the pool.

Suites: 66 in this file, 2442 across unit/managers + unit/mem_ledger + the
resident-carry file. The 4 failures in that sweep are pre-existing and
unrelated, parity-proven earlier against the untouched tree.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… anomaly

THE OBSERVATION. Measured per-token seam slopes on the reference rig are
2360.1 / 424.1 / 547.6 B/token. Rank 0 is 5.6x its peers and nothing in the
seam code explains it: `phase_flip_seam_reserve.py:217` carries the triple as
a FROZEN MEASURED CONSTANT (`basis_per_row_bytes`), derived from nothing.

THE MECHANISM, from the code. `phase_flip_runtime._staging_bytes` (:4403) peaks
at `incoming + max(outgoing, local)`, and the incoming leg (:4487-4497) is
`dst.row_nbytes(layer) * rows` summed over `tr.recv_layers[peer]`. So a rank's
per-ROW slope is exactly the number of full-attention layers it must RECEIVE at
the cutover. Nothing else in the formula scales with rows per layer.

HOW MANY LAYERS THAT IS, is decided by the mismatch between the two layouts:

    received_r = max(0, tp_share_r * n_attention_total - attention_held_r)

With `--phase-flip-tp-vector 32,16,16` and the cut 28,20,16 (attention [7,5,4]):

    rank 0:  0.500 * 16 - 7 = +1 layer  -> 1 * 2326.7 = 2326.7 B/token
    rank 1:  0.250 * 16 - 5 = -1 layer  -> sheds, receives nothing
    rank 2:  0.250 * 16 - 4 =  0 layers -> neutral, receives nothing

against measured 2360.1 / 424.1 / 547.6. Rank 0 lands within 1.4 %; the other
two predict ZERO, and their whole measured slope is the baseline every rank
pays regardless (checksums, the one-layer streaming window, allocator grain).

SO RANK 0 IS NOT ANOMALOUS. It is the only rank the flip asks to ACQUIRE KV:
the flip hands it half the token axis while the PP cut gave it only 7 of 16
attention layers. The 5.6x is the ratio between one received layer and a
baseline, not a per-rank pathology, and there is nothing rank-0-specific to
fix -- the asymmetry is the flip vector and the cut disagreeing by one layer.

WHY THIS MATTERS BEYOND THE EXPLANATION, and it invalidates an input of my own
work: THE SLOPE IS A FUNCTION OF THE CUT. `basis_per_row_bytes` is valid only
for the cut and flip vector it was measured at. `received_r` moves in WHOLE
LAYER steps, so any solver that carries the frozen triple across cuts is
pricing the seam of a layout it is not proposing. Moving one attention layer
onto rank 0 raises its slope by another ~2327 B/token; moving one off rank 1
starts charging rank 1 a slope it currently does not pay at all. The KV-floor
fixed point in this branch takes the slopes as parameters, which is why it was
not wrong for `29,19,16` -- that cut leaves the attention split at [7,5,4] --
but it would have been wrong for any cut that moved an attention layer, and
that is now stated rather than latent.

`seam_slope_bytes_per_token` predicts the vector from the flip weights, the
attention counts and the KV cell, so a solve can price the seam of the cut it
is actually considering.

TESTS. Red-first: 5 of 5 fail before the function exists. Green now: rank 0
predicted within 2 % of measured; ranks 1 and 2 predicting exactly zero (the
point of the decomposition -- their slope is all baseline); the measured triple
reproduced when the baselines are supplied; the slope moving by exactly one KV
cell when one attention layer moves onto rank 0 (the property that makes the
frozen constant unsafe); and a mis-sized flip vector refused.

NO REGRESSIONS: test/registered/unit/planner 2561 passed (2556 before, +5),
2 failed -- the same pre-existing test_rejected_evidence_pins pair. Hermetic
(CUDA_VISIBLE_DEVICES=""), no boot, no live flag touched.

FOR F4-r4's FUNDING WORK: the seam demand he is funding is not a per-rank
property to be measured once. It is `received_r` layers of KV per row, and it
re-derives from the layer cut and the flip vector. A rank funding a seam it
does not receive is over-reserving; a cut that changes the attention split
changes who pays.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ead of freezing it

A dependency-free module: it imports nothing from sglang, so the funding path
can read and cherry-pick it without pulling in a solver or a cost model.

THE SLOPE IS THE INCOMING LEG. `phase_flip_runtime._staging_bytes` (:4403)
peaks at `incoming + max(outgoing, local)`, and the incoming term (:4487-4497)
is `dst.row_nbytes(layer) * rows` summed over `tr.recv_layers[peer]`. Nothing
else there scales with rows-per-layer, so a rank's per-ROW slope is exactly the
number of full-attention layers it must RECEIVE at a pp->tp cutover:

    received_r = max(0, tp_share_r * n_attention_total - attention_held_r)

clamped at zero because a rank whose PP holding EXCEEDS its TP share SHEDS KV,
and shedding costs no incoming staging.

WHAT THAT EXPLAINS. With --phase-flip-tp-vector 32,16,16 and the cut 28,20,16
(attention [7,5,4]): +1 / -1 / 0 layers, i.e. 2326.7 / 0 / 0 B/row of transfer
against measured 2360.1 / 424.1 / 547.6. Rank 0 lands within 1.4 %; the other
two receive NOTHING and their entire measured slope is the baseline every rank
pays regardless. "Rank 0 is 5.6x its peers" is one received layer against a
baseline -- not a rank-0 pathology. Rank 0 is the only rank this flip asks to
ACQUIRE KV, because it gets half the token axis while holding 7 of 16
attention layers.

FOR THE FUNDING PATH: a rank that receives nothing is being taxed for bytes
that never move. On the shipping cut that is ranks 1 and 2.

WHY A FROZEN VECTOR CANNOT BE CARRIED ACROSS CUTS. `received_r` moves in WHOLE
LAYER steps. Move one attention layer off rank 2 and it starts paying a full
cell it never paid; move one off rank 0 and its slope RISES, because it still
needs half the token axis and now holds less of it. The planted test
`TheFrozenTripleMispricesAMovedCut` is that failure: at attention [7,6,3] the
frozen triple underprices rank 2 by a whole KV cell while the derivation
tracks.

WHAT THIS DELIBERATELY DOES NOT DO, and the brief asked for the opposite, so
the reasoning is here rather than in a reply that scrolls away:
`SHIP_PIN.basis_per_row_bytes` (phase_flip_seam_reserve.py:217) is NOT the live
sizing input and is not touched. It is a frozen PROVENANCE record whose job is
to stay fixed so `test_arming_floor_funding_662.py:1198` can assert the live
values still equal it and go red when they drift. The sizing path reads the
per-boot MEASURED record instead (`per_row_bytes=float(rec["per_row_bytes"])`,
:687, logged as "MEASURED BY A PREVIOUS BOOT"). Overwriting :217 with a derived
vector would have deleted a drift detector while changing no live behaviour at
all. A test pins that the frozen triple is still the measured one.

WHERE THE DERIVED VECTOR DOES BELONG, named as the boundary rather than forced:
the COLD path (:670) returns a SeamReserve with `per_row_bytes = 0.0`, and the
consumers (:927, :1008) then take the floor-bound branch because `a * t_floor
<= F` holds trivially at a = 0 -- a cold boot prices the per-token seam at
NOTHING, which is the cold-overshoot that has OOMed the 3080s. Deriving there
is the real fix, and it is not a small change: `read_seam_reserve(server_args,
world_rank)` receives neither the KV cell size nor the per-stage attention
counts, so wiring it needs those threaded in. Left undone deliberately while
F4-r4 is mid-deploy rather than shipped unvalidated behind a flag nobody would
turn on.

TESTS. 10 green. Red-first is not the module-missing collection error: a
cut-INDEPENDENT implementation returning the frozen triple was substituted and
4 cases go red, including all three planted-failure cases. Covered:
behaviour-neutrality at the current cut and vector (rank 0 within 2 % of the
frozen basis, tolerance named from the 1.4 % measurement error; the full vector
matching once the baselines are supplied); the two ranks that receive nothing
predicting exactly zero; the moved cut making a previously free rank pay a full
cell; the frozen triple underpricing it; moving a layer onto rank 0 raising its
slope by exactly one cell; three refusals; and the pin left untouched.

test/registered/unit/managers: 2029 passed, 4 failed -- all four pre-existing
on this base (this commit adds two NEW files and modifies nothing, per git
status). Two of them, test_collective_family_siblings_610.py
PrefillAdmissionBudgetTest, are in the admission-budget family and are flagged
separately. Hermetic (CUDA_VISIBLE_DEVICES=""), no boot, no live flag touched.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…the shared module

`pp_cut.seam_slope_bytes_per_token` now delegates to
`managers.seam_slope.derive_seam_slope_bytes_per_token` instead of carrying a
second implementation of the same formula.

Kept as its own commit so the previous one stays cherry-pickable into a deploy
line on its own: that commit adds a dependency-free module and its tests and
touches nothing else, and this one is a pure de-duplication that a cherry-pick
does not need.

WHY DE-DUPLICATING THIS PARTICULAR FORMULA IS NOT COSMETIC. The finding is that
the slope is NOT a constant -- it moves in whole-layer steps with the cut. Two
copies of a quantity whose whole point is that it re-derives are two things to
forget to re-derive.

planner 2561 passed, 2 failed (the same pre-existing
test_rejected_evidence_pins pair). Hermetic, no boot.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…chor it does not have

THE DEFECT. A cold ``SeamReserve`` has every measured field at zero,
``per_row_bytes`` included, so the per-token seam term is 0 on a first boot and
the pool is sized floor-only -- a pool whose cutover it cannot fund. The arming
FLOOR has been charged on a cold record since sgl-project#662-F4/A0; the SLOPE never was.

WHAT IS NOW DERIVED. sgl-project#685 established that a rank's per-row seam cost is the
full-attention layers it must RECEIVE at the cutover,
``max(0, share*n_total - held)``. Every term is rank-LOCAL and known at boot,
which is what makes it usable where nothing has been measured or exchanged:
the share from ``--phase-flip-tp-vector``, ``held`` from the attention modules
this runner actually built (``_lane_kv_bearing_layer_count``, family-neutral by
construction), the total from the model config, and the per-layer cell from the
configurator's own ``_cell_size`` divided by this rank's attention count. No
collective, no measurement, nothing transferred from another rig.
``derive_seam_slope_for_rank`` is the single-rank form; it agrees with the
whole-vector one, asserted.

A FALLBACK, NEVER AN OVERRIDE. Any non-cold provenance is returned untouched --
identity-asserted for both STORED and OVERRIDE. A measured record is a fact
about this rig and outranks a model of it.

ABSTAIN RATHER THAN APPROXIMATE. Missing flip vector, unparsable vector, rank
outside it, no attention layer on this runner, no single cell (the hybrid-SWA /
MiniMax case the existing cell read already refuses to invent for), no
full-attention layer in the config -- each leaves the slope at zero and says
which one. A rank that receives NOTHING is also left at zero: charging it a
per-token seam would reserve against a transfer that does not happen, which is
the over-reservation sgl-project#685 named.

AND HERE IS WHERE IT STOPS, WHICH IS THE HALF THAT MATTERS. The slope is
derived and announced, but the reserve stays INACTIVE and the budget path
still returns floor-only. ``SeamReserve.active`` requires ``id_space > 0``, and
the downstream solve anchors on that measurement point together with
``have_bytes`` (``t_floor = t_m + (have_m - A - F) // cell``). A derived slope
has no measurement point. Setting those two to make it active would fabricate
the anchor -- precisely the approximation this work is under orders not to
make, and the same watchdog-versus-live-input confusion as the
``basis_per_row_bytes`` catch one level down.

Consuming it needs an anchor-free branch. ``solve_pool_tokens`` IS that form
and takes no anchor, but it has NO live caller anywhere in the tree, so which
budget it is solved against (``budget_bytes`` net of the floor charge, or
something the sizer has not yet subtracted) is a design decision on the boot
path rather than a wiring detail. On a change that cannot be deployed or
measured from here, picking it would be guessing with the boot as the test.

SO WHAT THIS COMMIT ACTUALLY BUYS: the cold per-token seam is no longer
silently zero. It is computed, and a boot that would previously have been sized
floor-only in silence now prints the number, the layer count it came from and
the share it was derived against. That is the difference between an unfunded
cutover nobody can see coming and one named at boot. It is NOT yet the sizing
fix, and the tests say so rather than implying otherwise: a case pins that the
derived reserve does NOT become active, with the reason.

TESTS. Red-first is a genuine ImportError on the missing single-rank API, then
14 green. Covered: rank 0 receiving exactly one layer; the shedding and neutral
ranks receiving nothing; agreement with the whole-vector form; an out-of-range
rank refused; the cold reserve gaining the slope with a log naming it derived;
the consumption boundary pinned; STORED and OVERRIDE returned by identity; all
five abstentions; and two sizing cases showing a priced slope grants strictly
fewer tokens than a zero one -- so the derivation is not decoration once it can
be consumed.

test/registered/unit/managers: 2052 passed, 4 pre-existing failures (one
further ZMQ round-trip failure in the batch run passes 22/22 in isolation --
flaky, not this change). Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY: this
touches the boot sizing path and F4-r4 decides when it enters his line.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…e two commits to hold

MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test
evidence, whether it touches runtime, and whether an equivalent patch is
already on the serving line (verified with `git cherry`, not by message
matching -- four are: c41645c, ce60358, 658ea3a, 84b0171).

DRY RUN: clean. Merged into `integration/r2` -- the live line, since the
serving tree descends from its tip a73a0d8 -- in a throwaway worktree,
`--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts,
zero unmerged paths, so nothing was pre-resolved because nothing needed it.
Verified semantically as well as textually: on the MERGED tree, managers 2093
passed / 0 failed and planner 2574 passed / 2 failed, the two being the same
pre-existing test_rejected_evidence_pins pair that is already red on the base.

THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is
NOT an ancestor of integration/r2, so merging this branch drags in its whole
base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677
work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files,
+22731/-762. Approving this merge is approving that lineage, most of which is
not mine to vouch for. If only this work is wanted it must be cherry-picked
rather than merged.

NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four
already on serving -- merging them only reconciles integration with what is
already running; (2) the three sgl-project#624 test-only drift guards, which take managers
from 4 failures to 0 and should land early so the line stays green during
review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by
no serving path; (4) hold.

HOLD, two commits, both runtime and neither on the serving line:
  * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE
    format. Already queued to land on deploy WITH the W=8/W=4 probe after the
    sgl-project#694 soak verdict; merging it into integration first puts it in front of
    the soak meant to measure it.
  * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and
    abstention-guarded, but unsoaked, and the R' decision it waits on is not
    made.

Nothing else in the chain can move serving behaviour.

Docs only; no merge performed, no deploy, scratch worktree removed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… evaporates

THE DEFECT, MEASURED under lane load 2026-08-16: pp_to_tp abandoned on PP1
every ~3 s -- 18 abandons against 9 cutovers in 8 minutes, three ARM-UNFUNDED
in nine seconds -- while decode sat at 187.5 s against a 180 s budget. The
shortfall was identical every time, which is what gave it away:

    staging 733 MiB needed but only 691 MiB is spendable      (x8)

A runtime fluctuation does not repeat to the megabyte. That is a SIZING-time
constant.

WHERE IT CAME FROM. arming_draw_bytes excused the weights-arena tail from the
arming floor whenever the KV rung was RECORDED as able to pay it. On PP1: arena
814.9 MiB, rung_fund_bytes 953.8 MiB, so the draw fell 815 -> 138.9 and 815 MiB
stopped being permanently reserved. The pool was sized that much larger, and
the enlarged pool is what left 691 MiB spendable against a 733 MiB need.

WHY THE EXCUSE IS UNSOUND -- and it is not that the measurement was wrong.
rung_fund_bytes is a SINGLE-POINT reading. The rung's deliverable is
(current_rows - floor_rows) * bytes_per_row, and floor_rows tracks the LIVE
SET. At the abandon:

    current=473088 rows, floor=471983, slack=1105  ->  12.3 MiB deliverable

954 MiB promised, 12.3 MiB available. The funding collapses precisely when
occupancy is high, which is precisely when a seam is hard to fund. An excuse
granted at low fill is spent at high fill.

THE TWO CANDIDATE REPAIRS ARE ONE REPAIR. "Price the excuse against the worst
case at max fill" and "drop the excuse" give the same answer, because at max
fill current approaches floor_rows and the guaranteed deliverable is ZERO. So
the predicate is written in the general form -- excuse only what the rung is
GUARANTEED to deliver, via a new rung_guaranteed_bytes -- and a record that has
never measured a guarantee excuses nothing. The mechanism survives for a future
measurement instead of being deleted, and sgl-project#685's own condition ("conditional on
the recorded funding, never an unconditional removal") finally holds, because
the condition is now a bound the rung cannot fall below rather than a number it
was once observed at.

ON THE "THIRD PAYMENT" OBJECTION, which is the real tension and is answered
rather than ignored. TheArenaTailIsNotChargedTwice quotes
arming_floor_subtrahend_bytes: "Reserving it here as well would be the third
payment for one requirement." That reasoning holds only if payments one and two
actually discharge the requirement at flip time. Measured, they do not: the
rung covers 12.3 MiB of an 815 MiB tail at the fill where the seam runs. The
third payment is therefore not redundant -- it is the only one that holds when
it matters.

CORROBORATION I DID NOT CONSTRUCT. The restored floors are
    rank0 1728 (unchanged), rank1 1728 -> 1825, rank2 1728 -> 2467 MiB
which is exactly SHIP_PIN.basis_arming_floor_mib = (1728, 1825, 2467). The pin
already encodes the UN-EXCUSED floors, so the excuse had silently undercut the
pin its own regime was derived at; this returns the live floors to it.

COST, stated plainly: ~8.7k tokens of pool on rank1 and ~83k on rank2. rank0 is
untouched. That is the price of a seam that can arm, against a pool that is
larger and cannot.

TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
  test_arming_draw_excuse_696.py  6 passed (new, red-first: pre-fix it fails
                                  with "138.9 not >= 814.0", reproducing the
                                  PP1 arithmetic that produced the 42 MiB
                                  shortfall)
  test_arming_floor_funding_662   66 passed -- the four sgl-project#685 fixtures now state
                                  the condition as a GUARANTEE. Their intent is
                                  the conditional-relief mechanism, so only the
                                  field carrying "the rung can pay" changed; no
                                  expected value was flipped.
  managers + mem_cache            64 failed / 1908 passed, the same 64 as the
                                  pre-change baseline.

NOT DEPLOYED. Rides the next window with 8fb86ef, e21e87f, c738ef5
and the W-probe.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… dry run, and the vouch

The notes were written at 3c984ad and had gone stale in three ways.

Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697
(8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs
commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime
despite its ticket -- it adds a handover script and one test guard.

Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was
re-run rather than carried over. Still zero conflicts. Scope shrank with the
target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196.
Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new
test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in
test_rejected_evidence_pins.py and were re-verified as pre-existing by
aborting the merge and running that file on clean a157bf1 -- same 2 failed,
2 passed. The chain introduces no new failure. Because the target tip moved,
this was re-checked instead of quoting revision 1's finding.

Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so
revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both
lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list --
desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays
held on the unmade R' decision.

Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the
originals by git patch-id --stable -- f630947/c41645c8c9,
7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so
preferring the originals is free and the only real risk is double attribution.
1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is
diagnostic scaffolding and not mine to vouch for.

Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's
line, not this branch, and are listed only because they lift holds; sgl-project#696 is
what was DoSing the lanes.

Docs only. No merge performed; the dry-run worktree was removed.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…ribution, rank-2 root

1. RESTORE, NEVER REBUILD -- the pin (test_restore_never_rebuild_677.py).
A fence patches the BUILD entry points (weights_arena.allocate_arena,
weights_arena.pack_into_arena, torch.cuda.CUDAGraph) to raise, and the REAL
production mover PhaseFlipStacks.refill runs under it on both legs, including
the checksum-mismatch restore arm -- the one branch that touches the arena
twice and is likeliest to reach for a rebuild. arena_refill is deliberately
NOT fenced: it is the copy the flip exists to perform, and a pin that fenced
it would pass by forbidding the work. Can-fail arms: every entry point is
shown to actually raise, and a planted mover calling allocate_arena is caught.

Pinning production rather than a re-implemented loop is the point -- a
re-implementation keeps passing while production drifts (sgl-project#624).

2. PEAK-INSTANT ATTRIBUTION -- PhaseFlipRuntime._record_seam_peak, emitted on
the sgl-project#605 flight-recorder channel (not a new one: it already carries the torch
view, the NVML view and the boot id, and is append-only). Placed at
_staging_affordable, the instant the flip's demand is weighed against free
VRAM -- earlier the buffers do not exist, later the decision is already taken.

Carries staging_bytes, refill_destination_bytes, graph_workspace_bytes, the
reserve, driver free, allocator cached free, and a SIGNED unattributed_bytes.
Two choices, both pinned and both mutation-proven: unmeasured components are
None and never 0 (a zero reads as "costs nothing" -- the sgl-project#606 defaulted-
measurement defect), and the residual is signed, because a negative one means
the named terms OVER-count, a different defect that max(0, ...) would hide.
Guarded end to end: an instrument on the seam path may cost a line, never a
cutover.

3. THE RANK-2 ANOMALY -- the sgl-project#685 candidate is REFUTED. sgl-project#685 (0e50e48,
f1774d7) is an UnboundLocalError use-before-bind on a variable named
'cell' in the cold seam-pricing branch; it has no arena-tail content and no
1456 MiB figure exists anywhere in source or records.

The live candidate is arena GROWTH, from the sgl-project#690 image sizes:

  rank 0  PP 12619.6  TP 9614.9  growth    0.0  seam draw  909
  rank 1  PP  9014.0  TP 9614.9  growth  600.9  seam draw 1006
  rank 2  PP  7211.2  TP 9614.9  growth 2403.7  seam draw 1648

Rank 0's PP layout is the larger, so it never grows; rank 2 grows four times
what rank 1 does. The ordering matches the draws exactly and no other per-rank
term does. Stated as a CANDIDATE, not a finding: the excess over rank 0 is 16%
of the growth on rank 1 and 31% on rank 2, so growth explains the order and
not the size. Instrument 2 settles it without a dedicated experiment -- it
emits exactly this quantity at the peak instant.

FILED, NOT BUILT: TICKET_718_contiguous_destination_extent.md -- making a
peer's destination extent contiguous so a BAR1 write lands in place and the
receive buffer stops existing. The largest structural reduction available, and
the only one that removes a buffer rather than relocating it; it touches the
reshard wire format and the pool allocator together, so it belongs to a
deliberate design pass. The ticket names what must be decided, what must not
be assumed (it trades allocator freedom for staging bytes, and that freedom is
what keeps admission working under fragmentation), and makes a live reading
from instrument 2 its prerequisite.

Tests, hermetic (CUDA_VISIBLE_DEVICES="", no GPU, no serving contact):
  test_restore_never_rebuild_677.py    7 passed + 3 subtests
  test_seam_peak_attribution_677.py    8 passed
    MUTATION PROOF: null->0 and signed->floored fails 3 of 8
  FULL SUITE compared by failing NODE ID against base b786858, ANSI
  stripped (counts alone hid a real regression earlier in this branch):
    base 13 failed / 2181 passed;  now 13 failed / 2208 passed
    regressions: NONE. The 13 are pre-existing.
  ruff + codespell clean on all changed files.

Desk only. Live readings ride the later review boot.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…aim: the arena tails are measured and recorded

The sgl-project#702 repricing under the cold-spill doctrine found the number I said did
not exist. NOTE_677 section 8 stated "no 1456 MiB figure appears anywhere in
the records or source". WRONG: managers/phase_flip_seam_reserve.py,
record_path docstring, records the measured per-rank arena tails --
"1436 MiB on rank2 against 466 MiB on rank1 and 0 on rank0". 1436 rather than
1456, but plainly the figure the earlier candidate meant. I missed it by
grepping the docs tree and the ticket number instead of the module that owns
the quantity. The sgl-project#685 ticket attribution was still wrong (it is an
UnboundLocalError); the NUMBER was real.

It also settles, without a boot, what NOTE_677 left as a candidate. Against
the standing pool reduction per rank -- 704 / 801 / 1443 MiB, i.e.
arming_floor_subtrahend_bytes = floor - max(corridor law, already_reserved) --
the measured tail is 99.5% of rank 2's, 58% of rank 1's, and 0% of rank 0's.
Rank 2's floor excess IS its arena tail.

The repricing itself lands in the evidence tree (not a git repo):
/spinning/evidence-665-f1/NOTE_702_CUT_TABLE.md sections D1-D6, referenced
from an append-only block in PLAN_PERF_PIPELINE_2026-08-16.md (head-298
sha256 verified byte-identical, 81560e94b7628e6c). Headline: crediting the
standing reduction back, [31,17,16] recovers to 530,381 -- EXACTLY
[31,18,15]'s spill-funded figure, because once floors stop binding both bind
PP0 on the same bytes. The pool advantage that made [31,18,15] interesting was
an artefact of VRAM-resident floors, as the user said; with it gone [31,17,16]
wins outright on speed and is the only one of the two above the 14.1% noise
floor.

Build NOT undertaken and NOT small: nothing credits ON-DEMAND capacity against
the standing floor, and crediting the evict rung's PRICED capacity would be
pricing on intention rather than completion -- sgl-project#717's defect one layer up,
with an OOM at arm time as its failure mode. Ticket filed at D6 with one live
_record_seam_peak reading as prerequisite.

Desk only, no boot, no GPU, no serving contact.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026
…ke the verdict an observation

WINDOW-946FIX-0828 MEASURED A COMPENSATOR THAT WAS, FOR THE FIRST TIME IN THIS
FAMILY, PROVABLY ON THE DEFECT'S PATH AND STILL DID NOTHING. The sgl-project#946 escape
fired 8885 times (act_rid_hit) while PREFETCH RE-ISSUED stayed 0, cached>0
stayed 0, and the rid stayed frozen at told=8192. Root: `_prefetch_kvcache`
returned None on every one of its paths and the escape read that as success --
`prefetch(req); return "refetch"`.

ITS SUCCESS VALUE WAS NOT EVIDENCE THAT IT ACTED. Five earlier instances were
compensators placed off the path the failure takes; this one is a compensator
whose own report cannot distinguish acting from not acting.

SAME CLASS, NAMED SIBLINGS: sgl-project#872 (silent falsy paths), sgl-project#501 (the contract
promises what the code does not deliver), sgl-project#685 ("a silent precondition is
indistinguishable from dead code"), sgl-project#656/HANDOFF_678 4.0 (four early ABSTAIN
returns above the trace), sgl-project#767 (name the eaten branch).

THE VERDICT IS AN OBSERVATION, NOT A PROMISE. There are SIX silent exits
between the escape and registration -- two in `_prefetch_kvcache`, four inside
`prefetch_from_storage`. Enumerating them one at a time is how this family has
failed five times: a seventh added later would be silent again. So the verdict
is taken from the EFFECT -- did the operation appear in `ongoing_prefetch`
(unified_radix_cache.py:2996)? -- which covers every exit at once, including
ones nobody has written yet. Deep reasons come from the sgl-project#915 gate counters as a
per-call DELTA (anchor / too_short / rate_limited).

CONSEQUENCE, and it is where the old code leaked: the mark was cleared BEFORE
the attempt, so a DECLINED re-fetch spent it and the request never saw the
escape again -- invisible while every decline reported success. Now an OUTCOME
spends the mark, an attempt does not: issued clears it, a decline keeps it and
retries, bounded by REFETCH_DECLINE_CAP=3, then the terminator. Kein-Doppel-
Prefill unchanged: re-fetch preferred, told=0 the bounded last resort that
still names its discard.

sgl-project#915 WIRED AT LAST. `note_prefetch_gate` has recorded the decline reason on
every attempt since sgl-project#915 landed; `format_prefetch_gate` had ZERO callers, so it
was never reported -- PRESENT-BUT-UNWIRED for twelve days, printed by every boot
script and read as a note. Now logged on the census cadence.

CANDIDATE (iii) REFUTED BY MEASUREMENT, not by argument. The suspicion was the
sgl-project#915 host-tier capacity asymmetry (prefetch_capacity_limit = 0.5 *
mem_pool_host.size, cache_controller.py:841 -- NOT :729, a stale pointer in
sgl-project#915's own comment). The gate line is byte-identical across both stages of the
window (attempted=6, anchor/too_short/rate_limited ABSENT), so the escape never
reaches the gate at all: it returns at the ANCHOR exit above it. The escape's
precondition (`last_host_node.backuped`) is anti-correlated with the situation
the escape exists for.

TESTS, each module alone in a fresh process, 33-module blast radius derived from
the touched symbols, identical script both sides vs the 707f234 baseline:
  BEFORE  1017-11 = 1006 passed, 9 failed
  AFTER   1017 passed, 9 failed
Exactly ONE module differs: test_chunked_continuation_clamp_946.py 28 -> 39
(+11 new arms). Every other module matches its baseline EXACTLY, including
test_unified_radix_cache_unittest.py (642 passed / 860 skipped / 44 subtests,
both sides). The 9 failures are PRE-EXISTING at the pin and identical on both
sides: test_prefetch_progress_symmetry_580.py (7) and
test_collective_family_siblings_610.py (2). Extraction count probe: 9 == 9.

CAN-FAIL PROVEN IN BOTH DIRECTIONS, not asserted: a mutant returning "issued"
unconditionally is caught by the delivery arm; restoring the old consume-
before-attempt is caught by the mark arms. Four contract inversions recorded at
their sites -- stubs that returned None used to read as successful re-fetches,
which is the defect itself encoded as a test.

ruff (F401,F821,UP037) and ruff format clean. The two codespell hits in
scheduler.py are pre-existing at the pin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants