Repository navigation
fix: exclude logo png in gitignore - #785
Merged
Merged
Conversation
Contributor
Author
|
|
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
litmei
pushed a commit
to litmei/sglang_litmei
that referenced
this pull request
Jun 16, 2026
Add request queque eagle testcase
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
…ing the weights FOUNDATION SLICE, and it does not yet change any pool size. It removes the one obstacle that made the flip seam's arena tail unknowable at sizing time. WHY THIS TERM MATTERS. The tail is `max(0, layout_pp.total_bytes - layout_tp.total_bytes)` (phase_flip_boot.refill_high_water_bytes) and it binds the KV pool on this rig: 2215 MiB on rank 2 at cut (31,16,17) / vector (32,16,16), which raises that rank's arming floor from 1523 to 3226 MiB and holds the pool at 161378 tokens while the other two ranks could fund 564k and 704k. Today it can only be MEASURED, because `layout_tp` is built in phase_flip_boot:740 from a real TP-shaped worker -- long after the KV pool has been sized. That ORDERING, not the physics, is what forces the two-boot protocol and the cold/warm cliff sgl-project#782 documented: a first boot prices the tail at 0 and sizes 4.4x too large, and every later boot inherits its number from a predecessor. The code already knows the quantity is static; its own docstring says "the leg that must grow is a property of the LAYOUT SIZES". WHAT WAS IN THE WAY. plan_arena_layout needs only metadata -- shape, stride, dtype, storage_offset and the storage SIZE -- and every one of those is correct on a meta tensor, so a meta-device parameter set answers total_bytes exactly, with the same alignment and the same ordering, allocating nothing. Exactly one input is not correct: data_ptr() is 0 for every meta tensor, and that is what the function used to detect aliasing. Fed meta tensors it folded the whole model into ONE slot and returned a total wrong by orders of magnitude and structurally perfect -- a number that would have gone straight into the pool solve. FIX, two halves, both about not guessing: * `alias_of=` supplies the alias relation from the caller (the tie between lm_head and embed_tokens is a property of the checkpoint, known without loading it), and is validated: mismatched views, chains, absent canonicals and unknown names all refuse. * the inference path now REFUSES meta input, and refuses more than one non-empty tensor reporting data_ptr()==0, rather than returning the plausible wrong answer. Tests: test/registered/unit/mem_cache/test_meta_arena_layout_785.py, 9 pass. The acceptance property is byte-for-byte equality -- a meta layout and a real layout agree on total_bytes, slot names, offsets, nbytes and aliases -- since this number is subtracted from a memory budget and a rounding difference is a real difference. The guard is can-fail proven by pricing the bug it prevents rather than asserting it. `alias_of` is exercised on REAL tensors too and must agree with storage inference, so the derived number is the one the runtime later measures. 63 pass across the arena consumers (test_arena_high_water_631, test_weights_arena_exact_pin_695, test_layout_boundary_704) plus sgl-project#782/sgl-project#786. Pre-existing and NOT introduced here: 17 failures across test/registered/unit/model_executor/ and the black ratchet, identical in count and identity with this change reverted; server_args.py and mem_ledger/engine.py were already black-dirty at HEAD. New files are black and ruff clean. NOT CLAIMED: nothing consumes the derived tail yet. Wiring it into the sizing path ahead of the seam-record read, so a cold boot and a warm boot yield the same capacity, is the next slice.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
…riving it
THE TARGET THE WIRING HAS TO HIT, recorded before any derived number is
allowed near the pool solve.
The runtime prints both layout totals when rung 3 releases the tail. Boot
735-standT, cut (31,16,17) / attn (7,4,5) / vector (32,16,16):
PP0 released 72.0 MiB (TP layout needs 15925.8 of 16007.5 MiB)
PP2 released 2214.0 MiB (TP layout needs 8573.8 of 10789.2 MiB)
PP1 no release line -- no tail to release
Against the seam records that same boot wrote (arena_fixed_bytes): 82, 0 and
2215 MiB. So
arena_tail = max(0, layout_pp.total_bytes - layout_tp.total_bytes)
reproduces every measured rank to within 1 MiB. The term sgl-project#782 could only
inherit from a predecessor is a subtraction of two quantities the runtime
already computes.
WHY A TEST AND NOT A COMMENT. A census-calibrated estimate was tried first
and REJECTED on exactly these numbers. It reproduced the structure -- rank 2
the only positive tail, rank 1 exactly zero -- and was still unusable: 1.3%
low on the PP layout and 9% low on the TP layout, i.e. 628 MiB out on the
binding rank, about 63k tokens of pool. Getting the ranking right is not
evidence. The last test in this file keeps that bound explicit so a future
derivation cannot be accepted on structure alone.
Also records why the term is worth deriving: rank 2 carries the whole tail
and is the rank that binds. Its arming floor is corridor law + tail + margin
= 3226 MiB against 1523 on the other two, which holds the instance at 161378
tokens while ranks 0 and 1 could fund 564k and 704k.
13 pass with the meta-layout tests. Regression checked properly rather than
assumed: test/registered/unit/mem_cache/ reports 36 failed / 2296 passed with
the sgl-project#785 arena change reverted and 36 failed / 2305 passed with it, the +9
being the new tests -- so the failures are pre-existing and this introduces
none. Two other mem_cache tests do reference weights_arena
(test_pinned_budget_double_count_706, test_pinned_post_registry_550), which
is why the baseline was run rather than reasoned about.
NOT CLAIMED: still nothing consumes a derived tail. The wiring needs an exact
layout_tp at sizing time, and the hazard is now named -- producing TP-shaped
parameters means set_tp_partition_ratios, which mutates PROCESS-GLOBAL
partition ratios and would corrupt the live PP stack's own geometry unless
saved and restored around the meta build. That change cannot be proven
without a boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…hapes layers
Deriving the flip's arena tail on a first boot needs a TP-shaped model built at
KV-sizing time, when no TP stack exists yet -- the ordering is deliberate and
phase_flip_seam_reserve.py:72-91 states it: "neither exists until the TP stack
is built, which happens AFTER the PP pool is sized. There is no ordering that
makes them knowable in time, so they are not computed here at all."
Building such a probe means installing the flip vector for the duration and
handing the process back unchanged, which is what scoped_tp_partition_ratios is
for. But every production caller that shapes a real stack uses the process-
global set_tp_partition_ratios instead (phase_flip_boot.py:655), so nothing in
the tree demonstrated that a model built under the scope is sharded to the
scope's vector rather than to the process's installed plan. That was the one
unverified assumption in the derivation plan, and it is load-bearing: the only
discriminator that decides whether a plan applies is len(ratios) == tp_size, so
a build under a leaked plan does not raise -- it falls back to the even split
and loads the wrong units. A probe that inherited the PP plan would return a
plausible, wrong arena total, and the pool sized from it would be wrong in
exactly the way the two-boot cliff already is.
The chain that shapes a layer is layer construction -> tp_partition_size ->
get_tp_partition_ratios(family) -> overlay. This pins the whole chain below the
layer, on the three functions construction actually calls: the overlay is read
by tp_partition_sizes, tp_partition_size and tp_plan_active, it beats an
installed process plan, family vectors ride along, it restores after an
exception raised inside the scope, it nests, and it does not leak into another
thread -- the property that makes a probe safe to run inside a live boot while
the scheduler's own thread keeps reading the installed plan.
Includes the can-fail proof for the silent failure the scope's docstring names:
with a mismatched installed plan the split comes back even and plausible rather
than raising, and the scope is what repairs it.
The assumption holds, so no process-global fallback is needed. What is NOT
proven here is deliberate: no real model is constructed, because that needs a
checkpoint and a device and belongs in the integration test that checks the
derived tail against the measured {0: 82, 1: 0, 2: 2215} to 1 MiB.
12 tests, red first (the suite was written against functions whose overlay
behaviour was unverified). ruff check/format clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…of a previous boot The flip seam's arena tail is max(0, layout_pp - layout_tp) and it is a post in the KV pool solve, but layout_tp is built from a real TP-shaped worker AFTER the pool is sized (scheduler.py:1421 -> :1425 -> :1442). phase_flip_seam_reserve states the consequence verbatim: "there is no ordering that makes them knowable in time, so they are not computed here at all". The tail has therefore been carried between boots in an on-disk seam record that defaults to 0 when cold, which is the two-boot cliff behind the sgl-project#678 OOM class: a first boot prices rank 2 at 1523 MiB instead of 3226 and dies when the NEXTN draft weights land. A layout needs shape, stride, dtype, storage_offset and the storage size, and all five are correct on a meta tensor. So the TP layout is answerable at sizing time without allocating a byte. The one input that is NOT correct is data_ptr(), which is 0 on meta and is exactly what plan_arena_layout uses to detect aliasing -- fed meta tensors it folds a model into one slot and returns a total that is orders of magnitude wrong and structurally perfect. storage_alias_relation supplies the relation from the StorageImpl handle, which is a valid identity on meta and agrees with data_ptr grouping on real tensors. Strict on this path only: an installed vector whose length does not match the group size falls back to an EVEN split rather than raising, and that fallback is load-bearing across the tree (a blanket refusal fails 41 distributed tests). It is not touched -- it is simply not tolerated where the answer is charged into a memory budget. DESK GATE, three gloo ranks, CUDA_VISIBLE_DEVICES="", real checkpoint config: the derivation reproduces the boot's own logged layout_tp on every rank -- 15925.8 / 8573.78 / 8573.78 MiB against 15925.80 / 8573.78 / 8573.78 (boot_735_default791b.log:952/962/975). Zero MiB error, 0.1 s per rank. Can-fail proof: vector 22,21,21 moves the totals to 11921.06 / 10576.16 / 10576.16, far outside the 1 MiB bar. The alias detector has its own can-fail proof, because its healthy answer on this architecture is the empty relation -- which is also what a broken detector returns. This commit derives and instruments only; no sizing path consumes the number yet, so no boot changes size. The deciding gate remains the boot instrument checking the derivation against the same boot's own totals. Tests: 67 passed (the two new suites plus test_arena_tail_identity_785, test_meta_arena_layout_785, test_scoped_partition_overlay_785, test_corridor_arming_credit_784). ruff + codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…that measured it The derivation from the previous commit is checked here where the question actually lives. A desk run has to pin one GPU capability, because the compressed-tensors linear scheme is chosen from get_device_capability() at construction time (compressed_tensors.py:530) -- and this rig is mixed, rank 0 on sm120 and ranks 1/2 on sm86. So a desk agreement is evidence about the rig it did not run on. Two halves, one boot. At the sizing site, where the seam record that stands in for the tail is about to be spent, _instrument_arena_tail_derivation measures layout_pp from the LIVE PP model (it is loaded by then, so deriving it again would only add a way to be wrong about something this boot already knows), derives layout_tp from a meta TP model, parks the pair on the runner and announces it. Then in phase_flip_boot, at the first moment the MEASURED layout_tp exists -- immediately after plan_arena_layout(tp_named) -- the two are graded at a 1 MiB bar against each other. Both numbers come from one boot, so this is an identity check rather than a comparison against a remembered rig. The draft worker and the flip's own TP stack both reach the sizing path; neither owns the PP arena, and the TP stack running this would derive a TP layout from inside a TP layout. Both are excluded by name. Nothing consumes the derived number yet, so no boot changes size. That is the point of the ordering: wiring it into the budget before this grade exists would be sizing a pool against an unchecked derivation, which is exactly what the rejected census estimate did -- 628 MiB out on the binding rank, about 63k tokens of pool, with the structure right. A failed derivation must not cost a boot that would otherwise have started, so it degrades to a warning naming the exception and leaves the record-based path untouched. An instrument that can abort a boot has stopped being an instrument. Tests: 11 passed in the extended suite, including the can-fail set -- a 628 MiB miss (the census route's real error) FAILS, and two errors that CANCEL in the tail still fail, because a right tail from wrong layouts stops being right the moment the cut or the vector moves. ruff clean on the touched files; the unused pack_into_arena import ruff reports in phase_flip_boot pre-dates this branch and is left alone.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
The derivation is graded PASS on all three ranks at error 0.00 MiB against the same boot's measured totals (boot_735_tail785, commit fa7d387), so it is now allowed to reach the solve. Three sites, all on the cold path only: - _maybe_price_cold_arena_tail sets arena_fixed_bytes on a COLD record from this boot's own layouts. A non-cold record, or a cold one that already carries a tail, is returned untouched -- this adds a number where there was none and never replaces a measurement. - the arming floor read the draw only when reserve.active, and active requires an id-space ANCHOR a cold record cannot have. The anchor is genuinely absent; the tail is not, once derived. arming_draw_bytes already returns exactly the arena term when no per-leg measurement exists, so the condition widens rather than the arithmetic changing. This is what separates a cold arming floor of 1523 MiB from the real 3226 on rank 2. - the cold solve passed a literal 0 for solve_pool_tokens' arena_fixed_bytes. Its own log said "the fixed and arena terms are NOT charged"; the arena half of that sentence is no longer true and the log now says which term is charged and which is still missing. The warm path does not move: a measured record already carries the tail, and the derivation agrees with it exactly, so nothing about the currently serving configuration changes size. What changes is the FIRST boot of any configuration, which previously priced rank 2's tail at zero, solved 755649 tokens and OOMed when the NEXTN draft weights landed. That is the two-boot cliff, and it is the reason a configuration change silently sized against the previous configuration. Tests: 7 new, including the can-fail guards -- a rank whose PP layout is the smaller one must be charged nothing rather than a negative tail, and the charge must be the SUBTRACTION rather than layout_pp itself (10789 instead of 2215 MiB would size a rank to almost nothing while looking correctly priced). 46 passed across the sgl-project#784/sgl-project#785 family. ruff clean. The 51 failures in the mem_cache+managers sweep are pre-existing: the four named files run 12 failed / 10 passed both on this branch and on a detached worktree at the merge base 332f083.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
Every term that decides a rank's share of the PP-phase pool is a function of the cut: its weight layout, its arming floor (which carries the arena tail), and its per-token cell (because full attention sits on every 4th layer, so a contiguous cut DETERMINES how many attention layers a stage holds). Until sgl-project#785 the middle term could only be learned by booting, which is why cut selection was a hardware experiment. It is now a desk computation and a boot is what checks it. No rig constants live in the module. Budgets, per-layer weight mass and the residual overhead are fields on CutModel, calibrated from a boot's own instruments; the numbers sit in the test as a fixture. A solver carrying one machine's fitted vector is exactly the transfer this corpus forbids. The attention split is implied, not chosen. Treating it as a free parameter invented cuts that cannot exist -- a 16-layer stage holding 3 attention layers when every 4th layer is attention -- and the first version of this search did exactly that. The test pins that the shipped cut 31,16,17 implies the 7,4,5 the boot declares, and that 32,17,15 implies the 8,4,4 boot 735-full785 was launched with. The arming floor is not 1523 + tail either. Solved from the boot's own three floors against its three tails: max(1523, 1011 + tail), i.e. corridor law 819 plus arming margin 192 on top of the tail. Rank 2 checks: 819 + 2215.4 + 192 = 3226.4. Calibrated on boot 735-tail785 the model reproduces that boot's KV available column to within 1 MiB on every rank and its pool to within 0.05% of 161378. It also carries a BOUND rather than only a search result: sum(layout_pp) exceeds sum(layout_tp) by ~1732 MiB and is essentially cut-independent, so no cut can zero every arena tail. A cut relocates the tail onto a rank with headroom; it does not abolish it. The residual term is an affine fit over layer and attention counts with three unknowns fitted to three ranks, so it is calibration and not validation, and the module says so. It is also calibrated on a WARM boot and must not be read as a prediction for a cold one -- the cold solve is a different branch that undercharges. Tests: 14 new, all passing. Regression control: mem_cache + managers runs 51 failed / 4314 passed on this branch and 51 failed / 4268 passed on a detached worktree at the merge base 332f083 -- same failures, 46 more passing. ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
PRIOR-ART GATE FAILED, BY ME. python/sglang/srt/planner/pp_cut.py already solves the pipeline cut -- 2635 lines, per-family attention/linear decoupling, a measured-rate makespan objective and a per-rank memory feasibility constraint -- and pp_cut_calibration.py sits beside it. I wrote a parallel 300-line solver without reading srt/planner/ first. A second cut solver in the tree is exactly the debt this corpus keeps paying off, and the right move is to feed the existing one. The finding that came out of writing it is kept and does not need the duplicate to stand: the arming floor is max(1523, 1011 + tail) rather than 1523 + tail, and the attention split is IMPLIED by a contiguous cut rather than chosen -- which pp_cut.py already states at length in its module docstring (the "boundary can only fall on a multiple of 4" analysis), further confirming this was the module to extend. More to the point, pp_cut.py:412-432 makes RankResources.draft_residency_mib MANDATORY and refuses to solve without it. That is the input sgl-project#785's meta probe can now supply -- 1441.14 / 1352.35 / 1352.35 MiB on this rig, derived in 0.01 s without loading a weight -- and supplying it is the useful work, not re-deriving the solver around it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…as to fund it build_flip_draft_worker (phase_flip_boot.py:614) loads a whole second model -- the NEXTN/MTP draft -- at scheduler.py:1442. The KV pool was solved at :1425. So the budget never knew a 1.4 GiB model was still coming, and the only thing that made that survivable was a pool small enough to leave accidental slack: on the shipped cut the binding rank held 7027 MiB free and the draft fit by luck. Boot 735-full785 removed the luck. It solved 764512 tokens and then died in ct_embedding.create_weights asking for 406 MiB with 227.75 MiB free on rank 0, inside build_flip_draft_worker. Same shape as the sgl-project#678 OOM and the same cause as the arena tail: a post the solve cannot see because it is created later. Derived the same way the tail is -- a meta-device model under the flip scopes, 0.01 s, no weights allocated. On this rig 1441.14 / 1352.35 / 1352.35 MiB. Subtracted from the budget rather than folded into the seam reserve: it is resident weight, not a seam cost, and a term charged under the wrong name is a term the next reader misprices. This also supplies an input the planner already demands and nothing could previously produce: planner/pp_cut.py:412-432 makes RankResources.draft_residency_mib MANDATORY and refuses to solve a cut without it. Tests: 18 passed across the cold-pricing and alias suites. ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
The draft worker IS the draft, and the flip TP stack reaches the same sizing path after the primary has already paid; either charging twice would book a 1.4 GiB model two or three times. A configuration with no speculation has no second model to fund at all, so charging one there would shrink the pool for nothing -- the mirror of the defect the term fixes. The charge is also cached, because the solve consults the budget more than once and a meta build per call would put a model construction inside a sizing loop. Tests: 11 passed in the cold-pricing suite.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…t, and ban the pool pin TWO BOOTS, TWO OOMs, ONE SHAPE. Everything build_phase_flip_tp_stack creates is allocated at scheduler.py:1442, after the pool is solved at :1425: the arena above the PP weights, the flip draft's weights, its attention-backend workspaces, its CUDA graphs. Charging them one at a time moved the death rather than preventing it -- 735-full785 died in ct_embedding.create_weights, 735-draft785 died one stage later in the flashinfer prefill backend with the card completely full. So they are charged as ONE term. Two of the three components are DERIVED for this boot and therefore track the cut and the vector: the arena exceeds the PP weights by max(0, layout_tp - layout_pp), and the draft weights come from the same meta probe. Only the remainder is a measured constant, and it is measured on the boot that LIVED through the build -- 735-tail785 -- as the net VRAM consumed between "Memory pool end" and the end of the stack build, minus the two derived terms: 2294 / 1188 / 625 MiB. The TP stack's own KV pools are charged nothing, and that is checked rather than assumed: the boot logs "Memory pool end" nine times and the second and third readings on each rank equal the first (PP0 7.69 / 7.69), so those pools REBIND the existing allocation. Charging them would have been charging the pool against itself. POOL PIN BANNED AS A CLASS (user order 2026-08-21). --max-total-tokens was never wrong because of its value: it sat inert for weeks while the profiled capacity was 161378, and the moment sgl-project#785 raised the profile the same untouched flag capped a 764512-token pool at 430000. The guard is a date, not an exemption list, because 45 of this host's 68 argv recipes still carry it -- the cleanup covered the boot-chain scripts, not the recipe archive, and a list naming 45 files is a formality rather than a lock. What must be impossible is REINTRODUCING it, so every recipe written from the order onward must be clean and a sed-derived chain copying an old recipe into a new file is exactly what that catches. The guard has a can-fail proof. argv_cut321715.txt, which I wrote today from the pre-order default, was caught by it and deleted; its evidence lives in boot_735_cut321715.log. Tests: 4 in the new guard suite, 22 across the cold-pricing, alias and corridor suites. ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
… rung that already names it
PRIOR-ART GATE FIRST, and it changed the design twice.
(1) The spill ladder ALREADY has this rung. phase_flip_spill.py:173
DEPTH_DRAFT_GRAPHS = 4, CLI name "draft+graphs", with
IMPLEMENTED_DEPTH = DEPTH_ARENA_TAIL -- so rung 4 parses and is refused,
by design, rather than silently under-delivering. No new flag and no new
concept was needed; the work is wiring the rung that exists.
(2) The park/restore machinery exists too, and its own docstring names
this exact case: short_term_offload_register.py:1269 GraphFamilyRegister
("a whole LAYOUT/ALGO family's capture state, pre-captured at boot and
parked while its family is inactive"), with offload_family/onload_family,
over the sgl-project#93 VMM route. The credit side exists as well --
kv_backing_relief / recover_kv_backing, and the mixin already carries
paused rung graph bytes as registry posts feeding the KV budget
(model_runner_kv_cache_mixin.py:1488-1603). Nothing here needs inventing.
(3) Rung 4 carries a RECORDED REFUSAL that must not be overturned
silently: it "buys a phase-local spill of something the next TP phase
must re-capture, which is a different and much worse trade than rung 2".
That reasoning is about a RUNTIME per-flip spill where every flip pays a
recapture. What is proposed is a BOOT-time deferral paid once. The
arithmetic that changed the trade is measured, not argued: on boot
735-bal785 rank 0 binds the pool at 525462 tokens and needs 2242 MiB more
to fund the 669k reference, against a residual on that same rank of 2294
MiB. Same number. The refusal is therefore LEFT STANDING in
phase_flip_spill until the park/restore path is wired and exercised in a
real flip cycle; this commit only makes the sizer able to express the
credit, so the next step is wiring rather than arithmetic.
The credit is bounded by what was MEASURED, not by what is needed -- a
term that grows to close whatever gap it is shown is not a measurement,
and that has its own can-fail test. The draft WEIGHTS are never credited:
they are needed in the phase the draft actually runs in, and crediting
them would be the sgl-project#678 OOM one rung deeper.
Also: the pool-pin guard now distinguishes a recipe from prose about a
recipe. argv_hc_interval8192.HOSTRAM-NOTES.txt sits under the same glob,
was touched after the order, and quotes the flag it documents.
Tests: 6 new for the credit, 4 for the guard. mem_cache runs 8 failed /
1441 passed on this branch and 8 failed / 1408 passed at the merge base
332f083 -- same failures, 33 more passing. ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…to touch Formatting only, no behaviour change. These four files carry pre-existing formatting debt that test_black_ratchet_656 already reports, and the rung-4 wiring lands in the middle of it. Committing the reformat separately keeps the next commit's diff readable as the change it actually is: a reviewer can see that black moved no logic, because the formatting is already in the parent.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…e cutover that uses them The flip's TP stack is built at boot, but the PP phase runs on the boot stack and never executes a TP or draft forward. Two of that build's posts are therefore PHASE-COLD -- the attention-backend workspaces and the decode CUDA graphs -- and on rank 0 of this rig they measure 2294 MiB, which is the same number as the pool's shortfall against the 669k plain-TP reference. The sizer learned to express that credit in the parent commit; nothing passed it. This passes it, and pays for it. ONE PREDICATE DRIVES BOTH SIDES. phase_flip_spill.cold_stack_deferred is asked by the KV sizer before it solves the pool and by the boot before it decides whether to build. They cannot disagree, which matters because the direction that disagrees silently -- credit taken, deferral not performed -- sizes a pool into memory the same boot then allocates. That is the sgl-project#678 OOM class, and no test of either side alone can catch it. THE CARRIER PIN TRAVELS WITH THE CAPTURE. Rung 2's pin asserts that the draft parameters still lie inside the spill carrier at capture time. Deferring the capture while leaving the pin at the boot site would have checked an address set that nothing had baked yet -- a green assertion about nothing. It moved into build_cold_stack_posts, and it deliberately does not latch the built-flag, so a refusal cannot let a retry skip the capture entirely. THE RESTORE IS PRICED WHERE IT IS PERFORMED. The build happens inside _cutover, past the point of no return -- the same shape that took all three ranks down on 2026-08-09 when rung 2's re-commit was unpriced. _cold_stack_restore_bytes charges it into the pp->tp affordability verdict, SUMMED with the draft restore rather than max()'d with it, because the cutover's own ordering makes them coexist: the weight pages go back first precisely so the capture has addresses to bake. Both ends use the same measured constant, so the pool cannot be sized against one number and funded against another. IT IS A ONE-TIME MOVE, NOT A PER-FLIP SPILL, and that is what overturns the recorded refusal honestly. That refusal objected to a per-flip RE-CAPTURE, measured at 41% of decode throughput. A boot deferral pays no re-capture: the builder latches, so the second and every later pp->tp leg is a no-op. The GraphFamilyRegister park/onload path would be a different rung; it has no production caller today and none is added here. THE RUNG IS STILL REFUSED. IMPLEMENTED_DEPTH stays at DEPTH_ARENA_TAIL, so cold_stack_deferred answers False for every configuration reachable without the new SGLANG_PHASE_FLIP_SPILL_DEPTH_EXPERIMENTAL hatch, and the instance serving on --phase-flip-spill-depth arena is bit-identical under this commit. The hatch exists only to break the deadlock between "promote after metal" and "cannot reach it while unpromoted"; it warns on every resolution and is an env rather than a server arg, so an operator sweeping the depth flag keeps hitting the refusal. Promotion follows a real flip cycle, not this commit. THE CREDIT IS NOT YET SAFE TO SPEND, and this commit does not spend it. Under the first real agent load the 525462-token form OOM'd on rank 0 inside the first tp_to_pp flip: F.linear in the GDN in_proj_qkvz asked for 256 MiB with 131.69 MiB free. So the decode forward's transient is UNPRICED in today's sizing, and handing the deferral's 2294 MiB straight to the pool would reproduce that OOM with a larger pool behind it. The transient is a separate sizing post and lands separately; until it does, the hatch is the only way to reach this rung and the acceptance gate for it runs under load. Tests: 140 passed -- 33 new red-first tests across two files (predicate, interlock, hatch, sizer threading, builder contract, carrier-pin travel, idempotency, cutover restore, seam pricing), plus the existing phase-flip boot and runtime suites. Both new files carry CAN-FAIL guards aimed at the rung the rig actually serves on, and structural tests so a stray direct init_cuda_graphs or an unconsulted priced term fails. Both are registered with register_cpu_ci; the CI runner executes `python3 <file> -f`, so function-style pytest files were never being collected. Scoped regression over the 119 test files that reference any touched module: 2153 passed. A full failure-set comparison against the parent commit in a separate baseline worktree found 31 failures on the parent and, initially, 32 here. The one extra was mine and is fixed in this commit: test_phase_flip_arena_tail_631 pins the SOURCE TEXT of the _staging_bytes return, so that the arena tail stays ADDED to the max() and never becomes a third argument of it -- a difference one measured corridor breach wide and invisible in a value. Reformatting had collapsed the return onto one line and broken the pin without touching the arithmetic; the multi-line shape is restored and commented so the next reformat does not silently do it again. With that, the two failure sets are identical in both directions.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
… tensor an OOM landed on
gdn_prefill_scratch_mib began its accounting at the conv output -- one
allocation too late. in_proj_qkvz produces a packed [T, 2*key_dim + 2*value_dim]
tensor whose z half appears in no other term, and in_proj_ba adds 2*Hv columns
beside it. Neither was counted.
THE SPECIMEN. On 2026-08-21 an instance died at exactly that line under the
first real agent load, inside the first tp_to_pp flip:
qwen3_5.py:625 projected_states_qkvz, _ = self.in_proj_qkvz(hidden_states)
linear.py:835 return F.linear(x, layer.weight, bias)
torch.OutOfMemoryError: Tried to allocate 256.00 MiB.
GPU 0 total 31.34 GiB of which 131.69 MiB is free.
256 MiB is 8192 * 16384 * 2 exactly: --chunked-prefill-size 8192 tokens, and
16384 = 2*(16*128) + 2*(48*128) columns at --tp-size 1, where the GDN layer is
unsharded and carries full width. The number the estimate omitted is the number
that killed the instance.
THIS IS NOT A COSMETIC UNDER-REPORT. The prefill-admission guard derives its
threshold from this function, so the omission set the gate that was supposed to
prevent the allocation. The log shows the gate doing its arithmetic and then
letting the chunk through anyway:
corridor law 1024 MiB (prefill admission, 8192 tokens)
corridor shortfall of 981 MiB for rank 0 (was 953 MiB)
corridor cannot be restored ahead of this chunk
1024 + 257 = 1281, and the corrected estimate is 1270 MiB at those settings.
The guard was holding a threshold roughly one in_proj tensor too low. Making
the guard ENFORCING rather than advisory is a separate change and is not in
this commit; this one makes the number it enforces on honest.
THREE CONSUMERS SEE THE CORRECTED FIGURE, and none of them is the KV sizer:
the pinned-reserve diagnostic (server_args.py:13222), the planner's split
advisory (planner/split_probe.py:404) and the ledger term
(mem_ledger/engine.py:433). All three were reasoning from the low number. The
KV sizer charges no forward-activation term at all -- see the declining gates
at server_args.py:13168-13173, 13178-13183, 12298-12300 and
mem_ledger/engine.py:1265-1266 -- which is a real gap, deliberately left for a
separate change rather than folded in here.
Tests: 1236 passed in the server_args and mem_ledger suites. New file
test_gdn_in_proj_transient_785.py pins the specimen (the 257.5 MiB term at the
crash's own settings, the total in 1200-1350, chunk-proportional scaling, and
the per-rank share) and carries a CAN-FAIL guard asserting the estimate now
exceeds the 1024 MiB threshold that let the OOM through -- an estimate still at
or under it would leave the gate exactly as permissive as before. Two existing
pins moved with the formula and say why: the exact-formula assertion in
test_uneven_tp_args.py, and the diagnostic's per-layer figure, 63 -> 79 MiB.
The one remaining failure in those suites, test_r1_private_constant_gate_584,
reproduces on the parent commit in a baseline worktree.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…ime one The rung-4 deferral moves the flip TP stack's attention backends and decode CUDA graphs from boot to the first pp->tp cutover. The boot-time build runs inside phase_flip_tp_scope; the deferred one did not, and that asymmetry is the whole defect. Attention backends read their geometry FRESH from get_parallel() contextvars at construction (triton_backend.py:731-757), while the KV pool's row width is a value baked onto the ModelRunner at __init__. Unscoped, the cutover build saw the ambient primary topology (pp=3, tp=1, dcp disabled) instead of the flip TP geometry the pool was sized under, so uneven_dcp came out False, the forward skipped _dcp_write_gather, and the store kernel was handed raw per-rank rows against a pool baked for 4 replicated heads. On metal that is: store_cache rejected: k(24, 2, 256) row_dim=1024 -> k_rows=12, k_cache(201377, 4, 256), indices(24,) 24 tokens silently refolded into 12 rows (6 on the 1-head ranks), with no error until the capture died. Four changes: A. restore_deferred_cold_stack re-enters phase_flip_tp_scope, deriving n and world_rank exactly as the boot path does. B. phase_flip_tp_scope now SAVES and RESTORES the prior TP routing instead of forcing it off on exit. Without this, A would be worse than the bug: the cutover has already armed the routing and the TP phase continues after the deferred build returns, so a hard False would have routed every later collective back onto the primary tp=1 groups -- the silent no-op all-reduce that set_phase_flip_tp_active's own docstring exists to prevent. The boot path is unchanged; its prior value is always False. C. A named guard before init_attention_backends refuses a pool/ambient geometry mismatch with a message naming the pool head_num and row_dim, the ambient attn_dcp_size and attn_tp_size, and the uneven_dcp verdict. A silent refold becomes a loud error. It skips rather than raises on any pool shape it cannot interpret, so it stays inert on the healthy path. D. assert_row_schema_compatible's docstring claimed weighted DCP always replicates KV heads. It does not: attn_kv_replicated turns replication OFF when kv_heads >= tp_size, which is this rig (4 KV heads over tp=3 -> per- rank [2,1,1]). The check passed only because both pools happened to report the same head_num. Docstring corrected and the error now names the configuration actually observed. The check itself is unchanged. Tests, hermetic and CPU-only: test/registered/unit/managers/test_cold_stack_boot_deferral_785.py Red before the fix: 4 failed, 20 passed -- the new A/B/B/C tests. Green after: 24 passed. Together with test/registered/scheduler/test_phase_flip_boot.py: 55 passed, verified independently of the implementing run. Includes a can-fail proof for the part-C guard in both directions: matching geometry inside a scope builds cleanly, mismatching geometry raises before init_attention_backends is reached. NOT yet verified on hardware -- no multi-rank pp->tp cutover has executed this path on metal. That is the next GPU window.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…by hand --hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires an explicit --hicache-size. That refusal is only worth having if the number it demands comes from somewhere; a hand-picked --hicache-size is exactly the pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785) exists to prevent. planner/hicache_staging.py derives it. Two consumers, and sizing for one of them silently breaks the other: * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the moment it is handed to the storage backend until the write acks, so the resident set is drain rate x that residency, widened by a burst margin (write-through arrives in bursts, the drain is steady). * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch takes its destination slot from mem_pool_host.alloc() BEFORE the storage read is issued (hiradix_cache.py's _prefetch path). Sizing from the write side alone yields a tier that is correct and quietly serialises prefetch -- invisible to every capacity metric, visible only as latency. The tier takes the larger, rounded UP: rounding down would emit a tier smaller than the derivation justifying it. NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner probes rather than carrying a measured vector from one box to another, so nothing in this module encodes what this rig happens to do (~0.5 GB/s on its ZFS-backed file tier); that figure appears only as a test input. WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster than the backend drains them, NO finite size holds -- residency is set by the drain, so a faster producer grows the in-flight set without bound. sustainable() answers that question so a caller can refuse instead of emitting a number that cannot be correct. The remedy there is backpressure at the producer, a runtime mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer rather than blocking, trading "a bounded spike for an unbounded latency" -- which is right for a prefetch worker and wrong for a write path under a RAM budget, where re-inflating the pinned footprint is the failure the budget exists to prevent. That ring is the next posten, not this one. fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like every other pinned producer, so a tier that does not fit is refused where it is derived rather than discovered at allocation. Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and does not depend on it, but the supporting argument is unrecoverable. Recorded rather than reconstructed: rebuilding it from four citation fragments would read as primary evidence while being inference, and the next reader would cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed until the target exists. Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases, hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended tests: read consumer dropped (write-only sizing) -> 1 failed rounds down instead of up -> 1 failed zero-drain guard removed -> 2 failed burst-margin floor removed -> 1 failed sustainability predicate inverted -> 2 failed Source green again at 19 passed; 28 passed together with the role gate. Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of dd6967c with these files removed -- same test, same assertion. No pre-existing test changed status. ruff/black/codespell clean. Not wired to a probe yet, and named so it is not assumed: nothing measures the drain rate automatically, so a staging profile still needs the number supplied. Measuring it needs load against a running server, which belongs to another strand's boot window right now.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…ount the refusals A staging host tier is small on purpose, and nothing at runtime bounded what write-through put into it. The tier itself did, by running out -- which is not a bound but two silent failures. The read consumer starves: a prefetch takes its landing slot from mem_pool_host.alloc() BEFORE issuing the storage read, so a tier full of undrained write-through pages makes it evict, retry, truncate and abandon; a correct miss, invisible to every capacity metric and visible only as latency. And the write refusal itself is silent: write_backup learns about exhaustion by reading a None back out of an allocation that already happened, then -- with no rank-uniform floor published -- takes evict_host(), a RANK-LOCAL tree edit, which is the sgl-project#645 divergence. NOT sgl-project#720's ReadBufferPool. Its acquire() (read_buffer_pool.py:99-105) answers exhaustion by allocating a fresh UNCOUNTED pinned buffer, deliberately: "stalling the prefetch worker to save memory would trade a bounded spike for an unbounded latency". Right for a prefetch worker, and precisely wrong for a write path under a RAM budget -- the footprint that overflow re-inflates is the one --hicache-host-role staging exists to cap. StagingWriteRing never allocates; its only outcomes are admitted and refused-and-counted. THE RESIDENCY IS TWO-PHASE, and only one phase is refusable. write_backup allocates the host slots; write_backup_storage calls node.protect_host() and the protection drops only at the backup ack (ongoing_backup.pop -> release_host). The bandwidth-delay product that planner/hicache_staging sizes the tier from is the SECOND phase, so a ring spanning only the device->host copy would claim a bound it does not have. Phase one is admit(), taken before any allocation, where a refusal is actionable. Phase two is occupy(), which cannot be refused -- the page is already resident, so refusing would free nothing and merely hide the drain queue from the next admission. Phase two is keyed by the STORAGE OPERATION, not the node, and that is not cosmetic: in UnifiedRadixCache one write-through ack fans out into several storage backups after a node split, each with its own id, and lock_node is by then no longer among publish_nodes. A node-keyed charge would be stranded there -- and a leak shrinks the ring permanently, a strictly worse failure than the overshoot it protects against. So the node-keyed charge is retired at exactly one site (_finish_write_through_ack) before the hand-off. CAPACITY WITHOUT A NEW NUMBER. The read consumer is already bounded at runtime by cache_controller.prefetch_capacity_limit = int(0.5 * mem_pool_host.size). Both consumers share one tier, so the write bound is that number's complement. The module introduces no constant of its own and does not become a second sizing authority beside the planner (sgl-project#584/sgl-project#785). Built after _symmetrize_prefetch_capacity(), so the bound derives from the group-agreed number: a rank-dependent admission bound on this path is the sgl-project#645 defect. No hysteresis, deliberately: a refused backup is not lost work (the node stays in the tree and the next insert offers it again), so there is no thrash cost for a watermark to damp, and a resume fraction would be an invented number. Wired into both cache classes that have a construction site -- HiRadixCache (registry.py:115) and UnifiedRadixCache (registry.py:191). HiMambaRadixCache has none, stated at registry.py:108-109. Default unchanged: the ring is built only under --hicache-host-role staging. Under retention -- the default -- the attribute is None and every call site is one `is None` test. The diff is purely additive. Tests: test/registered/unit/mem_cache/test_staging_write_ring_810.py, 33 cases, hermetic (CUDA_VISIBLE_DEVICES=""). Both real write_backup implementations are driven UNBOUND over a fixture, in both directions: a ring with room admits exactly as before, a full ring refuses, and a refusal is shown never to reach evict_host and never to call cache_controller.write. The drain edges are driven through the real write_backup_storage and the real _drain_storage_control_queues_impl. Mutants, 15, all confirmed dying: hiradix admit gate inert -> 4 failed hiradix abort on failed write removed -> 1 failed hiradix abort on floor refusal removed -> 1 failed hiradix drain-phase occupy removed -> 2 failed hiradix release at D2H ack removed -> 1 failed hiradix release at backup ack removed -> 1 failed hiradix release on forced detach removed -> 1 failed unified admit gate inert -> 2 failed unified abort on failed write removed -> 1 failed unified drain-phase occupy removed -> 1 failed unified release at backup ack removed -> 1 failed unified release at D2H ack removed -> 1 failed role gate removed (retention gets a ring) -> 2 failed capacity is the whole tier, not the complement -> 2 failed exhaustion allocates instead of refusing -> 8 failed Source green again at 33 passed. "unified release at D2H ack removed" SURVIVED the first battery at 32/32 green: that call edge was untested. The test it now dies on (UnifiedDrainPhaseTest.test_the_two_phases_do_not_double_count_one_page) was added for it. Without the mutant round a leak edge would have shipped unchecked. ruff/black/codespell clean on the new files; the modified files carry 372 ruff findings before and after this change, i.e. none in these hunks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.