Skip to content

fix: exclude logo png in gitignore - #785

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:png
Jul 28, 2024
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:png

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 28, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs
zhyncs merged commit 6a387a6 into sgl-project:main Jul 28, 2024
@zhyncs
zhyncs deleted the png branch July 28, 2024 07:08
@zhyncs

zhyncs commented Jul 28, 2024

Copy link
Copy Markdown
Contributor Author


https://sglang.readthedocs.io/en/latest/
It's ok now.

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
litmei pushed a commit to litmei/sglang_litmei that referenced this pull request Jun 16, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…ing the weights

FOUNDATION SLICE, and it does not yet change any pool size. It removes the
one obstacle that made the flip seam's arena tail unknowable at sizing time.

WHY THIS TERM MATTERS. The tail is
`max(0, layout_pp.total_bytes - layout_tp.total_bytes)`
(phase_flip_boot.refill_high_water_bytes) and it binds the KV pool on this
rig: 2215 MiB on rank 2 at cut (31,16,17) / vector (32,16,16), which raises
that rank's arming floor from 1523 to 3226 MiB and holds the pool at 161378
tokens while the other two ranks could fund 564k and 704k.

Today it can only be MEASURED, because `layout_tp` is built in
phase_flip_boot:740 from a real TP-shaped worker -- long after the KV pool
has been sized. That ORDERING, not the physics, is what forces the two-boot
protocol and the cold/warm cliff sgl-project#782 documented: a first boot prices the
tail at 0 and sizes 4.4x too large, and every later boot inherits its number
from a predecessor. The code already knows the quantity is static; its own
docstring says "the leg that must grow is a property of the LAYOUT SIZES".

WHAT WAS IN THE WAY. plan_arena_layout needs only metadata -- shape, stride,
dtype, storage_offset and the storage SIZE -- and every one of those is
correct on a meta tensor, so a meta-device parameter set answers total_bytes
exactly, with the same alignment and the same ordering, allocating nothing.
Exactly one input is not correct: data_ptr() is 0 for every meta tensor, and
that is what the function used to detect aliasing. Fed meta tensors it folded
the whole model into ONE slot and returned a total wrong by orders of
magnitude and structurally perfect -- a number that would have gone straight
into the pool solve.

FIX, two halves, both about not guessing:
  * `alias_of=` supplies the alias relation from the caller (the tie between
    lm_head and embed_tokens is a property of the checkpoint, known without
    loading it), and is validated: mismatched views, chains, absent
    canonicals and unknown names all refuse.
  * the inference path now REFUSES meta input, and refuses more than one
    non-empty tensor reporting data_ptr()==0, rather than returning the
    plausible wrong answer.

Tests: test/registered/unit/mem_cache/test_meta_arena_layout_785.py, 9 pass.
The acceptance property is byte-for-byte equality -- a meta layout and a real
layout agree on total_bytes, slot names, offsets, nbytes and aliases -- since
this number is subtracted from a memory budget and a rounding difference is a
real difference. The guard is can-fail proven by pricing the bug it prevents
rather than asserting it. `alias_of` is exercised on REAL tensors too and
must agree with storage inference, so the derived number is the one the
runtime later measures. 63 pass across the arena consumers
(test_arena_high_water_631, test_weights_arena_exact_pin_695,
test_layout_boundary_704) plus sgl-project#782/sgl-project#786.

Pre-existing and NOT introduced here: 17 failures across
test/registered/unit/model_executor/ and the black ratchet, identical in
count and identity with this change reverted; server_args.py and
mem_ledger/engine.py were already black-dirty at HEAD. New files are black
and ruff clean.

NOT CLAIMED: nothing consumes the derived tail yet. Wiring it into the sizing
path ahead of the seam-record read, so a cold boot and a warm boot yield the
same capacity, is the next slice.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…riving it

THE TARGET THE WIRING HAS TO HIT, recorded before any derived number is
allowed near the pool solve.

The runtime prints both layout totals when rung 3 releases the tail. Boot
735-standT, cut (31,16,17) / attn (7,4,5) / vector (32,16,16):

    PP0  released   72.0 MiB (TP layout needs 15925.8 of 16007.5 MiB)
    PP2  released 2214.0 MiB (TP layout needs  8573.8 of 10789.2 MiB)
    PP1  no release line -- no tail to release

Against the seam records that same boot wrote (arena_fixed_bytes): 82, 0 and
2215 MiB. So

    arena_tail = max(0, layout_pp.total_bytes - layout_tp.total_bytes)

reproduces every measured rank to within 1 MiB. The term sgl-project#782 could only
inherit from a predecessor is a subtraction of two quantities the runtime
already computes.

WHY A TEST AND NOT A COMMENT. A census-calibrated estimate was tried first
and REJECTED on exactly these numbers. It reproduced the structure -- rank 2
the only positive tail, rank 1 exactly zero -- and was still unusable: 1.3%
low on the PP layout and 9% low on the TP layout, i.e. 628 MiB out on the
binding rank, about 63k tokens of pool. Getting the ranking right is not
evidence. The last test in this file keeps that bound explicit so a future
derivation cannot be accepted on structure alone.

Also records why the term is worth deriving: rank 2 carries the whole tail
and is the rank that binds. Its arming floor is corridor law + tail + margin
= 3226 MiB against 1523 on the other two, which holds the instance at 161378
tokens while ranks 0 and 1 could fund 564k and 704k.

13 pass with the meta-layout tests. Regression checked properly rather than
assumed: test/registered/unit/mem_cache/ reports 36 failed / 2296 passed with
the sgl-project#785 arena change reverted and 36 failed / 2305 passed with it, the +9
being the new tests -- so the failures are pre-existing and this introduces
none. Two other mem_cache tests do reference weights_arena
(test_pinned_budget_double_count_706, test_pinned_post_registry_550), which
is why the baseline was run rather than reasoned about.

NOT CLAIMED: still nothing consumes a derived tail. The wiring needs an exact
layout_tp at sizing time, and the hazard is now named -- producing TP-shaped
parameters means set_tp_partition_ratios, which mutates PROCESS-GLOBAL
partition ratios and would corrupt the live PP stack's own geometry unless
saved and restored around the meta build. That change cannot be proven
without a boot.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…hapes layers

Deriving the flip's arena tail on a first boot needs a TP-shaped model built at
KV-sizing time, when no TP stack exists yet -- the ordering is deliberate and
phase_flip_seam_reserve.py:72-91 states it: "neither exists until the TP stack
is built, which happens AFTER the PP pool is sized. There is no ordering that
makes them knowable in time, so they are not computed here at all."

Building such a probe means installing the flip vector for the duration and
handing the process back unchanged, which is what scoped_tp_partition_ratios is
for. But every production caller that shapes a real stack uses the process-
global set_tp_partition_ratios instead (phase_flip_boot.py:655), so nothing in
the tree demonstrated that a model built under the scope is sharded to the
scope's vector rather than to the process's installed plan. That was the one
unverified assumption in the derivation plan, and it is load-bearing: the only
discriminator that decides whether a plan applies is len(ratios) == tp_size, so
a build under a leaked plan does not raise -- it falls back to the even split
and loads the wrong units. A probe that inherited the PP plan would return a
plausible, wrong arena total, and the pool sized from it would be wrong in
exactly the way the two-boot cliff already is.

The chain that shapes a layer is layer construction -> tp_partition_size ->
get_tp_partition_ratios(family) -> overlay. This pins the whole chain below the
layer, on the three functions construction actually calls: the overlay is read
by tp_partition_sizes, tp_partition_size and tp_plan_active, it beats an
installed process plan, family vectors ride along, it restores after an
exception raised inside the scope, it nests, and it does not leak into another
thread -- the property that makes a probe safe to run inside a live boot while
the scheduler's own thread keeps reading the installed plan.

Includes the can-fail proof for the silent failure the scope's docstring names:
with a mismatched installed plan the split comes back even and plausible rather
than raising, and the scope is what repairs it.

The assumption holds, so no process-global fallback is needed. What is NOT
proven here is deliberate: no real model is constructed, because that needs a
checkpoint and a device and belongs in the integration test that checks the
derived tail against the measured {0: 82, 1: 0, 2: 2215} to 1 MiB.

12 tests, red first (the suite was written against functions whose overlay
behaviour was unverified). ruff check/format clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…of a previous boot

The flip seam's arena tail is max(0, layout_pp - layout_tp) and it is a
post in the KV pool solve, but layout_tp is built from a real TP-shaped
worker AFTER the pool is sized (scheduler.py:1421 -> :1425 -> :1442).
phase_flip_seam_reserve states the consequence verbatim: "there is no
ordering that makes them knowable in time, so they are not computed here
at all". The tail has therefore been carried between boots in an on-disk
seam record that defaults to 0 when cold, which is the two-boot cliff
behind the sgl-project#678 OOM class: a first boot prices rank 2 at 1523 MiB instead
of 3226 and dies when the NEXTN draft weights land.

A layout needs shape, stride, dtype, storage_offset and the storage size,
and all five are correct on a meta tensor. So the TP layout is answerable
at sizing time without allocating a byte. The one input that is NOT
correct is data_ptr(), which is 0 on meta and is exactly what
plan_arena_layout uses to detect aliasing -- fed meta tensors it folds a
model into one slot and returns a total that is orders of magnitude wrong
and structurally perfect. storage_alias_relation supplies the relation
from the StorageImpl handle, which is a valid identity on meta and agrees
with data_ptr grouping on real tensors.

Strict on this path only: an installed vector whose length does not match
the group size falls back to an EVEN split rather than raising, and that
fallback is load-bearing across the tree (a blanket refusal fails 41
distributed tests). It is not touched -- it is simply not tolerated where
the answer is charged into a memory budget.

DESK GATE, three gloo ranks, CUDA_VISIBLE_DEVICES="", real checkpoint
config: the derivation reproduces the boot's own logged layout_tp on
every rank -- 15925.8 / 8573.78 / 8573.78 MiB against 15925.80 / 8573.78
/ 8573.78 (boot_735_default791b.log:952/962/975). Zero MiB error, 0.1 s
per rank. Can-fail proof: vector 22,21,21 moves the totals to 11921.06 /
10576.16 / 10576.16, far outside the 1 MiB bar. The alias detector has
its own can-fail proof, because its healthy answer on this architecture
is the empty relation -- which is also what a broken detector returns.

This commit derives and instruments only; no sizing path consumes the
number yet, so no boot changes size. The deciding gate remains the boot
instrument checking the derivation against the same boot's own totals.

Tests: 67 passed (the two new suites plus test_arena_tail_identity_785,
test_meta_arena_layout_785, test_scoped_partition_overlay_785,
test_corridor_arming_credit_784). ruff + codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…that measured it

The derivation from the previous commit is checked here where the question
actually lives. A desk run has to pin one GPU capability, because the
compressed-tensors linear scheme is chosen from get_device_capability() at
construction time (compressed_tensors.py:530) -- and this rig is mixed,
rank 0 on sm120 and ranks 1/2 on sm86. So a desk agreement is evidence
about the rig it did not run on.

Two halves, one boot. At the sizing site, where the seam record that
stands in for the tail is about to be spent,
_instrument_arena_tail_derivation measures layout_pp from the LIVE PP
model (it is loaded by then, so deriving it again would only add a way to
be wrong about something this boot already knows), derives layout_tp from
a meta TP model, parks the pair on the runner and announces it. Then in
phase_flip_boot, at the first moment the MEASURED layout_tp exists --
immediately after plan_arena_layout(tp_named) -- the two are graded at a
1 MiB bar against each other. Both numbers come from one boot, so this is
an identity check rather than a comparison against a remembered rig.

The draft worker and the flip's own TP stack both reach the sizing path;
neither owns the PP arena, and the TP stack running this would derive a
TP layout from inside a TP layout. Both are excluded by name.

Nothing consumes the derived number yet, so no boot changes size. That is
the point of the ordering: wiring it into the budget before this grade
exists would be sizing a pool against an unchecked derivation, which is
exactly what the rejected census estimate did -- 628 MiB out on the
binding rank, about 63k tokens of pool, with the structure right.

A failed derivation must not cost a boot that would otherwise have
started, so it degrades to a warning naming the exception and leaves the
record-based path untouched. An instrument that can abort a boot has
stopped being an instrument.

Tests: 11 passed in the extended suite, including the can-fail set --
a 628 MiB miss (the census route's real error) FAILS, and two errors that
CANCEL in the tail still fail, because a right tail from wrong layouts
stops being right the moment the cut or the vector moves. ruff clean on
the touched files; the unused pack_into_arena import ruff reports in
phase_flip_boot pre-dates this branch and is left alone.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
The derivation is graded PASS on all three ranks at error 0.00 MiB against
the same boot's measured totals (boot_735_tail785, commit fa7d387), so
it is now allowed to reach the solve.

Three sites, all on the cold path only:

- _maybe_price_cold_arena_tail sets arena_fixed_bytes on a COLD record
  from this boot's own layouts. A non-cold record, or a cold one that
  already carries a tail, is returned untouched -- this adds a number
  where there was none and never replaces a measurement.
- the arming floor read the draw only when reserve.active, and active
  requires an id-space ANCHOR a cold record cannot have. The anchor is
  genuinely absent; the tail is not, once derived. arming_draw_bytes
  already returns exactly the arena term when no per-leg measurement
  exists, so the condition widens rather than the arithmetic changing.
  This is what separates a cold arming floor of 1523 MiB from the real
  3226 on rank 2.
- the cold solve passed a literal 0 for solve_pool_tokens'
  arena_fixed_bytes. Its own log said "the fixed and arena terms are NOT
  charged"; the arena half of that sentence is no longer true and the log
  now says which term is charged and which is still missing.

The warm path does not move: a measured record already carries the tail,
and the derivation agrees with it exactly, so nothing about the currently
serving configuration changes size. What changes is the FIRST boot of any
configuration, which previously priced rank 2's tail at zero, solved
755649 tokens and OOMed when the NEXTN draft weights landed. That is the
two-boot cliff, and it is the reason a configuration change silently
sized against the previous configuration.

Tests: 7 new, including the can-fail guards -- a rank whose PP layout is
the smaller one must be charged nothing rather than a negative tail, and
the charge must be the SUBTRACTION rather than layout_pp itself (10789
instead of 2215 MiB would size a rank to almost nothing while looking
correctly priced). 46 passed across the sgl-project#784/sgl-project#785 family. ruff clean.
The 51 failures in the mem_cache+managers sweep are pre-existing: the
four named files run 12 failed / 10 passed both on this branch and on a
detached worktree at the merge base 332f083.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
Every term that decides a rank's share of the PP-phase pool is a function
of the cut: its weight layout, its arming floor (which carries the arena
tail), and its per-token cell (because full attention sits on every 4th
layer, so a contiguous cut DETERMINES how many attention layers a stage
holds). Until sgl-project#785 the middle term could only be learned by booting,
which is why cut selection was a hardware experiment. It is now a desk
computation and a boot is what checks it.

No rig constants live in the module. Budgets, per-layer weight mass and
the residual overhead are fields on CutModel, calibrated from a boot's
own instruments; the numbers sit in the test as a fixture. A solver
carrying one machine's fitted vector is exactly the transfer this corpus
forbids.

The attention split is implied, not chosen. Treating it as a free
parameter invented cuts that cannot exist -- a 16-layer stage holding 3
attention layers when every 4th layer is attention -- and the first
version of this search did exactly that. The test pins that the shipped
cut 31,16,17 implies the 7,4,5 the boot declares, and that 32,17,15
implies the 8,4,4 boot 735-full785 was launched with.

The arming floor is not 1523 + tail either. Solved from the boot's own
three floors against its three tails: max(1523, 1011 + tail), i.e.
corridor law 819 plus arming margin 192 on top of the tail. Rank 2
checks: 819 + 2215.4 + 192 = 3226.4.

Calibrated on boot 735-tail785 the model reproduces that boot's KV
available column to within 1 MiB on every rank and its pool to within
0.05% of 161378. It also carries a BOUND rather than only a search
result: sum(layout_pp) exceeds sum(layout_tp) by ~1732 MiB and is
essentially cut-independent, so no cut can zero every arena tail. A cut
relocates the tail onto a rank with headroom; it does not abolish it.

The residual term is an affine fit over layer and attention counts with
three unknowns fitted to three ranks, so it is calibration and not
validation, and the module says so. It is also calibrated on a WARM boot
and must not be read as a prediction for a cold one -- the cold solve is
a different branch that undercharges.

Tests: 14 new, all passing. Regression control: mem_cache + managers runs
51 failed / 4314 passed on this branch and 51 failed / 4268 passed on a
detached worktree at the merge base 332f083 -- same failures, 46 more
passing. ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
PRIOR-ART GATE FAILED, BY ME. python/sglang/srt/planner/pp_cut.py already
solves the pipeline cut -- 2635 lines, per-family attention/linear
decoupling, a measured-rate makespan objective and a per-rank memory
feasibility constraint -- and pp_cut_calibration.py sits beside it. I
wrote a parallel 300-line solver without reading srt/planner/ first. A
second cut solver in the tree is exactly the debt this corpus keeps
paying off, and the right move is to feed the existing one.

The finding that came out of writing it is kept and does not need the
duplicate to stand: the arming floor is max(1523, 1011 + tail) rather
than 1523 + tail, and the attention split is IMPLIED by a contiguous cut
rather than chosen -- which pp_cut.py already states at length in its
module docstring (the "boundary can only fall on a multiple of 4"
analysis), further confirming this was the module to extend.

More to the point, pp_cut.py:412-432 makes RankResources.draft_residency_mib
MANDATORY and refuses to solve without it. That is the input sgl-project#785's meta
probe can now supply -- 1441.14 / 1352.35 / 1352.35 MiB on this rig,
derived in 0.01 s without loading a weight -- and supplying it is the
useful work, not re-deriving the solver around it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…as to fund it

build_flip_draft_worker (phase_flip_boot.py:614) loads a whole second
model -- the NEXTN/MTP draft -- at scheduler.py:1442. The KV pool was
solved at :1425. So the budget never knew a 1.4 GiB model was still
coming, and the only thing that made that survivable was a pool small
enough to leave accidental slack: on the shipped cut the binding rank
held 7027 MiB free and the draft fit by luck.

Boot 735-full785 removed the luck. It solved 764512 tokens and then died
in ct_embedding.create_weights asking for 406 MiB with 227.75 MiB free on
rank 0, inside build_flip_draft_worker. Same shape as the sgl-project#678 OOM and
the same cause as the arena tail: a post the solve cannot see because it
is created later.

Derived the same way the tail is -- a meta-device model under the flip
scopes, 0.01 s, no weights allocated. On this rig 1441.14 / 1352.35 /
1352.35 MiB. Subtracted from the budget rather than folded into the seam
reserve: it is resident weight, not a seam cost, and a term charged under
the wrong name is a term the next reader misprices.

This also supplies an input the planner already demands and nothing could
previously produce: planner/pp_cut.py:412-432 makes
RankResources.draft_residency_mib MANDATORY and refuses to solve a cut
without it.

Tests: 18 passed across the cold-pricing and alias suites. ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
The draft worker IS the draft, and the flip TP stack reaches the same
sizing path after the primary has already paid; either charging twice
would book a 1.4 GiB model two or three times. A configuration with no
speculation has no second model to fund at all, so charging one there
would shrink the pool for nothing -- the mirror of the defect the term
fixes. The charge is also cached, because the solve consults the budget
more than once and a meta build per call would put a model construction
inside a sizing loop.

Tests: 11 passed in the cold-pricing suite.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…t, and ban the pool pin

TWO BOOTS, TWO OOMs, ONE SHAPE. Everything build_phase_flip_tp_stack
creates is allocated at scheduler.py:1442, after the pool is solved at
:1425: the arena above the PP weights, the flip draft's weights, its
attention-backend workspaces, its CUDA graphs. Charging them one at a
time moved the death rather than preventing it -- 735-full785 died in
ct_embedding.create_weights, 735-draft785 died one stage later in the
flashinfer prefill backend with the card completely full. So they are
charged as ONE term.

Two of the three components are DERIVED for this boot and therefore
track the cut and the vector: the arena exceeds the PP weights by
max(0, layout_tp - layout_pp), and the draft weights come from the same
meta probe. Only the remainder is a measured constant, and it is measured
on the boot that LIVED through the build -- 735-tail785 -- as the net
VRAM consumed between "Memory pool end" and the end of the stack build,
minus the two derived terms: 2294 / 1188 / 625 MiB.

The TP stack's own KV pools are charged nothing, and that is checked
rather than assumed: the boot logs "Memory pool end" nine times and the
second and third readings on each rank equal the first (PP0 7.69 / 7.69),
so those pools REBIND the existing allocation. Charging them would have
been charging the pool against itself.

POOL PIN BANNED AS A CLASS (user order 2026-08-21). --max-total-tokens
was never wrong because of its value: it sat inert for weeks while the
profiled capacity was 161378, and the moment sgl-project#785 raised the profile the
same untouched flag capped a 764512-token pool at 430000. The guard is a
date, not an exemption list, because 45 of this host's 68 argv recipes
still carry it -- the cleanup covered the boot-chain scripts, not the
recipe archive, and a list naming 45 files is a formality rather than a
lock. What must be impossible is REINTRODUCING it, so every recipe
written from the order onward must be clean and a sed-derived chain
copying an old recipe into a new file is exactly what that catches. The
guard has a can-fail proof. argv_cut321715.txt, which I wrote today from
the pre-order default, was caught by it and deleted; its evidence lives
in boot_735_cut321715.log.

Tests: 4 in the new guard suite, 22 across the cold-pricing, alias and
corridor suites. ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
… rung that already names it

PRIOR-ART GATE FIRST, and it changed the design twice.

(1) The spill ladder ALREADY has this rung. phase_flip_spill.py:173
DEPTH_DRAFT_GRAPHS = 4, CLI name "draft+graphs", with
IMPLEMENTED_DEPTH = DEPTH_ARENA_TAIL -- so rung 4 parses and is refused,
by design, rather than silently under-delivering. No new flag and no new
concept was needed; the work is wiring the rung that exists.

(2) The park/restore machinery exists too, and its own docstring names
this exact case: short_term_offload_register.py:1269 GraphFamilyRegister
("a whole LAYOUT/ALGO family's capture state, pre-captured at boot and
parked while its family is inactive"), with offload_family/onload_family,
over the sgl-project#93 VMM route. The credit side exists as well --
kv_backing_relief / recover_kv_backing, and the mixin already carries
paused rung graph bytes as registry posts feeding the KV budget
(model_runner_kv_cache_mixin.py:1488-1603). Nothing here needs inventing.

(3) Rung 4 carries a RECORDED REFUSAL that must not be overturned
silently: it "buys a phase-local spill of something the next TP phase
must re-capture, which is a different and much worse trade than rung 2".
That reasoning is about a RUNTIME per-flip spill where every flip pays a
recapture. What is proposed is a BOOT-time deferral paid once. The
arithmetic that changed the trade is measured, not argued: on boot
735-bal785 rank 0 binds the pool at 525462 tokens and needs 2242 MiB more
to fund the 669k reference, against a residual on that same rank of 2294
MiB. Same number. The refusal is therefore LEFT STANDING in
phase_flip_spill until the park/restore path is wired and exercised in a
real flip cycle; this commit only makes the sizer able to express the
credit, so the next step is wiring rather than arithmetic.

The credit is bounded by what was MEASURED, not by what is needed -- a
term that grows to close whatever gap it is shown is not a measurement,
and that has its own can-fail test. The draft WEIGHTS are never credited:
they are needed in the phase the draft actually runs in, and crediting
them would be the sgl-project#678 OOM one rung deeper.

Also: the pool-pin guard now distinguishes a recipe from prose about a
recipe. argv_hc_interval8192.HOSTRAM-NOTES.txt sits under the same glob,
was touched after the order, and quotes the flag it documents.

Tests: 6 new for the credit, 4 for the guard. mem_cache runs 8 failed /
1441 passed on this branch and 8 failed / 1408 passed at the merge base
332f083 -- same failures, 33 more passing. ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…to touch

Formatting only, no behaviour change. These four files carry pre-existing
formatting debt that test_black_ratchet_656 already reports, and the rung-4
wiring lands in the middle of it. Committing the reformat separately keeps the
next commit's diff readable as the change it actually is: a reviewer can see
that black moved no logic, because the formatting is already in the parent.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…e cutover that uses them

The flip's TP stack is built at boot, but the PP phase runs on the boot stack
and never executes a TP or draft forward. Two of that build's posts are
therefore PHASE-COLD -- the attention-backend workspaces and the decode CUDA
graphs -- and on rank 0 of this rig they measure 2294 MiB, which is the same
number as the pool's shortfall against the 669k plain-TP reference. The sizer
learned to express that credit in the parent commit; nothing passed it.

This passes it, and pays for it.

ONE PREDICATE DRIVES BOTH SIDES. phase_flip_spill.cold_stack_deferred is asked
by the KV sizer before it solves the pool and by the boot before it decides
whether to build. They cannot disagree, which matters because the direction
that disagrees silently -- credit taken, deferral not performed -- sizes a pool
into memory the same boot then allocates. That is the sgl-project#678 OOM class, and no
test of either side alone can catch it.

THE CARRIER PIN TRAVELS WITH THE CAPTURE. Rung 2's pin asserts that the draft
parameters still lie inside the spill carrier at capture time. Deferring the
capture while leaving the pin at the boot site would have checked an address
set that nothing had baked yet -- a green assertion about nothing. It moved
into build_cold_stack_posts, and it deliberately does not latch the built-flag,
so a refusal cannot let a retry skip the capture entirely.

THE RESTORE IS PRICED WHERE IT IS PERFORMED. The build happens inside _cutover,
past the point of no return -- the same shape that took all three ranks down on
2026-08-09 when rung 2's re-commit was unpriced. _cold_stack_restore_bytes
charges it into the pp->tp affordability verdict, SUMMED with the draft restore
rather than max()'d with it, because the cutover's own ordering makes them
coexist: the weight pages go back first precisely so the capture has addresses
to bake. Both ends use the same measured constant, so the pool cannot be sized
against one number and funded against another.

IT IS A ONE-TIME MOVE, NOT A PER-FLIP SPILL, and that is what overturns the
recorded refusal honestly. That refusal objected to a per-flip RE-CAPTURE,
measured at 41% of decode throughput. A boot deferral pays no re-capture: the
builder latches, so the second and every later pp->tp leg is a no-op. The
GraphFamilyRegister park/onload path would be a different rung; it has no
production caller today and none is added here.

THE RUNG IS STILL REFUSED. IMPLEMENTED_DEPTH stays at DEPTH_ARENA_TAIL, so
cold_stack_deferred answers False for every configuration reachable without the
new SGLANG_PHASE_FLIP_SPILL_DEPTH_EXPERIMENTAL hatch, and the instance serving
on --phase-flip-spill-depth arena is bit-identical under this commit. The hatch
exists only to break the deadlock between "promote after metal" and "cannot
reach it while unpromoted"; it warns on every resolution and is an env rather
than a server arg, so an operator sweeping the depth flag keeps hitting the
refusal. Promotion follows a real flip cycle, not this commit.

THE CREDIT IS NOT YET SAFE TO SPEND, and this commit does not spend it. Under
the first real agent load the 525462-token form OOM'd on rank 0 inside the
first tp_to_pp flip: F.linear in the GDN in_proj_qkvz asked for 256 MiB with
131.69 MiB free. So the decode forward's transient is UNPRICED in today's
sizing, and handing the deferral's 2294 MiB straight to the pool would
reproduce that OOM with a larger pool behind it. The transient is a separate
sizing post and lands separately; until it does, the hatch is the only way to
reach this rung and the acceptance gate for it runs under load.

Tests: 140 passed -- 33 new red-first tests across two files (predicate,
interlock, hatch, sizer threading, builder contract, carrier-pin travel,
idempotency, cutover restore, seam pricing), plus the existing phase-flip boot
and runtime suites. Both new files carry CAN-FAIL guards aimed at the rung the
rig actually serves on, and structural tests so a stray direct init_cuda_graphs
or an unconsulted priced term fails. Both are registered with register_cpu_ci;
the CI runner executes `python3 <file> -f`, so function-style pytest files were
never being collected.

Scoped regression over the 119 test files that reference any touched module:
2153 passed. A full failure-set comparison against the parent commit in a
separate baseline worktree found 31 failures on the parent and, initially, 32
here. The one extra was mine and is fixed in this commit:
test_phase_flip_arena_tail_631 pins the SOURCE TEXT of the _staging_bytes
return, so that the arena tail stays ADDED to the max() and never becomes a
third argument of it -- a difference one measured corridor breach wide and
invisible in a value. Reformatting had collapsed the return onto one line and
broken the pin without touching the arithmetic; the multi-line shape is
restored and commented so the next reformat does not silently do it again.
With that, the two failure sets are identical in both directions.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
… tensor an OOM landed on

gdn_prefill_scratch_mib began its accounting at the conv output -- one
allocation too late. in_proj_qkvz produces a packed [T, 2*key_dim + 2*value_dim]
tensor whose z half appears in no other term, and in_proj_ba adds 2*Hv columns
beside it. Neither was counted.

THE SPECIMEN. On 2026-08-21 an instance died at exactly that line under the
first real agent load, inside the first tp_to_pp flip:

    qwen3_5.py:625  projected_states_qkvz, _ = self.in_proj_qkvz(hidden_states)
    linear.py:835   return F.linear(x, layer.weight, bias)
    torch.OutOfMemoryError: Tried to allocate 256.00 MiB.
    GPU 0 total 31.34 GiB of which 131.69 MiB is free.

256 MiB is 8192 * 16384 * 2 exactly: --chunked-prefill-size 8192 tokens, and
16384 = 2*(16*128) + 2*(48*128) columns at --tp-size 1, where the GDN layer is
unsharded and carries full width. The number the estimate omitted is the number
that killed the instance.

THIS IS NOT A COSMETIC UNDER-REPORT. The prefill-admission guard derives its
threshold from this function, so the omission set the gate that was supposed to
prevent the allocation. The log shows the gate doing its arithmetic and then
letting the chunk through anyway:

    corridor law 1024 MiB (prefill admission, 8192 tokens)
    corridor shortfall of 981 MiB for rank 0 (was 953 MiB)
    corridor cannot be restored ahead of this chunk

1024 + 257 = 1281, and the corrected estimate is 1270 MiB at those settings.
The guard was holding a threshold roughly one in_proj tensor too low. Making
the guard ENFORCING rather than advisory is a separate change and is not in
this commit; this one makes the number it enforces on honest.

THREE CONSUMERS SEE THE CORRECTED FIGURE, and none of them is the KV sizer:
the pinned-reserve diagnostic (server_args.py:13222), the planner's split
advisory (planner/split_probe.py:404) and the ledger term
(mem_ledger/engine.py:433). All three were reasoning from the low number. The
KV sizer charges no forward-activation term at all -- see the declining gates
at server_args.py:13168-13173, 13178-13183, 12298-12300 and
mem_ledger/engine.py:1265-1266 -- which is a real gap, deliberately left for a
separate change rather than folded in here.

Tests: 1236 passed in the server_args and mem_ledger suites. New file
test_gdn_in_proj_transient_785.py pins the specimen (the 257.5 MiB term at the
crash's own settings, the total in 1200-1350, chunk-proportional scaling, and
the per-rank share) and carries a CAN-FAIL guard asserting the estimate now
exceeds the 1024 MiB threshold that let the OOM through -- an estimate still at
or under it would leave the gate exactly as permissive as before. Two existing
pins moved with the formula and say why: the exact-formula assertion in
test_uneven_tp_args.py, and the diagnostic's per-layer figure, 63 -> 79 MiB.
The one remaining failure in those suites, test_r1_private_constant_gate_584,
reproduces on the parent commit in a baseline worktree.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 21, 2026
…ime one

The rung-4 deferral moves the flip TP stack's attention backends and decode
CUDA graphs from boot to the first pp->tp cutover. The boot-time build runs
inside phase_flip_tp_scope; the deferred one did not, and that asymmetry is
the whole defect.

Attention backends read their geometry FRESH from get_parallel() contextvars
at construction (triton_backend.py:731-757), while the KV pool's row width is
a value baked onto the ModelRunner at __init__. Unscoped, the cutover build
saw the ambient primary topology (pp=3, tp=1, dcp disabled) instead of the
flip TP geometry the pool was sized under, so uneven_dcp came out False, the
forward skipped _dcp_write_gather, and the store kernel was handed raw
per-rank rows against a pool baked for 4 replicated heads. On metal that is:

  store_cache rejected: k(24, 2, 256) row_dim=1024 -> k_rows=12,
  k_cache(201377, 4, 256), indices(24,)

24 tokens silently refolded into 12 rows (6 on the 1-head ranks), with no
error until the capture died.

Four changes:

A. restore_deferred_cold_stack re-enters phase_flip_tp_scope, deriving n and
   world_rank exactly as the boot path does.

B. phase_flip_tp_scope now SAVES and RESTORES the prior TP routing instead of
   forcing it off on exit. Without this, A would be worse than the bug: the
   cutover has already armed the routing and the TP phase continues after the
   deferred build returns, so a hard False would have routed every later
   collective back onto the primary tp=1 groups -- the silent no-op all-reduce
   that set_phase_flip_tp_active's own docstring exists to prevent. The boot
   path is unchanged; its prior value is always False.

C. A named guard before init_attention_backends refuses a pool/ambient
   geometry mismatch with a message naming the pool head_num and row_dim, the
   ambient attn_dcp_size and attn_tp_size, and the uneven_dcp verdict. A
   silent refold becomes a loud error. It skips rather than raises on any pool
   shape it cannot interpret, so it stays inert on the healthy path.

D. assert_row_schema_compatible's docstring claimed weighted DCP always
   replicates KV heads. It does not: attn_kv_replicated turns replication OFF
   when kv_heads >= tp_size, which is this rig (4 KV heads over tp=3 -> per-
   rank [2,1,1]). The check passed only because both pools happened to report
   the same head_num. Docstring corrected and the error now names the
   configuration actually observed. The check itself is unchanged.

Tests, hermetic and CPU-only:
  test/registered/unit/managers/test_cold_stack_boot_deferral_785.py
Red before the fix: 4 failed, 20 passed -- the new A/B/B/C tests.
Green after: 24 passed.
Together with test/registered/scheduler/test_phase_flip_boot.py: 55 passed,
verified independently of the implementing run.

Includes a can-fail proof for the part-C guard in both directions: matching
geometry inside a scope builds cleanly, mismatching geometry raises before
init_attention_backends is reached.

NOT yet verified on hardware -- no multi-rank pp->tp cutover has executed
this path on metal. That is the next GPU window.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…by hand

--hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires
an explicit --hicache-size. That refusal is only worth having if the number
it demands comes from somewhere; a hand-picked --hicache-size is exactly the
pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785)
exists to prevent.

planner/hicache_staging.py derives it. Two consumers, and sizing for one of
them silently breaks the other:

  * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the
    moment it is handed to the storage backend until the write acks, so the
    resident set is drain rate x that residency, widened by a burst margin
    (write-through arrives in bursts, the drain is steady).
  * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch
    takes its destination slot from mem_pool_host.alloc() BEFORE the storage
    read is issued (hiradix_cache.py's _prefetch path). Sizing from the write
    side alone yields a tier that is correct and quietly serialises prefetch
    -- invisible to every capacity metric, visible only as latency.

The tier takes the larger, rounded UP: rounding down would emit a tier
smaller than the derivation justifying it.

NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner
probes rather than carrying a measured vector from one box to another, so
nothing in this module encodes what this rig happens to do (~0.5 GB/s on its
ZFS-backed file tier); that figure appears only as a test input.

WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster
than the backend drains them, NO finite size holds -- residency is set by the
drain, so a faster producer grows the in-flight set without bound. sustainable()
answers that question so a caller can refuse instead of emitting a number that
cannot be correct. The remedy there is backpressure at the producer, a runtime
mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice
on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer
rather than blocking, trading "a bounded spike for an unbounded latency" --
which is right for a prefetch worker and wrong for a write path under a RAM
budget, where re-inflating the pinned footprint is the failure the budget
exists to prevent. That ring is the next posten, not this one.

fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like
every other pinned producer, so a tier that does not fit is refused where it
is derived rather than discovered at allocation.

Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that
DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists
nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and
does not depend on it, but the supporting argument is unrecoverable. Recorded
rather than reconstructed: rebuilding it from four citation fragments would
read as primary evidence while being inference, and the next reader would
cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed
until the target exists.

Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases,
hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended
tests:
  read consumer dropped (write-only sizing) -> 1 failed
  rounds down instead of up                 -> 1 failed
  zero-drain guard removed                  -> 2 failed
  burst-margin floor removed                -> 1 failed
  sustainability predicate inverted         -> 2 failed
Source green again at 19 passed; 28 passed together with the role gate.

Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this
change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence
assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of
dd6967c with these files removed -- same test, same assertion. No
pre-existing test changed status. ruff/black/codespell clean.

Not wired to a probe yet, and named so it is not assumed: nothing measures
the drain rate automatically, so a staging profile still needs the number
supplied. Measuring it needs load against a running server, which belongs to
another strand's boot window right now.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 22, 2026
…ount the refusals

A staging host tier is small on purpose, and nothing at runtime bounded what
write-through put into it. The tier itself did, by running out -- which is not
a bound but two silent failures. The read consumer starves: a prefetch takes
its landing slot from mem_pool_host.alloc() BEFORE issuing the storage read, so
a tier full of undrained write-through pages makes it evict, retry, truncate
and abandon; a correct miss, invisible to every capacity metric and visible
only as latency. And the write refusal itself is silent: write_backup learns
about exhaustion by reading a None back out of an allocation that already
happened, then -- with no rank-uniform floor published -- takes evict_host(), a
RANK-LOCAL tree edit, which is the sgl-project#645 divergence.

NOT sgl-project#720's ReadBufferPool. Its acquire() (read_buffer_pool.py:99-105) answers
exhaustion by allocating a fresh UNCOUNTED pinned buffer, deliberately:
"stalling the prefetch worker to save memory would trade a bounded spike for an
unbounded latency". Right for a prefetch worker, and precisely wrong for a
write path under a RAM budget -- the footprint that overflow re-inflates is the
one --hicache-host-role staging exists to cap. StagingWriteRing never
allocates; its only outcomes are admitted and refused-and-counted.

THE RESIDENCY IS TWO-PHASE, and only one phase is refusable. write_backup
allocates the host slots; write_backup_storage calls node.protect_host() and
the protection drops only at the backup ack (ongoing_backup.pop ->
release_host). The bandwidth-delay product that planner/hicache_staging sizes
the tier from is the SECOND phase, so a ring spanning only the device->host
copy would claim a bound it does not have. Phase one is admit(), taken before
any allocation, where a refusal is actionable. Phase two is occupy(), which
cannot be refused -- the page is already resident, so refusing would free
nothing and merely hide the drain queue from the next admission.

Phase two is keyed by the STORAGE OPERATION, not the node, and that is not
cosmetic: in UnifiedRadixCache one write-through ack fans out into several
storage backups after a node split, each with its own id, and lock_node is by
then no longer among publish_nodes. A node-keyed charge would be stranded
there -- and a leak shrinks the ring permanently, a strictly worse failure than
the overshoot it protects against. So the node-keyed charge is retired at
exactly one site (_finish_write_through_ack) before the hand-off.

CAPACITY WITHOUT A NEW NUMBER. The read consumer is already bounded at runtime
by cache_controller.prefetch_capacity_limit = int(0.5 * mem_pool_host.size).
Both consumers share one tier, so the write bound is that number's complement.
The module introduces no constant of its own and does not become a second
sizing authority beside the planner (sgl-project#584/sgl-project#785). Built after
_symmetrize_prefetch_capacity(), so the bound derives from the group-agreed
number: a rank-dependent admission bound on this path is the sgl-project#645 defect.

No hysteresis, deliberately: a refused backup is not lost work (the node stays
in the tree and the next insert offers it again), so there is no thrash cost
for a watermark to damp, and a resume fraction would be an invented number.

Wired into both cache classes that have a construction site -- HiRadixCache
(registry.py:115) and UnifiedRadixCache (registry.py:191). HiMambaRadixCache
has none, stated at registry.py:108-109.

Default unchanged: the ring is built only under --hicache-host-role staging.
Under retention -- the default -- the attribute is None and every call site is
one `is None` test. The diff is purely additive.

Tests: test/registered/unit/mem_cache/test_staging_write_ring_810.py, 33 cases,
hermetic (CUDA_VISIBLE_DEVICES=""). Both real write_backup implementations are
driven UNBOUND over a fixture, in both directions: a ring with room admits
exactly as before, a full ring refuses, and a refusal is shown never to reach
evict_host and never to call cache_controller.write. The drain edges are driven
through the real write_backup_storage and the real
_drain_storage_control_queues_impl.

Mutants, 15, all confirmed dying:
  hiradix admit gate inert                     -> 4 failed
  hiradix abort on failed write removed        -> 1 failed
  hiradix abort on floor refusal removed       -> 1 failed
  hiradix drain-phase occupy removed           -> 2 failed
  hiradix release at D2H ack removed           -> 1 failed
  hiradix release at backup ack removed        -> 1 failed
  hiradix release on forced detach removed     -> 1 failed
  unified admit gate inert                     -> 2 failed
  unified abort on failed write removed        -> 1 failed
  unified drain-phase occupy removed           -> 1 failed
  unified release at backup ack removed        -> 1 failed
  unified release at D2H ack removed           -> 1 failed
  role gate removed (retention gets a ring)    -> 2 failed
  capacity is the whole tier, not the complement -> 2 failed
  exhaustion allocates instead of refusing     -> 8 failed
Source green again at 33 passed.

"unified release at D2H ack removed" SURVIVED the first battery at 32/32 green:
that call edge was untested. The test it now dies on
(UnifiedDrainPhaseTest.test_the_two_phases_do_not_double_count_one_page) was
added for it. Without the mutant round a leak edge would have shipped unchecked.

ruff/black/codespell clean on the new files; the modified files carry 372 ruff
findings before and after this change, i.e. none in these hunks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant