Skip to content

feat: add e2e latency - #704

Merged
zhyncs merged 2 commits into
sgl-project:mainfrom
zhyncs:e2e
Jul 23, 2024
Merged

zhyncs merged 2 commits into
sgl-project:mainfrom
zhyncs:e2e

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 23, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs
zhyncs requested review from Ying1123 and merrymercy July 23, 2024 19:17
@zhyncs
zhyncs merged commit fa7ccb3 into sgl-project:main Jul 23, 2024
@zhyncs
zhyncs deleted the e2e branch July 23, 2024 19:51
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
* Update run_suite.py

* Update test_retract_decode.py

* Update test_reasoning_content.py
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… arena ladder, and that reprices it

The actuator survey came back and it changes the design, so the design changes
rather than the finding being reinterpreted.

## R1: there is no runtime cross-rank weight mover, and the absence is deliberate

regime_stages.py:100 defines REACH_NO_WEIGHT_MOVER -- "no runtime actuator
moves weights, switching arms needs a restart (sgl-project#354/sgl-project#357)". StagePlan.reachable
(:156) excludes it, so such a stage cannot be selected, and regime_act.py:121-130
wires exactly kv/vram/phase. Corroborated at uneven_perf.py:7744-7751.

So sgl-project#363's "weight mover" is a REFUSAL CODE, not an actuator -- the opposite of
the counter-versus-actuator failure, and honest. It is not something to wire to.

What exists is weights_arena.py + phase_flip_boot.py:361: a fixed-address VRAM
arena refilled by a host->device memcpy from a boot-baked pinned image,
checksum-verified, with a rollback arm, and containing no dist.* call at all.
On this no-P2P rig that is the better primitive anyway -- a rank-to-rank
transfer would stage through host memory regardless -- and the fixed address
keeps captured CUDA graphs valid across a rung change.

## The consequence, which is not favourable

The arena is sized for the DEEPEST rung and resident at EVERY rung: a rank does
not get its weight bytes back when the ladder sits shallow. solve_arena_ladder()
models this, and two things fall out.

First, the shallow rungs get poorer. Over a rank0 span of [31,38] the top rung
drops from 518,433 to 390,700 live-equivalent -- below the incumbent's observed
434,878. The ladder costs ~10% pool at the top to buy up to 1.231x pipelined
prefill at the bottom. Standing alone that is a marginal trade, and this commit
says so in the design rather than shipping the flattering number.

Second, and structurally: with free memory pinned by the arena, a rung's pool
depends only on its ATTENTION count. Two rungs sharing an attention profile
price identically and the faster one strictly dominates. Under an arena the
ladder's real axis is the attention-count vector, not the raw layer cut.

That invalidates my own earlier Slice 1 pair. [33,15,16] <-> [35,14,15] was
chosen because it moves zero attention layers, which is exactly what makes it
useless AS A LADDER PAIR. It is retained only to prove the mover mechanically;
a pair that actually trades must cross an attention boundary, so Slice 1b uses
[35,14,15] <-> [38,13,13].

The alternative mover -- reallocate per rung, restoring the full per-rung pool
at the cost of a CUDA graph recapture -- is named and left open. Both solvers
are in the branch; Slice 2 measures the recapture cost and closes the fork.

## Part B repriced by its own survey

The cross-rank LSE merge is real, N-way and rank-order deterministic
(layers/dcp/comm.py:228-262, all_gather places rank i at index i), which RETIRES
the merge-order risk. But two blockers are confirmed: DCP groups are subsets of
one PP stage by construction (parallel_state.py:3142-3159) and under PP prefill
no DCP group is instantiated at all (dcp_group_guard.py:1-42); and the KV pool
is DIMENSIONED by weight ownership -- model_runner_kv_cache_mixin.py:2466-2470
filters attention layer ids to the rank's own [start_layer, end_layer) before
the pool is built, so a rank has no row-space for a layer it does not own.
Part B is construction, not assembly; slice 3 splits into 3a/3b/3c at 4-7
windows, with the pool allocation rework on the critical path.

layers/dcp/test_weightless_kv_math.py is confirmed test-only: it imports no
torch and reimplements the LSE formula locally. It is not evidence of a wired
path.

## Test results

60 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean on all new files,
codespell clean.

- test_ladder_controller_704.py (10, new): oscillation falsifiers on adversarial
  fill traces. The anti-flap test carries a CAN-FAIL PROOF: with the hysteresis
  collapsed to a bare threshold the controller emits 400 moves under a 1-token
  jitter; with it intact, zero. An earlier version of that test probed an
  INACTIVE threshold and passed for both, i.e. it was vacuous -- fixed by
  exposing active_descend_threshold() and asserting a fixed point.
- test_layout_ladder_704.py (25, +5): arena sizing, the equal-attention-profile
  domination property, the deeper-ladder-costs-pool trade, and an unaffordable
  arena refused with its shortfall.
- test_pp_cut_family_pool_704.py (6) and the two sgl-project#702 suites (19): unchanged.

The controller is a DECISION FUNCTION and says so in its docstring: it moves no
bytes and nothing is wired to it yet. Stated because a decision function
mistaken for an actuator is the exact confusion this survey was run to avoid.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…h reserve is the blocker

Review-gate item 2. Every term from config or instruments, nothing fitted.

KV term is byte-exact from config, not fitted: kv_cache_dtype fp8_e4m3 (1 byte)
x num_key_value_heads 4 x head_dim 256 = 1024 B per token per ATTENTION layer
for K, 2048 B for K+V. The boot log's per-rank K sizes at 436,766 tokens match
on all three ranks (2.92 / 2.08 / 1.67 GiB predicted and logged). This
falsifies the 4096 B bf16 constant in Slot-3's doc directly.

Mamba residency derived from the allocation sites (memory_pool.py:583-608,
:655-665, :693-705) with max_mamba_cache_size 12 (13 slots),
max_running_requests 4 (5 spec slots), speculative_num_draft_tokens 4,
temporal_state_shape (48,128,128), conv_dim 10240, win 3, bf16:

  temporal_state                 13 x 48x128x128 x 2B  = 19.50 MiB/GDN-layer
  conv_state                     13 x 10240x3 x 2B     =  0.762
  intermediate_ssm_state_cache   5 x 4 x 48x128x128x2B = 30.00
  intermediate_conv_window_cache 5 x 10240x6 x 2B      =  0.586
  total                                                = 50.85 MiB/GDN-layer

The gate's 19.5 MB/GDN-layer is the temporal component ALONE. The full
residency is 50.85, dominated by the SPECULATIVE intermediate cache at 30.00,
so a model charging only 19.5 under-charges GDN layers by 2.6x.

Forward validation against the live [28,20,16] boot chain (avail-after-weights
minus avail-after-pool): predicted 6.874 / 4.910 / 3.928 GiB against measured
6.91 / 4.95 / 3.97 -- under 1.1 percent on every rank. The residual is nearly
CONSTANT at ~40 MiB rather than scaling with layers or attention, which is the
evidence that the per-layer terms are complete: a missing per-layer term would
have shown a slope.

CORRECTION to my own blocker claim. I previously named the +-7 percent
unbooted-floor uncertainty as what blocks bootable predictions. Wrong in
emphasis. Retro-prediction runs the equation backwards and needs everything the
sizer sets aside before the pool; every term is now config-derived except the
GRAPH RESERVE, which the chain puts at 6,372 / 2,650 / 2,377 MiB per rank after
subtracting the measured floors. Next to that the floor uncertainty is
second-order. The fix is not to model the graph reserve independently but to
have the pool solve consume the sizer's own reserve terms, as it must already
consume the sgl-project#676 floor.

Structural finding encoded (gate 3.5): no cut keeping rank2 = layers 48-63 can
beat the incumbent pool. Rank2 is byte-identical across both boots (weights
10.40 GiB in each log) and binds at 436,766, a hard min-rule ceiling. Pool
gains require shrinking rank2's ATTENTION count or Part B decoupling, not
rank0/rank1 rebalancing.

Also on record: the incumbent's binder is PP2 per the boot log (PP1 cap
463,406, PP2 436,766). My rev5 calibration assigned it to rank1 and solved a
free constant from that assumption; this derivation needs no such fit and
supersedes it.

Docs only. Items 1 (canonical pp_cut convergence) and 3 (sgl-project#701 slice-3 rework)
not started.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…take the four gate corrections

The review gate adjudicated the attention-only divisor in this branch's favour
-- white-box at the allocator, byte-exact against the boot log's K sizes -- and
found four defects of mine. This lands the convergence and all four.

## Convergence: one canonical solver

Merged c5afff7 (Slot-2 rev5), which adopts the attention-only rule and adds
the per-layout arming floor and mamba terms. My duplicate FamilyPoolModel /
stage_family_capacities / family_phase_pool are DELETED; rev5's PhasePoolModel
is the only pool model. Slot-2 leads the solver; this branch owns
layout_ladder, ladder_controller and the arena model, which now consume rev5
instead of reimplementing it.

Two additions to pp_cut.py that do not duplicate rev5:
- kv_mib_per_token_per_attn_layer_from_config() + kv_dtype_width_bytes()
- decoupled_phase_pool(), the part B projection, rebased on PhasePoolModel

## E1: the KV cell was FITTED, and was wrong by 2x

DESIGN_704 read the cell as 4096 B (bf16). The shipped config is fp8_e4m3 and
the cell is 2048 B for K+V, 1024 B for K. Fitting it against an observed pool
is what produced both the dtype error and the bogus "0.83 of observed" fudge --
they cancelled into something that looked calibrated.

The cell is now consumed from config (2 x kv_heads x head_dim x dtype_width).
Unknown dtypes raise rather than default, because a wrong default is a silent
2x on every pool number; 'auto' raises too, since it names no width.

## E2/E3: the binding rank, and the retraction

I claimed the incumbent binds on rank1. The boot log says PP2 binds at 436,766.
The functional form was right; my free-bytes vector was not, and that produced
the retraction:

[33,15,16] leaves rank2 holding layers 48-63 -- byte-identical to the
incumbent's rank2, measured cap 436,766. Under the min-rule no cut keeping that
rank2 can exceed it, so my claimed 457,604 over-predicted an UNCHANGED rank's
measured capacity by 4.8%. Expected actual ~387k, about -11%. The arm is
withdrawn. [32,16,16] had already failed its gate on metal at 416,796, the same
error class. The "discriminating experiment" justification was void: the
divisor was settled white-box, so the window would have bought nothing.

What replaces it is stronger than the arm I lost: NO cut that keeps rank2 =
layers 48-63 can beat the incumbent pool. Pool-positive rungs must shrink
rank2's attention count, or wait for part B. That is structural, not a number
needing re-measurement.

## E4: GDN residency was under-charged 2.6x, and it falsified a claim

The full per-GDN-layer figure is 50.85 MiB, not 19.5: temporal_state 19.5 +
speculative intermediate_ssm_state_cache 30.0 (5 spec slots x 4 draft tokens) +
conv_state 0.762 + intermediate_conv_window 0.586.

This falsified a structural claim of mine. §3.7 said two rungs sharing an
attention profile price EXACTLY the same, so the deeper strictly dominates.
That held only because the 30 MiB speculative term was missing. Corrected, such
rungs differ by their GDN residency (~3,250 tokens at 8 attention layers): a
weak real trade, not a domination. test_under_an_arena_equal_attention_profiles
_have_equal_pool failed the moment the constant landed and is replaced by
test_under_an_arena_same_attention_profile_differs_only_by_gdn_residency, which
asserts the gap is explicable by GDN residency and nothing else.

## Part B defects D4/D5/D6

- D4: the byte-identity gate as written could never pass. "decoupled vs
  coupled" is A-vs-B -- an LSE merge sums in a different float order than
  monolithic attention -- so it would fail forever on correct code and be
  waived. Respecified: (1) decoupled-vs-decoupled determinism, byte-identical
  across runs and boots, CPU-sampled inputs; (2) agreement with the coupled
  reference within a tolerance fixed before the run. Gate 1 is never waived.
- D5: "TP identity wins" was unconditional and cannot be. The measured TP
  vector puts ~43.8% of KV rows on rank0, the rank with the least free bytes at
  exactly the deep rungs part B exists to unlock, so prize 1 and prize 3 can
  contradict. TP identity now carries a per-rung feasibility bound, and the
  rung yields before the vector does.
- D6: committing rung changes only at a fully-quiescent boundary can starve.
  640 chunks per max-length prompt with overlapping admissions means the
  boundary may never arrive -- and ascent is needed precisely while chunked
  prefills are active. That is the ladder's own sgl-project#701-shaped wedge. Paired with
  an admission hold plus bounded drain and an urgency-derived deadline;
  asymmetric, since only a safety-seeking ascent may hold admission.

## Model discipline

LadderInputs now REQUIRES an arming_floor_for(counts) provider and refuses
construction without one -- a constant floor is the E3 error. Rev5's own
docstring records the known gap: the sgl-project#676 solver derives the floor from a
measured seam draw, so an unbooted cut has no solved floor, leaving ~±500 MiB
(~±32,000 tokens, ~7%) of uncertainty on every unbooted rung. No rung's
predicted pool is a boot gate on its own; the retro-prediction gate against all
four measured points (434,878 / 435,822 / 436,766 / 416,796) comes first, and
Slot-2 owns it.

The withdrawn headline table is deleted rather than patched. Re-deriving those
numbers before the retro gate passes would repeat the mistake.

## Test results

63 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean.

- test_kv_cell_from_config_704.py (7, replaces test_pp_cut_family_pool_704.py):
  reproduces the logged K sizes 2.92/2.08/1.67 GB from config alone with zero
  free parameters; pins that a bf16 reading misses the log by exactly 2x;
  refuses unknown dtypes and 'auto'; keeps the 0.47-bytes-per-element
  dimensional record that settled the dispute.
- test_layout_ladder_704.py (25) and test_ladder_controller_704.py (12):
  ported to rev5's API. test_descend_only_happens_at_low_fill now states its
  thresholds RELATIVE to the solved ladder -- its absolute token constants
  silently encoded the KV dtype and became meaningless when the cell was
  corrected.
- rev5's test_pp_cut_phase_pool_702.py + test_pp_cut_prefill_speed_702.py (19):
  green across the merge.

The rig fixtures are labelled STRUCTURAL, NOT CALIBRATED PREDICTORS: they pin
monotonicity, hysteresis and arena residency, which are invariant to the free
vector. No pool value in them is bootable.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…age_pp_capacities

Last reference to my deleted duplicate model. The refusal itself is rev5's and
is unchanged: a stage with no attention layer carries no token-scaling KV, so
pricing it as unbounded capacity is how a cut that cannot serve gets armed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…-0 amendment

The arena argument establishes the SIGN of the trade, not its size. The
specific 1.231x / 390,700 came from the fitted bf16 cell and went out with the
rest; restating it in corrected-looking form before the retro-prediction gate
passes is the same mistake in a smaller font.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
This is the instrument that unblocks the four-boot retro-prediction gate.

The blocker was not the arming floor and not the graph reserve as a quantity.
It was that the sizer builds budget_posts -- its own named decomposition of the
KV budget -- on every boot and hands it to budget_exhausted_message ONLY when
the budget runs out. The sizer names every term of its arithmetic exactly when
it fails, and discards the naming when it succeeds. So the planner had no
instrument for the reserve it must not double-count, and had to re-derive it
from config in a different process, where it could not even be compared.

Three independent re-derivations, three misses against the live [28,20,16]
boot: subtracting derived_rank_auto_reserve_mib (4160 MiB, called directly)
gave PP2 = 524,620 tokens against a measured 436,766, i.e. +20 percent; the
non-budget slack path gave -3.8 percent; the configured-budget variant gave
-12 percent. The boot actually takes the budget_mib branch because
rank_gpu_memory_mib is set, which is a fourth path again. Re-deriving a number
the sizer already computes is the shape this corpus has paid for repeatedly.

So the fix is the one the gate's meta-lesson implies: create the instrument and
consume it. One info line emits the posts and the final rest_memory on the
success path, in the same GiB units the failure message already uses. The pool
solve then consumes these exactly as it already consumes the sgl-project#676 arming floor,
rather than recomputing them.

Not included, and stated rather than implied: the consuming side. PhasePoolModel
cannot require a sizer_reserve field until a boot has emitted one -- the posts
do not exist in any log on disk today, which is precisely the defect. The next
boot produces them, and the consuming field lands with real values instead of a
placeholder.

Verification: compiles, ruff clean, logger already in scope at module line 168.
model_executor suite 594 passed / 15 failed, and the 15 are PRE-EXISTING --
verified by reverting this file, re-running, and getting the identical
594/15, then restoring. This commit adds one log statement and changes no
control flow.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…RO weight bytes

DESIGN_704 specified the ladder's actuator as an H2D arena_refill per rung
change -- one contiguous copy of the target layout's host image, 451-901 MiB a
step -- because no cross-rank weight mover exists (regime_stages.py:100,
REACH_NO_WEIGHT_MOVER). Implementing it showed that is more work than the
existing primitives require.

plan_arena_layout fixes slot offsets by SORTED TENSOR NAME, and
bind_arena_views rebinds parameters to arena views without copying any bytes
(weights_arena.py:495-519). Plan the arena over the UNION of the rungs' tensors
rather than per rung and a tensor shared by two rungs sits at the same offset
in both -- so a rung change is a rebind plus a PP-boundary change, and no
weight byte moves at all.

## The union is doing real work, and there is a can-fail proof

With PER-RUNG layouts, 14 of 56 shared tensors land at DIFFERENT offsets,
because model.layers.28.* sorts lexically between ...27.* and ...3.* and
displaces every slot after it. A per-rung arena would therefore have to
relocate weights that are already resident and unchanged, purely because a name
sorted differently. Under the union, zero move. Without that check every
zero-copy assertion in the suite would pass vacuously.

## The price, and what is NOT claimed

Cost moves from bandwidth to residency: a layer that changes hands must be
resident on BOTH ranks, so the world holds 64 + (layers that move)
layer-images -- one extra layer (~451 MiB) for the slice-1a pair. That is the
same trade DESIGN_704 section 3.7 already prices for the arena, and on this rig
it strictly dominates the refill, since with no P2P a rank-to-rank transfer
stages through host anyway.

arena_refill is NOT replaced. It stays correct for the PHASE FLIP, where the
two layouts are genuinely different tensor sets (PP weights vs TP width-shards)
with no useful union. The union is for the LADDER, where consecutive rungs
differ by one or two whole layers.

Scope is carried in the data structure, not in a reader's memory: "zero bytes"
must not be read as "nothing to do". A GDN layer carries per-sequence recurrent
state (temporal_state ~19.5 MiB/layer + conv_state ~0.762) that travels with
the layer, so slice 1a flips only at QUIESCENCE. Every FlipDelta carries
requires_quiescence=True and an out_of_scope string, and a test asserts both.
Live-state transfer is slice 1b.

## Boot ticket, prepared and deliberately not self-arming

scripts/boot_704_1a_29_19_16.sh splits slice 1a into a zero-code half and a
code half:

  1a-i  two STATIC boots, [28,20,16] <-> [29,19,16], changing exactly one
        variable (--pp-layer-ratio). Both cuts are attention profile (7,5,4)
        and layer 28 is LINEAR, so no attention layer and no KV row moves.
  1a-ii the union arena plus runtime wiring.

The ticket refuses without GO=1, a /spinning/gpu-arb/ claim, a clean tree, and
exactly one --pp-layer-ratio in the source argv -- a boot that changes zero or
two variables proves nothing. It stops before launching: F4-r4 owns the queue.

Falsifiable prediction, written to disk BEFORE the window as the review gate
requires: P1 pool 436,766 +- 2500; P2 PP2 binds; P3 K sizes 2.92/2.08/1.67 GB
unchanged. The reasoning is structural, not a rung-table re-derivation (which
is forbidden until Slot-2's retro-gate closes): rank2 keeps layers 48-63
byte-identically so cap2 is unchanged; rank1 sheds a layer so its cap rises;
rank0 gains ~502 MiB of weights plus GDN residency, dropping cap0 by ~36,600
tokens from >=550,000, still far above 436,766. If P1 fails, the corrected pool
model is wrong where it matters -- record and STOP, do not retune inside the
window.

Independent second prize: PrefillTiming.fixed_ms defaults to zero, so every
speedup in DESIGN_704 is an UPPER BOUND until a second measured cut exists.
This boot is that cut, and pins the per-layer slope against the fixed per-stage
intercept for the first time. Worth the window even if P1..P3 all hold.

## Test results

76 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean.

- test_weights_arena_union_704.py (13, new): the zero-copy offset invariant;
  zero bytes_to_copy across a flip; residency overhead equal to exactly the
  moved layer's tensors; binding mutates no arena byte; views alias rather than
  copy; shape and dtype conflicts refused; unknown/single rung refused; an
  undersized arena refused before binding; flip symmetry; the quiescence scope
  flag; and the per-rung-layouts can-fail proof.
- 63 existing planner tests unchanged and green.

Boot-script gates verified by execution: refuses without GO, refuses without a
GPU claim, and the argv swap changes exactly one line.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
This is the instrument that unblocks the four-boot retro-prediction gate.

The blocker was not the arming floor and not the graph reserve as a quantity.
It was that the sizer builds budget_posts -- its own named decomposition of the
KV budget -- on every boot and hands it to budget_exhausted_message ONLY when
the budget runs out. The sizer names every term of its arithmetic exactly when
it fails, and discards the naming when it succeeds. So the planner had no
instrument for the reserve it must not double-count, and had to re-derive it
from config in a different process, where it could not even be compared.

Three independent re-derivations, three misses against the live [28,20,16]
boot: subtracting derived_rank_auto_reserve_mib (4160 MiB, called directly)
gave PP2 = 524,620 tokens against a measured 436,766, i.e. +20 percent; the
non-budget slack path gave -3.8 percent; the configured-budget variant gave
-12 percent. The boot actually takes the budget_mib branch because
rank_gpu_memory_mib is set, which is a fourth path again. Re-deriving a number
the sizer already computes is the shape this corpus has paid for repeatedly.

So the fix is the one the gate's meta-lesson implies: create the instrument and
consume it. One info line emits the posts and the final rest_memory on the
success path, in the same GiB units the failure message already uses. The pool
solve then consumes these exactly as it already consumes the sgl-project#676 arming floor,
rather than recomputing them.

Not included, and stated rather than implied: the consuming side. PhasePoolModel
cannot require a sizer_reserve field until a boot has emitted one -- the posts
do not exist in any log on disk today, which is precisely the defect. The next
boot produces them, and the consuming field lands with real values instead of a
placeholder.

Verification: compiles, ruff clean, logger already in scope at module line 168.
model_executor suite 594 passed / 15 failed, and the 15 are PRE-EXISTING --
verified by reverting this file, re-running, and getting the identical
594/15, then restoring. This commit adds one log statement and changes no
control flow.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
Defect I introduced in 84abeb7, caught by its own first metal output.

The instrument labelled each line with self.tp_rank. Under the flip's primary
topology (tp=1, pp=N) every PP stage has tp_rank 0, so all three stages name
stage 0's card. That is the sgl-project#201 defect, and the budget-exhausted path a few
lines below already guards against it in a comment that says so verbatim:
"tp_rank alone makes every PP stage name stage 0's card". I wrote the new line
without applying the warning sitting next to it.

Confirmed by the first boot to carry the line: PP0, PP1 and PP2 all emitted
"[rank 0]". A second boot in the same log happens to show 0/1/2 because that
configuration has distinct tp_ranks, which is exactly the kind of intermittent
correctness that makes the label untrustworthy rather than merely ugly.

Fixed to _rank_vector_index(), the accessor this module already uses for the
same purpose (it resolves server_args.world_rank(pp_rank, tp_rank) and falls
back to tp_rank only when server_args is stubbed). Field renamed to
world_rank so consumers cannot silently read the old meaning.

The already-captured data remains usable: the log prefix (PP0/PP1/PP2) carries
the true stage, so the six lines in docs/dev/DATA_704_budget_posts.txt are
correctly attributable despite the wrong bracket. Captured to the repo because
/root/current_boot.log is a symlink that moves on the next boot and this is the
retro-prediction gate's input data.

First metal values, [28,20,16] budget_mib branch (the branch my external
re-derivations kept missing): weights+runtime 15.688 / 9.826 / 10.449 GiB,
mamba+spec+activation reserve 0.895 / 0.639 / 0.511, GGUF scratch 0.000,
rest 14.472 / 7.894 / 8.375.

Verification: compiles, ruff clean, model_executor 594 passed / 15 failed with
the 15 already established as pre-existing by revert-and-rerun on the previous
commit. Still one log statement, no control flow.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
Defect I introduced in 84abeb7, caught by its own first metal output.

The instrument labelled each line with self.tp_rank. Under the flip's primary
topology (tp=1, pp=N) every PP stage has tp_rank 0, so all three stages name
stage 0's card. That is the sgl-project#201 defect, and the budget-exhausted path a few
lines below already guards against it in a comment that says so verbatim:
"tp_rank alone makes every PP stage name stage 0's card". I wrote the new line
without applying the warning sitting next to it.

Confirmed by the first boot to carry the line: PP0, PP1 and PP2 all emitted
"[rank 0]". A second boot in the same log happens to show 0/1/2 because that
configuration has distinct tp_ranks, which is exactly the kind of intermittent
correctness that makes the label untrustworthy rather than merely ugly.

Fixed to _rank_vector_index(), the accessor this module already uses for the
same purpose (it resolves server_args.world_rank(pp_rank, tp_rank) and falls
back to tp_rank only when server_args is stubbed). Field renamed to
world_rank so consumers cannot silently read the old meaning.

The already-captured data remains usable: the log prefix (PP0/PP1/PP2) carries
the true stage, so the six lines in docs/dev/DATA_704_budget_posts.txt are
correctly attributable despite the wrong bracket. Captured to the repo because
/root/current_boot.log is a symlink that moves on the next boot and this is the
retro-prediction gate's input data.

First metal values, [28,20,16] budget_mib branch (the branch my external
re-derivations kept missing): weights+runtime 15.688 / 9.826 / 10.449 GiB,
mamba+spec+activation reserve 0.895 / 0.639 / 0.511, GGUF scratch 0.000,
rest 14.472 / 7.894 / 8.375.

Verification: compiles, ruff clean, model_executor 594 passed / 15 failed with
the 15 already established as pre-existing by revert-and-rerun on the previous
commit. Still one log statement, no control flow.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…nge, not by copying

Slice 1a showed a rung change moves no weight BYTES (union arena). This wires
the other half: changing which layers a rank EXECUTES, at runtime.

## The model already permits it

Three facts, each verified at file:line, fix the design:

- make_layers (utils/common.py:1970-2010) builds a ModuleList of length
  num_hidden_layers with PPMissingLayer placeholders outside the owned range.
  Layer indices are therefore GLOBAL on every rank, so a boundary change is not
  an index shift.
- start_layer/end_layer are PROPERTIES over mutable _start_layer/_end_layer
  backing fields (models/qwen3_5.py:1452-1457).
- The decoder forward iterates range(self.start_layer, self.end_layer)
  (qwen3_5.py:1483) and reads those properties on EVERY pass.

So a real layer parked outside the active range is simply not executed, and the
boundary change is a range mutation: no module swapping, no reallocation, no
bytes. Composed with the union arena, a rung change moves nothing at all -- a
test pins that every parameter keeps its data_ptr() across a flip while the
executed set changes.

Note qwen3_next.py is NOT the PP model: it asserts is_first_rank and
is_last_rank at :1165. qwen3_5.py is the PP-capable one.

## Load wide, run narrow

At boot a rank builds and loads real modules for the UNION of the ranges it may
occupy, then runs whichever sub-range the ladder selects. The union must be in
force during LOADING too, because weight loading is gated on the same range
(qwen3_5.py:1563-1564, :1707-1708).

## Three failure modes, enforced rather than documented

1. Entering a non-resident range is SILENTLY WRONG, not loud. PPMissingLayer is
   a pass-through (layers/utils/common.py:109-127), so executing a range whose
   weights never loaded yields plausible output from a shallower model instead
   of an error. Every flip verifies residency first, and a hole inside the
   claimed union is caught at construction.

2. A half-applied boundary is worse than no flip. Dependent structures (the KV
   pool's full_attention_layer_ids filter at
   model_runner_kv_cache_mixin.py:2466-2470, the GDN state maps) are keyed by
   the owned range. Observers run INSIDE the flip; if one raises, the range is
   rolled back and the flip fails.

3. The third came out of a test failure rather than foresight. The actuator
   originally ASSUMED the model already sat at its current rung. Under load-wide
   it does not -- it arrives wide -- so the constructor now performs the
   narrowing step explicitly. Left as it was, the actuator's belief and the
   model's actual range would have disagreed from the first forward, silently
   executing a layer the rank did not own. Pinned by
   test_construction_performs_the_narrowing_step.

World-level tiling is validated separately, because it cannot be checked
locally: a gap silently DROPS layers and an overlap silently COMPUTES THEM
TWICE, while every individual rank's range looks entirely sensible.

## Scope, unchanged

Flips at QUIESCENCE only, and the refusal names why: a GDN layer's
per-sequence recurrent state travels with the layer and slice 1a moves none of
it, so an in-flight sequence would continue against state that is no longer
there. Live-state transfer is slice 1b. arena_refill is untouched and remains
correct for the phase flip, whose two layouts are disjoint tensor sets with no
useful union.

## Test results

90 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean.

- test_layout_boundary_704.py (14, new): the active range selects what runs
  while an unowned layer stays resident; no storage moves across a flip; flip
  back restores exactly; refusal when not quiescent and when the target is not
  resident; a hole in the union caught at construction; unknown rung refused;
  same-rung flip is a no-op; world tiling rejects gap/overlap/short-cover; the
  narrowing step; observer notification with rollback on failure; and an
  end-to-end test that the union arena and the boundary actuator compose.
- test_weights_arena_union_704.py (13) unchanged, including the can-fail proof
  that per-rung layouts displace 14 of 56 shared tensors.
- 63 planner tests unchanged and green.

Remaining for a GPU window: boot-time union load and registration of the
dependent-structure observers. The actuator itself is complete.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… the budget post is short

The operator flagged the metal mamba posts as ~14 percent below my derived
50.85 MiB/GDN-layer and suspected an over-charged spec-slot sub-term in my
model. That suspicion is wrong, and settling it needed no new instrument:
MambaPool already logs every sub-term at construction (memory_pool.py:756-765).

Read off the live boots, per GDN layer, all three ranks: allocated 51.20 MiB
against my derived 50.85 -- inside the log's own 2-decimal GiB rounding (four
terms at 0.01 GiB each = +-1 MiB/layer). The individual terms match as well:
ssm_state is exactly layers x 13 slots x 1.5 MiB (21 -> 0.400, 15 -> 0.286,
12 -> 0.229 GiB) and intermediate_ssm is exactly layers x 5 spec slots x 4
draft x 1.5 MiB (30.0 MiB/layer on all three). The spec-slot count of 5 is
correct, not over-charged.

What IS off is the sizer's own budget post. post/allocated = 0.852 on all
three ranks -- exactly constant, so a systematic formula divergence rather
than rounding or noise. The sizer hands out roughly 158 / 111 / 89 MiB more
mamba memory than it charges to its budget.

Consequence for the retro-prediction gate, and it changes the plan: the pool
solve must consume the ALLOCATED figure from the Mamba Cache line, not the
budget post. Feeding PhasePoolModel from the posts alone -- which is what I
had planned -- would have baked a systematic 150 MiB/rank optimism into the
model.

Open sub-item, recorded rather than guessed: which term in
handle_max_mamba_cache produces the 0.852. It is exactly constant, so it is
one factor and not an accumulation, but naming it needs a read of that
function rather than more arithmetic from outside. It does not block the gate,
because the gate now consumes the allocation.

Docs only.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…brators, and one of them is none

PrefillTiming models a stage as fixed_ms[r] + ms_per_layer[r] * n_r, and one
measured cut cannot separate the two -- both fit the single point exactly.
prefill_timing_from_measurement therefore defaults fixed_ms to zero, the
OPTIMISTIC end of the family, which is exactly why every speedup in DESIGN_704
is an upper bound rather than a prediction. Two cuts resolve it per rank by
elimination.

The arithmetic is four lines. The reason this is a module is that the result
can be confidently wrong in two specific ways, and doing the sums BEFORE the
window changed what the boot should ask for.

## The slice-1a pair is three calibrators at once

  rank0: 28 -> 29  (dn=+1)  expected dt ~1.76 ms  -- weak, the binding cost
  rank1: 20 -> 19  (dn=-1)  expected dt ~7.74 ms  -- strong
  rank2: 16 -> 16  UNCHANGED                      -- NOT CALIBRATED AT ALL

rank2 holds 16 layers in both cuts, so the pair carries no information about
its intercept. The solver refuses to report one rather than emitting the
fixed_ms=0 fallback, which at a call site is indistinguishable from a
measurement.

## Sample count is a precondition, not an afterthought

The slope is a difference of two means over the layer delta, so
stderr(s) = sqrt(2)*sigma/|dn|, and dn=1 puts the entire per-stage noise onto
the slope. Because rank0's slope is small it is far dearer to pin than rank1's:

  per-chunk SD | chunks for 10% on rank0 (1.757) | on rank1 (7.740)
          1 ms |    65                           |   4
          2 ms |   260                           |  14
          3 ms |   584                           |  31
          5 ms |  1620                           |  84

One max-length prompt is 327680/512 = 640 chunks, which clears the 584 needed
at SD=3 ms -- so ONE full-length prefill per arm suffices for rank0 at 10%,
provided the per-chunk SD is measured and reported rather than assumed. The
boot ticket now demands mean, per-chunk SD, N and the standard error per stage;
without the SE the intercept cannot be gated and must not be published.

## Two refusals, because a confident wrong number is worse than none

- A NEGATIVE intercept is reported, not clamped. It means the linear model does
  not hold across the pair (most likely a non-constant per-layer cost), and a
  clamped fit looks calibrated while being wrong. Only floating-point dust is
  rounded to zero.
- STRUCTURAL refusals precede data-dependent ones. Whether a pair can calibrate
  a rank at all is a property of the two LAYOUTS; whether the numbers come out
  sane is a property of the MEASUREMENT. Reporting the second while the first
  is outstanding sends the reader to inspect timings when the pair was never
  capable of the answer. Found by test failure: the rank2 refusal was being
  masked by rank0's negative intercept.

## Also in this commit, per review

- DESIGN_704 sec 3.9 failure mode 3 is restated as a CONTRACT rather than a bug
  anecdote: the actuator OWNS the active range and never infers it; there is
  exactly one writer of _start_layer/_end_layer once an actuator exists, and any
  other writer is a defect. The anecdote is kept in one parenthesis because the
  next reader needs the rule, not the story.
- The model-class correction is carried explicitly: models/qwen3_next.py is NOT
  the PP model (it asserts is_first_rank and is_last_rank at :1165);
  models/qwen3_5.py is the PP-capable class. Deduplicated to one statement.

## Test results

99 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean,
boot script syntax-checked.

- test_timing_calibration_704.py (9, new): synthetic round-trip recovers a known
  slope and intercept exactly; an unchanged rank is refused; the allowed
  fallback is flagged and stays optimistic with stderr None (never 0.0);
  slope uncertainty scales as 1/|dn|; the precision gate rejects rank0 and
  passes rank1 at the same noise; samples_needed sizes the run and a larger
  delta needs fewer; identical cuts, negative intercepts and mismatched rank
  counts all refused.
- 90 existing tests unchanged and green.

No rung table re-derived; Slot-2's retro-gate still owns that. Boot ticket
remains non-self-arming and queued behind F4-r4.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…icit solver output

Two independent calibrations converged on the same blind spot from the same
structural cause, so it is now a first-class output rather than a footnote each
of us rediscovers.

Pool side (sgl-project#702 rev5): two metal points give two equations for three ranks.
Rank1 binds at [28,20,16] and rank0 at [32,16,16]; rank2 binds at neither, so
its free constant is bounded, not identified. Timing side
(planner/timing_calibration.py): the slice-1a pair [28,20,16] <-> [29,19,16] is
three different calibrators -- rank0 weak, rank1 strong, rank2 carrying ZERO
information because its layer count does not move.

Every candidate cut so far keeps rank2 at 16 layers, and the [33,15,16]
discriminator does too. A window spent on it cannot identify rank2 on either
axis. calibration_coverage() makes that visible BEFORE the window is spent.

The key distinction the type encodes: a rank whose layer count does not move is
UNIDENTIFIED, not weakly levered. Its regressor column is identically zero, so
no sample size recovers it, and chunks_for_target() returns infinity rather
than a large finite number that would invite someone to buy more chunks.

The sample-demand formula is the two-sample standard error on a difference of
boot means, N = (sqrt(2) * SD / (target * |dn| * ms_per_layer))^2, which
reproduces the timing analysis independently: 583 chunks for rank0 and 30.0 for
rank1 against its 584 and 31, at SD=3 ms and a 10 percent target. Agreement
from two directions on numbers neither fitted to the other.

publishable_intercept() gates an intercept before it reaches the solver:
refused without a standard error, refused at N=0, and refused when |value| < 2
SE. Refusing beats emitting fixed_ms=0, because downstream a zero reads as a
MEASUREMENT that the fixed cost is absent when in fact the pair could not see
it. This is the same refusal the timing solver already makes, expressed as a
shared gate so the two solvers cannot disagree about what counts as measured.

suggest_rank_moving_arm() proposes arms in BOTH directions when a rank's term
is load-bearing, since one direction confounds the term with any monotone
drift.

Tests: 9, red first. ruff clean. Nothing rig-specific: coverage is a property
of the two layer vectors, sample demand a property of the lever and the noise.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… both become guards

A boundary change is not finished when the range changes. Two structures follow
the owned layer range and NEITHER raises on its own if it does not follow, so
each is now an observer that refuses and rolls the flip back.

## (a) CUDA graphs bake the executed layer set -- and this corrects my own claim

The decode graph captures the model's own forward
(decode_cuda_graph_runner.py:1770), which iterates
range(self.start_layer, self.end_layer). A CUDA graph records actual kernel
launches, so the executed set is fixed AT CAPTURE TIME and a replay ignores any
later change to _start_layer/_end_layer.

After an unguarded flip the model reports the new range while every graph
replay still runs the old one. That is the worst shape available here: the
layer count silently reverts on exactly the fast path, and only under replay,
so an eager smoke test would show the flip working.

CORRECTION TO DESIGN_704 sec 3.7. I argued the arena's advantage over
reallocation was that it "keeps captured CUDA graphs valid across a rung
change". Wrong as stated. The arena keeps parameter ADDRESSES stable, which the
graph's captured pointers do need, but the graph also bakes the executed layer
SET -- so a rung change requires recapture EITHER WAY. The arena-vs-realloc
fork narrows accordingly: recapture is common to both, and the arena's
remaining advantages are avoiding the weight copy and avoiding reallocation.
Slice 2 must therefore measure recapture as a cost of EVERY rung change, not as
a penalty unique to the realloc design.

## (b) The KV and mamba pools are built for a span and indexed off its base

Their layer id lists are filtered to the owned range at build time
(model_runner_kv_cache_mixin.py:2460-2470) and rows are addressed as
layer_id - pool.start_layer (memory_pool.py:1576, :2889). Two failures follow:

- a layer newly activated outside the built span has NO ROWS AT ALL -- no KV
  for a full-attention layer, no mamba slot for a linear one;
- the indexing BASE must not move. If a downstream rank's pool were rebuilt
  with the new start (rank1: 28 -> 29), every cached row would shift by one
  layer and be silently misattributed.

So "load wide, run narrow" applies to the caches too: build the pools over the
UNION, at a cost of one extra layer's rows per boundary that moves. In the
slice-1a pair rank0 keeps start=0 and rank2 keeps start=48, so rank1 is the
only rank whose base moves -- the one place a naive rebuild would corrupt
silently.

## Test results

104 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean.

test_layout_boundary_704.py grows to 19. The graph guard carries a HAZARD-FIRST
test: it demonstrates that a graph captured at rung A really does keep replaying
rung A's layers after an unguarded flip, so the guard test below it cannot pass
vacuously. Also: graphs with no recapture hook refuse the flip and roll back; a
range escaping the pools' built span is refused; pools built over the union
accept every rung; and the indexing-base arithmetic is pinned directly.

Remaining for a GPU window is unchanged: boot-time union load (weights, KV and
mamba pools) plus registration of these observers against the real registries.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ytes -> tokens

Completes the instrument chain the retro-prediction gate needs.

The chain is: budget - sum(budget_posts) = rest, then rest MINUS a per-rank
reserve becomes available_bytes, then // cell_size becomes
max_total_num_tokens. The first link I emitted in 84abeb7 and it is live on
metal. This is the last one.

The reserve between rest and available_bytes is the only term emitted nowhere,
and it is not small or uniform. Backed out of the live [28,20,16] boot it is
8,848 / 3,818 / 5,164 MiB across the three stages -- a 2.3x spread. It tracks
per-rank CUDA-graph capture, which depends on the shard, so config cannot
predict it; my three external re-derivations missed by +20, -3.8 and -12
percent respectively.

planner/plan.py's auto_reserve_mib docstring states the value is one "which the
boot itself derives and logs". The deriving is real; the logging was not. A
172 MB boot log contains only the server_args echo (rank_auto_reserve_mib=
'auto', rank_user_reserve_mib=1024) and no derived per-rank figure. That is the
third instance this session of a term the sizer computes and discards, after
the budget posts and the lumped mamba post.

With this line the reserve is recoverable as (rest - available_bytes) with no
fourth re-derivation, and cell_size is published alongside so the
bytes-to-tokens conversion can be checked rather than assumed. Independently
confirmed en route: rest = budget - sum(posts) reproduces the logged rest
exactly on all three ranks (14.472 / 7.894 / 8.376 GiB), so that link needs no
further validation.

Verification: compiles, logger already in scope at line 72, model_executor 594
passed / 15 failed with the 15 established pre-existing earlier by
revert-and-rerun. The 3 ruff findings in this file are also pre-existing --
they sit at lines 372/375/380, ahead of this edit, and reproduce identically on
HEAD. One log statement, no control flow change.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ytes -> tokens

Completes the instrument chain the retro-prediction gate needs.

The chain is: budget - sum(budget_posts) = rest, then rest MINUS a per-rank
reserve becomes available_bytes, then // cell_size becomes
max_total_num_tokens. The first link I emitted in 84abeb7 and it is live on
metal. This is the last one.

The reserve between rest and available_bytes is the only term emitted nowhere,
and it is not small or uniform. Backed out of the live [28,20,16] boot it is
8,848 / 3,818 / 5,164 MiB across the three stages -- a 2.3x spread. It tracks
per-rank CUDA-graph capture, which depends on the shard, so config cannot
predict it; my three external re-derivations missed by +20, -3.8 and -12
percent respectively.

planner/plan.py's auto_reserve_mib docstring states the value is one "which the
boot itself derives and logs". The deriving is real; the logging was not. A
172 MB boot log contains only the server_args echo (rank_auto_reserve_mib=
'auto', rank_user_reserve_mib=1024) and no derived per-rank figure. That is the
third instance this session of a term the sizer computes and discards, after
the budget posts and the lumped mamba post.

With this line the reserve is recoverable as (rest - available_bytes) with no
fourth re-derivation, and cell_size is published alongside so the
bytes-to-tokens conversion can be checked rather than assumed. Independently
confirmed en route: rest = budget - sum(posts) reproduces the logged rest
exactly on all three ranks (14.472 / 7.894 / 8.376 GiB), so that link needs no
further validation.

Verification: compiles, logger already in scope at line 72, model_executor 594
passed / 15 failed with the 15 established pre-existing earlier by
revert-and-rerun. The 3 ruff findings in this file are also pre-existing --
they sit at lines 372/375/380, ahead of this edit, and reproduce identically on
HEAD. One log statement, no control flow change.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ady for the boot

Derives nothing. Reads the four instrumented terms and refuses what it cannot
read. That discipline is the session's most expensive lesson: three external
re-derivations of the sizer's arithmetic missed the measured boot by +20, -3.8
and -12 percent, because the reserve tracks per-rank CUDA-graph capture and
config cannot see it.

RankInstruments carries one PP stage as emitted numbers: budget, the three
budget posts, rest, mamba allocated, available_bytes, cell_size, tokens, the
per-layout arming floor and the layer split.

verify_sizing_chain asserts the sizer's own identity, rest = budget -
sum(posts). It reconciles on all three live stages within 0.9 MiB against a 3
MiB tolerance set by the posts' three-decimal-GiB rounding. This is the one
link validatable without a new boot, and it pins that the emission is COMPLETE:
an unemitted post would surface here as a residue.

recover_reserve_mib recovers the reserve as rest - available_bytes rather than
modelling it: 8,848 / 3,818 / 5,164 MiB on the live boot, a 2.3x spread.

Two traps encoded as behaviour rather than comments:

* mamba_charge_mib returns the ALLOCATION, not the budget post. The post
  under-charges by a constant 0.852 on every rank, so a solve fed from posts
  carries ~150 MiB/rank of systematic optimism -- which was the original plan
  for this field before the discrepancy was resolved.
* the arming floor is NOT subtracted separately. It is held back after the
  profiler, so it already sits inside the recovered reserve; charging it again
  would understate every pool by ~2 GiB. A test pins that the floor fits inside
  the reserve and that the prediction still lands on the boot.

predict_tokens_for_cut takes no default reserve and raises when it is missing,
because the reserve does not transfer between layouts. An unbooted cut is an
extrapolation and the call site has to say so.

world_pool_tokens applies the PP min-rule and reports the binding stage.

Retro-prediction status: the incumbent reproduces from instruments alone
(436,766, chain reconciled). The remaining three boots close the four-boot gate
the moment a boot carrying 2a6305d emits available_bytes directly -- until
then that field is pinned from tokens x cell_size, which the instrument will
confirm or refute rather than being assumed correct.

Tests: 10, red first. Planner suite 2643 passed, 2 failed -- the pre-existing
test_rejected_evidence_pins pair. ruff clean.

Prior queue item for the record: the sgl-project#701 slice-3 rework landed earlier as
845ac92 (five gate holes, 24 tests, cross-pass falsifiers against a real
PrefillAdder).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ntics inherited

Slot-3's LadderInputs already consumes PhasePoolModel and requires an
arming_floor_for provider, but it builds the model itself in pool_model_for.
That is two solvers which must agree and are computed twice -- the shape this
corpus keeps paying for. solve_rung_pool is the single surface the rung table
and the boundary actuator call instead.

Inherited rather than restated:

* NO DEFAULT RESERVE. The per-rank reserve tracks CUDA-graph capture and does
  not transfer between layouts -- 3,818 / 5,164 / 8,848 MiB across three stages
  of one boot, a 2.3x spread. A rung with no reserve source raises, naming the
  spread, instead of returning a number that would read as measured.
* provenance is a FIELD, so an extrapolated rung cannot be mistaken for a
  measured one further down the ladder; the extrapolated path also carries an
  explicit caveat string.
* calibration coverage is returned whenever a reference cut is given, and every
  unidentified rank produces a caveat naming it. Relevant immediately: every
  candidate cut so far, [33,15,16] included, keeps rank2 at 16 layers, so rank2
  is unidentified on both the pool and timing axes.

Two metal-derived corrections are built in rather than left to the caller:

* mamba is charged at the ALLOCATION, not the budget post -- the post
  under-charges by a constant 0.852 on all three ranks, worth ~150 MiB/rank of
  systematic optimism.
* the arming floor is NOT subtracted here. It is held back after the profiler
  so it already sits inside the recovered reserve; charging it twice would
  understate every rung by ~2 GiB. It is still resolved and required, because a
  rung whose floor cannot be produced is a rung nobody has sized.

Providers accept either a callable or a mapping so this can match existing call
sites without a second surface appearing to serve them.

One test caught itself: the unbooted-rung case originally tripped on a missing
rest before reaching the reserve check, so it was not testing what it named.
Fixed to supply rest and withhold only the reserve.

Tests: 9, red first. Planner suite 2652 passed, 2 failed -- the pre-existing
test_rejected_evidence_pins pair. ruff clean.

Coordination note: I could not reach Slot-3 by name from this session to agree
the signature first. The providers are deliberately callable-or-mapping and the
field names are positional-free, so his call sites can be matched without a
breaking change; if he wants per_stage keyed by stage rather than positional I
will adapt rather than have him adapt.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ive requirements

FOUR-BOOT GATE VERDICT: pinned available_bytes CONFIRMED at the binder,
REFUTED at the non-binders, and the refutation is fully explained.

  PP0  pinned 6,261,477,376  metal 8,526,565,376  -26.57%
  PP1  pinned 4,472,483,840  metal 4,741,928,960   -5.68%
  PP2  pinned 3,577,987,072  metal 3,575,365,632   +0.07%   <- binder

I had pinned available_bytes = pool x cell for EVERY stage, which assumes every
stage is exactly saturated. Under the min-rule only the BINDER is; the others
carry slack. So the pin was determinate only at PP2 and is confirmed there to
0.07 percent. Same pattern in the reserve: mine 5,163.8 MiB vs metal 5,166.3 at
PP2 (-0.05 percent), but +7.2 and +32.3 percent at PP1/PP0. The method is
validated where it was decidable and the error elsewhere is the min-rule, not a
modelling defect. World pool 436,446, binding rank PP2, both as the boot
reports.

BONUS SETTLED: cell_size is 14336 / 10240 / 8192 = exactly 7 / 5 / 4 attention
layers x 2048 B. The sgl-project#702 attention-divisor now rests on the running system's
ABSOLUTE values, not on a ratio, so the multiple-of-4 degeneracy that made
[28,20,16] and [32,16,16] non-discriminating no longer matters.

CORRECTION to my 0.852 claim, on F4-r4's challenge, which was right. The budget
post is "mamba state pool + speculative intermediate state + prefill activation
reserve" while the allocated line sums conv + ssm + intermediate_ssm +
intermediate_conv. Those are not the same quantity, so "the mamba post
under-charges mamba by 14.8 percent" is WITHDRAWN. What survives is stronger in
kind: the post covers a nominal SUPERSET of the allocated line and is
nevertheless SMALLER by 0.155 / 0.111 / 0.089 GiB, which cannot be legitimate.
The gap is real and at least that size; which sub-term carries it needs the
post's three components emitted separately -- the same missing-instrument fix a
third time. The solver charges the ALLOCATION either way, because that is what
the allocator took.

Reserve figures corrected throughout to metal: 6,688 / 3,561 / 5,166 MiB, a
1.88x spread (my earlier 8,848 / 3,818 / 5,164 back-out was the non-binder
artifact above).

ENTRY POINT reshaped to Slot-3's five requirements:
 1. attn_counts DERIVED from layer_families via PoolContext, never passed --
    an inconsistent pair is now unrepresentable, and it is the sgl-project#702 divisor.
 2. arming_floor_for required, no default (E3), and still NOT subtracted: it
    lives inside the recovered reserve and charging it twice would understate
    every rung by ~2 GiB.
 3. binding stage and per-stage caps RETURNED, so the controller does not
    recompute them in a second code path.
 4. PoolContext is built once by the caller; each solve is pure arithmetic with
    no NVML, census or config parsing, so an enumeration over thousands of cuts
    is cheap.
 5. KV cell from kv_mib_per_token_per_attn_layer_from_config, converged VERBATIM
    from feat/704-prefill-ladder into pp_cut together with kv_dtype_width_bytes
    and its width table, so there is one implementation rather than two that
    must agree. It yields exactly 2048 B, matching metal.
 Optional explain=True returns per-stage weights/mamba/floor/free-for-KV terms.

Tests: 21 across the two suites, red first. Planner suite 2654 passed, 2 failed
-- the pre-existing test_rejected_evidence_pins pair. ruff clean.

Slot-3 was not reachable by name from this session; providers accept callable or
mapping so his call sites can be matched without a breaking change.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
Builds the instrument this ticket has now needed three times (budget posts,
then available_bytes, now the post's own parts), and closes the recurring
reconciliation blocker.

The post has been labelled "mamba state pool + speculative intermediate state +
prefill activation reserve" all along while emitting ONE number. That lump is
why the illegitimate gap could not be attributed: the post nominally covers a
SUPERSET of the "Mamba Cache is allocated" line -- it adds the prefill
activation reserve -- yet measures 0.155 / 0.111 / 0.089 GiB LESS on
PP0/PP1/PP2. A term covering more cannot legitimately measure less, but with a
single number there is no way to say which of the three carries it. Next boot
there will be.

decompose_mamba_budget_post is a pure function, so it is testable without a
boot. Two properties are pinned separately: the parts SUM to the lump exactly
(a decomposition that changed the total would silently move the KV budget), and
the residual is NAMED as the state pool rather than left anonymous. A negative
residual is reported, not clamped -- a negative state pool is precisely the
error this instrument exists to surface, and clamping would hide it. With no
components the lump is returned unchanged, so the other budget branches are
untouched.

Wired at all five sites that actually move the number: four speculative-
intermediate subtractions across the branches, plus the prefill activation
reserve.

One self-inflicted defect caught and fixed before push. The accumulator started
as a bound method, which broke eleven ceiling-fit tests immediately: their
SimpleNamespace doubles bind handle_max_mamba_cache as a plain function and
cannot resolve self._note_mamba_component. That is the sgl-project#624 stub-drift class
with me on the wrong side of it. Fixed by making the helper module-level and
duck-typed -- production must not demand more of a stand-in than the behaviour
under test needs -- and the reason is recorded in its docstring so it does not
get "tidied" back into a method.

Tests: 6, red first. model_executor suite 600 passed / 15 failed, and the 15 are
the SAME pre-existing set as before this change (594 + the 6 new = 600). ruff
clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…t#706 lanes) into the tool-call argument-loss fix

Brings the branch onto the commit the 30030 tree currently holds so the
deploy is a fast-forward. No conflicts with the adapter or detector
changes -- the incoming commits touch KV sizing and mamba slicing.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… the instrument to fix it

Desk model of the per-rank reserve, on existing metal. Verdict: UNDETERMINED on
current data, but two things ARE settled and the recommended next step is not a
fit.

Correction to my own gate-close reasoning first. I had said only the BINDER's
reserve is determinate. That is true of the PINNED available_bytes (derived as
pool x cell) and false of the EMITTED one: the boot publishes each rank's own
capacity -- 594,766 / 463,079 / 436,446 tokens, all different -- and the world
pool is the min taken afterwards. So reserve = rest - available_bytes is
determinate on EVERY rank. Three points, not one:

  PP0  28 layers / 7 attn / 21 gdn / 31,800 MiB budget -> 6,687.8 MiB
  PP1  20 / 5 / 15 / 18,800                            -> 3,561.2
  PP2  16 / 4 / 12 / 19,800                            -> 5,166.3

FALSIFIED, from three points and without any fit: single-factor positive
scaling in layers. PP1 and PP2 are both 3080s; PP1 carries MORE layers (20 vs
16) and has LESS reserve (3,561 vs 5,166). Reserve is non-monotone in layers,
so no a + b*layers with b > 0 can produce the triple. Same for attn and gdn,
which are proportional here.

CONFOUNDED, exactly rather than statistically: on the incumbent attn =
layers/4 on every rank (28/7, 20/5, 16/4 all exactly 4.0) and gdn = 3*attn. The
regressors are perfectly collinear, so the three hypotheses give IDENTICAL
residuals to the digit (+452.6 / -1357.9 / +905.3, max 1358). Budget-only fits
better but still poorly (max 734). Two-factor forms have three parameters for
three points, fit exactly with zero information, and are unidentifiable -- they
must not be published as models. This is the same degeneracy that made
[28,20,16] and [32,16,16] non-discriminating for the pool divisor; there it was
broken by an absolute value, and no absolute value exists here because the
reserve is not emitted decomposed.

RECOMMENDATION: the fourth instrument, not a curve. Every previous blocker in
this ticket dissolved the same way -- the sizer computed the term and did not
emit it (budget posts, available_bytes, the mamba post's components).
derived_rank_auto_reserve_mib returns 4,160 MiB UNIFORMLY when called with this
boot's arguments while metal shows 6,688 / 3,561 / 5,166, so the held-back
amount is not that function's output alone and fitting three points would be
modelling a black box the process already knows. Emitting what is actually held
back, per rank, settles the form in ONE boot.

If a boot must discriminate instead, it needs a non-multiple-of-4 cut.
[33,15,16] qualifies (ratios 4.125 / 3.75 / 4.0); per-hypothesis predictions are
published so it becomes a free calibration point either way -- PP0 separates the
forms by 165-455 MiB, PP1 by up to 219. PP2 is predicted identical by three of
four forms, which is the rank2 blind spot again. Note [33,15,16] is no longer
needed for the pool divisor (settled on absolute grounds by cell_size), so this
is now its only reason to boot.

Discipline retained: this note does not license carrying a reserve across
layouts. It quantifies how wrong that is -- up to ~1.6 GiB per rank between two
ranks of the SAME card type in a single boot, larger than most rung-to-rung pool
deltas, which is precisely why extrapolated rungs cannot gate a window.

Docs only.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
The fourth instrument, and the one the reserve analysis asked for instead of a
curve fit.

The quantity between the profiler's rest and what the configurator actually
receives lands on metal at 6,688 / 3,561 / 5,166 MiB, while
derived_rank_auto_reserve_mib returns 4,160 UNIFORMLY for the same boot's
arguments. So the holdback is not that function's output. It is also not
fittable from what exists: DESIGN_704 showed single-factor layer models are
falsified outright by non-monotonicity (PP1 carries more layers than PP2 yet
holds back less), and that layers/attn/gdn are EXACTLY collinear on any
multiple-of-4 cut, giving identical residuals to the digit. Fitting three
collinear points would have been modelling a black box the process already
knows.

Emitted at _config_from_budget, whose own comment calls it "the one funnel every
sizing path reaches" -- so this catches every sizing path rather than one
branch. The holdback is precisely the delta across _seam_adjusted_budget there,
which is why no external re-derivation reproduced it (+20, -3.8, -12 percent on
three attempts): it is a seam adjustment, not a graph-capture reserve, and I had
been modelling the wrong quantity entirely.

budget_holdback_mib and budget_holdback_fraction are pure, so they are tested
without a boot, and they REPORT rather than sanitise. A negative holdback means
the seam handed budget back -- a real and loud state -- and clamping it to zero
would hide exactly the accounting-error class this ticket has spent four
instruments chasing. The fraction helper returns None on a zero budget rather
than dividing, so no caller has to guard it.

A test pins the three metal reserves against the helper, so the instrument and
the analysis cannot drift apart.

Tests: 6, red first. model_executor suite 606 passed / 15 failed -- the same
pre-existing set (600 + the 6 new). ruff clean. One log statement and two pure
functions; no control flow change.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ently disabled

Defect I introduced in 5f3e61f, caught in pre-boot review.

budget_exhausted_message derives its --max-running-requests-ceiling advice by
summing the posts named MAMBA_BUDGET_POST. Once the post is emitted as three
named parts that sum matches nothing, so the hint vanished from the refusal
message EXACTLY on the boots that carry the new instrument -- the worst possible
correlation, since those are the boots someone would be debugging.

Fixed with mamba_post_total_gb, which accepts both shapes (the lump and the
parts) so callers stop matching a name, plus MAMBA_POST_PART_NAMES as the one
declaration of what the decomposition emits. decompose_mamba_budget_post now
takes the residual's name from that tuple, so the emitter and the summers cannot
drift apart -- drift is precisely how the hint broke.

A test pins that the emitted names equal the declared set, and another that
unrelated posts (weights, GGUF scratch) are not swept into the total.

Tests: 3 more, red first (9 in the file). model_executor 609 passed / 15 failed
-- the same pre-existing set (606 + 3). ruff clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
Builds the instrument this ticket has now needed three times (budget posts,
then available_bytes, now the post's own parts), and closes the recurring
reconciliation blocker.

The post has been labelled "mamba state pool + speculative intermediate state +
prefill activation reserve" all along while emitting ONE number. That lump is
why the illegitimate gap could not be attributed: the post nominally covers a
SUPERSET of the "Mamba Cache is allocated" line -- it adds the prefill
activation reserve -- yet measures 0.155 / 0.111 / 0.089 GiB LESS on
PP0/PP1/PP2. A term covering more cannot legitimately measure less, but with a
single number there is no way to say which of the three carries it. Next boot
there will be.

decompose_mamba_budget_post is a pure function, so it is testable without a
boot. Two properties are pinned separately: the parts SUM to the lump exactly
(a decomposition that changed the total would silently move the KV budget), and
the residual is NAMED as the state pool rather than left anonymous. A negative
residual is reported, not clamped -- a negative state pool is precisely the
error this instrument exists to surface, and clamping would hide it. With no
components the lump is returned unchanged, so the other budget branches are
untouched.

Wired at all five sites that actually move the number: four speculative-
intermediate subtractions across the branches, plus the prefill activation
reserve.

One self-inflicted defect caught and fixed before push. The accumulator started
as a bound method, which broke eleven ceiling-fit tests immediately: their
SimpleNamespace doubles bind handle_max_mamba_cache as a plain function and
cannot resolve self._note_mamba_component. That is the sgl-project#624 stub-drift class
with me on the wrong side of it. Fixed by making the helper module-level and
duck-typed -- production must not demand more of a stand-in than the behaviour
under test needs -- and the reason is recorded in its docstring so it does not
get "tidied" back into a method.

Tests: 6, red first. model_executor suite 600 passed / 15 failed, and the 15 are
the SAME pre-existing set as before this change (594 + the 6 new = 600). ruff
clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
The fourth instrument, and the one the reserve analysis asked for instead of a
curve fit.

The quantity between the profiler's rest and what the configurator actually
receives lands on metal at 6,688 / 3,561 / 5,166 MiB, while
derived_rank_auto_reserve_mib returns 4,160 UNIFORMLY for the same boot's
arguments. So the holdback is not that function's output. It is also not
fittable from what exists: DESIGN_704 showed single-factor layer models are
falsified outright by non-monotonicity (PP1 carries more layers than PP2 yet
holds back less), and that layers/attn/gdn are EXACTLY collinear on any
multiple-of-4 cut, giving identical residuals to the digit. Fitting three
collinear points would have been modelling a black box the process already
knows.

Emitted at _config_from_budget, whose own comment calls it "the one funnel every
sizing path reaches" -- so this catches every sizing path rather than one
branch. The holdback is precisely the delta across _seam_adjusted_budget there,
which is why no external re-derivation reproduced it (+20, -3.8, -12 percent on
three attempts): it is a seam adjustment, not a graph-capture reserve, and I had
been modelling the wrong quantity entirely.

budget_holdback_mib and budget_holdback_fraction are pure, so they are tested
without a boot, and they REPORT rather than sanitise. A negative holdback means
the seam handed budget back -- a real and loud state -- and clamping it to zero
would hide exactly the accounting-error class this ticket has spent four
instruments chasing. The fraction helper returns None on a zero budget rather
than dividing, so no caller has to guard it.

A test pins the three metal reserves against the helper, so the instrument and
the analysis cannot drift apart.

Tests: 6, red first. model_executor suite 606 passed / 15 failed -- the same
pre-existing set (600 + the 6 new). ruff clean. One log statement and two pure
functions; no control flow change.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…ently disabled

Defect I introduced in 5f3e61f, caught in pre-boot review.

budget_exhausted_message derives its --max-running-requests-ceiling advice by
summing the posts named MAMBA_BUDGET_POST. Once the post is emitted as three
named parts that sum matches nothing, so the hint vanished from the refusal
message EXACTLY on the boots that carry the new instrument -- the worst possible
correlation, since those are the boots someone would be debugging.

Fixed with mamba_post_total_gb, which accepts both shapes (the lump and the
parts) so callers stop matching a name, plus MAMBA_POST_PART_NAMES as the one
declaration of what the decomposition emits. decompose_mamba_budget_post now
takes the residual's name from that tuple, so the emitter and the summers cannot
drift apart -- drift is precisely how the hint broke.

A test pins that the emitted names equal the declared set, and another that
unrelated posts (weights, GGUF scratch) are not swept into the total.

Tests: 3 more, red first (9 in the file). model_executor 609 passed / 15 failed
-- the same pre-existing set (606 + 3). ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…-pointed

This was assigned as a pin-only line-number fix, on the strength of a
diagnosis I gave in the previous report. That diagnosis was WRONG and is
withdrawn: I said the pinned assert had moved to server_args.py:13693-13694
and changed shape to view.disable_overlap_schedule. That site is
_validate_mamba_no_buffer, an unrelated mamba assert. I read a bare
disable_overlap_schedule grep hit as the pinned guard without reading its
context -- the same infer-from-a-hit error sgl-project#635 named ("a constant is evidence
about the code that reads it and nothing else").

Reading the real guard shows the row was not merely mis-pointed. The single
conjunction the row describes has been SPLIT INTO TWO statements that went in
OPPOSITE directions:

  server_args.py:17583  if self.pp_size > 1:
  server_args.py:17584-17586  assert self.disable_overlap_schedule
  server_args.py:17598  assert self.speculative_algorithm is None
                               or self.enable_phase_flip

1. The speculation half is still a hard assert, but it GREW AN EXEMPTION:
   "or self.enable_phase_flip". Per the comment at :17587-17597, a phase-flip
   instance runs PP for prefill with no draft worker built at all and arms
   speculation on the TP decode stack at cutover, so the combination is
   enforced by construction instead of refused.

2. The overlap half is now effectively an auto-disable. arg_groups/
   overrides.py:2163 _pipeline_parallel_overlap_disable sets
   disable_overlap_schedule=True and logs a warning in a post-process pass
   that runs BEFORE the assert, so that assert can no longer fire.

So the row's claim -- "a hard assert, not an auto-disable ... every PP number
is therefore a no-spec number" -- was wrong in both halves, and wrong in the
direction that matters to this strand: it forbade on paper the exact
configuration sgl-project#704 is built on (PP=3 prefill flipping to TP=3+EAGLE decode).
Updating only the line number would have re-pointed a false verdict at the
code that falsifies it.

Row corrected: BLOCKED for PLAIN pipeline parallelism, EXEMPT under
--enable-phase-flip, with the auto-disable of the sibling half recorded so the
register does not imply both halves still refuse. Level unchanged at BLOCKED;
whether a phase-flip-exempt row should keep that level is a register-owner
call, not mine.

Tests: the two reds were ready-made falsifiers, but they pinned the OLD single
conjunction, so they could not express the split. Rewritten to pin both halves
plus the exemption, each with its own falsifier proven by breakage:

  test_evidence_cites_land_on_BOTH_halves_of_the_guard
    red when the evidence carries the stale :16240-16245 cite
  test_the_register_records_the_PHASE_FLIP_EXEMPTION
    red when the row text is reverted to the pre-#704b wording
  test_the_overlap_half_is_now_an_AUTO_DISABLE
    red when the row stops naming disable_overlap_schedule. Note the weaker
    form of this check passed against the stale text: "auto-disable" appears
    there too, inside "not a quiet auto-disable" -- the opposite claim. Naming
    the field is what makes it a real check.

  test/registered/unit/planner/test_rejected_evidence_pins.py  2 failed,
    2 passed -> 6 passed
  test/registered/unit/planner  full suite, no remaining failures

Hermetic, CUDA_VISIBLE_DEVICES="", interpreter
/spinning/htsglang-gpu/.venv/bin/python3 (system python3 cannot collect these
suites; see 2c0542e). Register text and its pins only; no engine behaviour
touched.

Register rows carry a length budget (test_wizard_v2.py:482-490: why <= 400
chars, gain/cost <= 120) -- "a paragraph in the gain field is how this becomes
the wall it replaced". My first pass at the corrected row blew the why budget
at ~600 chars and turned that test red; the detail moved into evidence, which
carries no budget. why is now 295. Full planner suite: 2805 passed, 0 failed
(2 failed at the parent commit).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
… the sizer must NOT claim it

VERDICT: no dead reservation found. The gap is the arming floor (deliberate)
plus min-rule residue (structural), so this is the labeled accounting table the
brief asks for in that case, not a forced fix.

Per-rank, from the sgl-project#704/sgl-project#707 instrument chain rather than modelled:

  rank  budget  weights  mamba    rest  holdback  allowed   used   slack
  PP0   31800  16064.5   916.5  14819.5  6690.1   8129.5  5964.7  2164.8
  PP1   18800  10061.8   654.3   8083.4  3562.7   4520.7  4260.5   260.2
  PP2   19800  10699.8   523.3   8576.3  5167.9   3408.4  3408.4     0.0

Total slack 2,425 MiB, and the shape is the finding: it is almost all on PP0,
and the BINDER PP2 has exactly zero.

HOLDBACK is deliberate, established in sgl-project#707: the pool is capped so the resting
free column still holds the arming floor. Sizing below it is the boot that holds
the corridor and never flips (sgl-project#656 E/G). Not claimable.

SLACK is structural and still not claimable BY THE SIZER. The pool is one global
token count, so a non-binding rank cannot spend its extra allowance without the
binder moving -- PP0's 2,164.8 MiB is not booked-and-untouched, it is
unreachable at this cut. A sizer that claimed it would hand the pool tokens PP2
cannot back, and PP2 is the rank that OOMs. That is the sgl-project#593-family direction
the brief warns about, so I wrote NO red-first test for it: the falsifier would
have to assert a wrong answer.

WHERE THE CAPACITY ACTUALLY IS: the slack is a property of the CUT, and that is
already solved on another strand. sgl-project#702 rev5 found the incumbent [28,20,16] is
not pool-optimal -- the binding rank switches, so moving layers OFF the binder
raises the pool, and [30,18,16] gives +20 percent AND 1.11x pipelined prefill.
Claim it by rebalancing the cut, not by relaxing a reserve.

WHAT I DO NOT CLAIM: the ticket's 2.0 / 5.7 / 3.7 GiB per card is a DIFFERENT
operating point from the instrumented boot, and the instruments that would
attribute it per-term only exist on boots carrying 2a6305d / 5f3e61f /
f55c1a8. Mapping this table onto those three numbers is the cross-boot
arithmetic this strand has already retracted twice, so I did not. Ask for the
next boot to capture the three instrument lines at that operating point and the
table re-runs against it directly -- a window item.

One thing that CAN be said without it: against the measured arming floors
(1728 / 1825 / 2467 MiB) the per-card free exceeds the ~1024 MiB corridor target
BY CONSTRUCTION, because on a flip-enabled boot the arming floor is the binding
level, not the corridor. Reading the gap against 1024 overstates it on every
rank.

Docs only. No sizer change, deliberately.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…le, and what the phase matrix owes

Docs only. No code changed, nothing built or wired.

## sgl-project#702 decision table (docs/DECISION_702_cut_table.md)

Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool
reads 436,275 here against the 436,766 in older notes; all percentages use
436,275 so they are internally consistent.

  cut          attn      pool               speed    noise(14.1%)  gate
  [28,20,16]   (7,5,4)   436,275            1.0000   --            is the gate
  [30,18,16]   (7,5,4)   436,275  (+0.0%)   1.1111   BELOW IT      PASSES
  [33,15,16]   (8,4,4)   415,859  (-4.7%)   1.3299   above         FAILS

Switch cost 1575 ms from the incumbent either way (distance-independent, per
#704a); payback 15.75 s and 6.35 s respectively.

THE TRADE: the two candidates fail in OPPOSITE places, which is the decision.
[30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated
by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A
noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears
the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is
offered; the pick is the user's.

[30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the
(7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the
binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and
[517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two
layers between PP0 and PP1 cannot move a pool PP2 sets.

CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its
coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool,
which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut
including the incumbent) and therefore not attributable to any cut.

SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates,
per lead depth, ONLY the tail split that minimises pipelined time. At n0=30
that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated --
yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275
(+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a
roomier, slightly slower cut at the same lead depth cannot appear on it.
[30,18,16] strictly dominates the incumbent (equal pool, more speed) and was
invisible for exactly that reason, which is why it never reached the sgl-project#702 block.
The fix would be to keep the Pareto set over (pool, speed) per lead depth
instead of the speed-argmin; other pool-preserving cuts may also be hidden.

Validation-boot evidence is specified in falsification order, and includes the
one that matters most: for [30,18,16] the prefill A/B is expected to be
INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its
own noise, and that result must be rejected rather than celebrated.

Appended to the morning plan file as a dated block per its amendment
convention; head-298 sha256 verified byte-identical before and after
(81560e94b7628e6c8...). That file is not version-controlled, hence this copy.

## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md)

Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law
2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485
are the same ticket, not a number collision: the matrix doctrine plus its
prefill-column solver, and the seam/memory certification gating it.

VERDICT: PARTIALLY DELIVERED, do not close.

  DELIVERED  prefill column for the attention-vs-GDN/linear boundary
             (c08f613, b675630, b3c9802, e645aa7), with a second
             axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503
             447249a). Its own feasibility facet (407b382) is RED: the
             threshold is not reachable at the certification pool.

  SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED
             reasoning -- against the honest proportional (uneven-TP) baseline
             the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round.
             But this closes ONE CANDIDATE CELL (concentration), not the
             column. Named tension: the directive says single-family arms are
             DIAGNOSTIC only and must never be phrased as a phase-level
             verdict, and that commit is worded as one. The narrow reading is
             adopted.

  OPEN       O1 no (family x phase) structure exists anywhere -- pp_cut.py
             solves prefill only, family_split.py returns by_family with NO
             phase argument or dimension; the "matrix" is prose over two
             single-phase solvers. O2 the decode solver is DEAD CODE:
             solve_family_placement has exactly two references, its definition
             (family_split.py:141) and its test, zero call sites. O3 vocab,
             experts, nonlinear kernels and per-quant-lane linear layers were
             never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR
             and training tenants -- entirely untouched, and the largest open
             surface. O5 the prefill column is not arm-able while Facet B is
             red.

  NOT sgl-project#485   sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming
             machinery. They consume the prefill-column solve as an input and
             must not be counted toward the matrix; counting them would
             inflate its delivery with work on a different axis.

Status in one line: one of two columns, one of many families, one of several
workloads -- and the table is still prose.

Every load-bearing claim spot-verified against the tree rather than taken from
the survey: directive text read at source, solve_family_placement call sites
grepped, d937d5f's baselines read from the commit body, [30,18,16] priced
directly rather than inferred from frontier membership.

Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py).
Desk only, no GPU, deploy tree untouched.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 18, 2026
…ne consumer

Boot v7pp8 booted clean with the ledger no longer refusing, and still died:
PP2 priced at weights 6.248 + mamba 0.384 + pool 11.923 GiB against an 18.55
GiB budget, 0.15 GB free, dead on a 32 MiB decode allocation. The pool was
not mis-priced. There was nothing left for the transients a price does not
cover.

rank_user_reserve_mib is the corridor headroom and it had exactly ONE
consumer in the tree, phase_flip_runtime.py:473. A plain PP boot driven by
--rank-gpu-memory-mib therefore spent its budget to the last MiB, and no
auto-reserve line appears anywhere in the boot log to make up for it.

Held back as a NAMED post, so the planner reads it off the same 'KV budget
posts' line it already consumes rather than re-deriving it -- the discipline
sgl-project#704 established when it made that line the instrument.

GATED TO THE GAPPED PATH. Every shipped configuration was solved against
today's arithmetic; moving the reserve into the posts globally would shrink
each of their KV pools by a gigabyte per card, and no boot here has evidence
for changing solved, measured configurations. The gapped cut is the new path
and the one with the demonstrated OOM, so it is the only sizing that moves.
The corridor is a target and not a floor, so the holdback never drives the
pool negative and says so when it declines.

TESTS. 23 passing, including that a contiguous set and a boot with no layer
set price EXACTLY as before. One test earned its keep immediately: the
'or 1024' idiom copied from the flip path conflates an explicit 0 with unset,
so the reserve could not be switched off. Fixed in the code.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
… while the peers spin

BOOT-PROVEN (boot 22, boot_855_1033b_0840f82601_0831_131955, 13:24-13:26).
The first TP forward after a pp_to_tp cutover touches a shape specialisation
never loaded in this process. Rank 0's crash-time stack (log line 53248 ff.):

  _dcp_write_scatter (flashinfer_backend.py:2574)
    run (triton/runtime/jit.py:743) -> _init_handles (compiler.py:466)
      loadBinary -> cuModuleLoadData (libcuda.so) + 11 libcuda frames

NOT the compiler -- cuModuleLoadData. The module is built; LOADING it into the
CUDA context blocks, because the load needs the device and the device is
saturated by the peers' barlink BAR1 spin kernels, which are waiting in
all_gather for this very rank. The cycle closes and the spin deadline fires:
Bar1CollectiveAborted (ranks 1/3 and 2/3, group flip_dcp:0) -> SIGQUIT.

WHY THE EXISTING MECHANISM MISSED IT, measured: 352 build-window lines in that
boot, ZERO in 13:24-13:26, all 352 carrying 'full cuda-graph capture warmup'.
cold_build_window had exactly three production callers (barlink BAR1 build,
sampling warmup #603b, capture warmup) and none is on the path a cutover
re-dispatches into. sgl-project#640 on a path sgl-project#615 never saw. The fix makes that path a
caller rather than building a second mechanism beside it.

A STALE COMMENT NEARLY REFUTED THIS FIX, and the correction ships with it.
sampler_warmup.py said wrapping a lazy build in cold_build_window "does NOT
work ... the window is PROCESS-LOCAL". True when written (8bddb93,
2026-08-06); falsified ONE DAY LATER by sgl-project#615 (38ec4fb, 2026-08-07), which
hooked publication into cold_build_window so "every existing call site
therefore becomes group-visible without moving". Never revised, both ancestors
of this pin, verified with git log -S rather than assumed. The paragraph is
annotated in place rather than deleted, because it is still right about ITS
module (warm-at-boot + barrier REMOVES the race; a window only EXTENDS it).

WHY A WINDOW OVER THE REAL FORWARDS, NOT AN ENUMERATED WARM SET. The warm set
is not knowable by inspection -- it varies with direction, spec-decode, the
sgl-project#887 one-chunk grant and any backend swap -- and missing one member reproduces
the wedge exactly, at the next first-loader. Whatever loads, loads under the
window. Residual stated in the code: this EXTENDS deadlines (900 s cap) rather
than removing the race; boot 22's stall was ~150 s, so the cap is not binding.

FIRST-LOADER CENSUS, as a standing table: devtools/CENSUS_1033c_first_loaders.md
-- every first-loader site x triggering state change x covering window x
collective proximity. Rows 1-6 verified at file:line (row 5 on metal); rows 7
(runtime recapture / drafter switch) and 8 (rung change sgl-project#704, resume-restore
sgl-project#89) are written UNVERIFIED, so the table is a lower bound on the covered set
and never a proof that nothing else is exposed. It also names the inverse
column (armed windows that may cover paths that no longer first-load -- the
352 are a count of WARMUPS, not of BUILDS, and the window carries no
modules-loaded counter, so live and dead coverage are indistinguishable from
the log) and answers the upstream-minimal question: there are not four window
mechanisms but two plus a front door, with one site (barlink_device.py:865)
bypassing the front door. Named, not rebuilt.

#1033d, same boot, INDEPENDENT DEFECT, minimal hardening only: the prefetch IO
aux thread caught only Empty, so a page whose geometry did not match the
incoming binding (mha.py:556 reshape, '[2,16,1,4,256]'=32768 against a 16384
page) ended the thread three seconds before the cutover -- killing no process,
setting no exit code, appearing in no health probe, and leaving storage
prefetch dead for the rest of the boot. It now fails the OPERATION (host slots
released, so the requester gets a refusal instead of an unreachable
completion), logs loudly with a counter, and stays alive. The broad except is
defensible only because the alternative is silent thread death with no
supervisor above the loop; the underlying two-geometry host pool across a flip
(sgl-project#718/sgl-project#719/sgl-project#875 family, and the fork's own 'sgl-project#939 RE-HOME VIA RE-READ ... source
page 16384 elems vs destination page 32768 elems' line names the same mismatch
two lines earlier while handling it correctly) is NOT fixed here and is its own
posten.

DESK CHECKS, matched to each edit's failure class:
* devtools/check_1033c_cutover_window.py -- 19 cases, hermetic, driving the real
  run_batch wrapper: RED-FIRST arm reproducing the boot-22 condition (counter
  unarmed -> no window -> nothing published), first forward covered AND
  published, window closes (the sgl-project#431 open-without-close shape), budget finite
  and steady state byte-identical, window closed and budget spent even when the
  forward raises, and an AST check that the arming sits under no `if` (the naive
  grep form of that assertion failed on a COMMENT -- fixed to ask the AST).
* AST check on the aux thread: broad handler present, logs at ERROR, releases
  the failed operation's host slots, and continues.
* Method-split check: wrapper delegates, window only on the armed path, lazy
  import off the fast path, inner body intact.
ruff F401/F821/UP037 clean.

DESK-PROVEN. Metal proof is boot 23.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 20, 2026
The morning's module decides WHETHER to switch and what it costs. It could
not be booted: four pieces of the running process do not follow a boundary
change, and every one of them fails QUIETLY -- none raises, all produce
plausible output from the wrong layer set. Each falls here with a hermetic
test of its own.

1. ONE SOURCE OF TRUTH FOR THE RANGE. ModelRunner snapshots the range at init
   (model_runner.py:1185-1186) and derives three further things from the
   snapshot: num_effective_layers (:1200-1212), which is the KV pool's
   layer_num= at nine call sites, and the hybrid-SWA layer id lists
   (:1729-1745). sgl-project#704's actuator moves only model._start_layer, so all three
   stayed at their boot values. New ModelRunnerRangeMirror pulls them along,
   registered as an observer so a failure rolls the flip back.

   The SWA narrowing is DESTRUCTIVE -- adjust_hybrid_swa_layers_for_pp assigns
   the filtered list back over the attribute it filtered -- so re-running it
   after a widening intersects with an already-narrowed list and can never
   recover a dropped layer id. The mirror therefore snapshots the PRISTINE
   lists and re-derives every rung from those. It is built at the UNION and
   checks that it was, because only there are they pristine. The inclusive
   `end_layer + 1` bound is reproduced, not silently fixed: a mirror that
   corrected it would make the first flip change the lists for a reason
   unrelated to the boundary. Under SGLANG_PP_LAYER_SET the move is a refusal
   by name, not a recount -- the env names one fixed set per stage.

2. GRAPH RECAPTURE. A captured prefill graph records the decoder loop over
   owned_layer_ids(start, end) (qwen3_5.py:1720), so the executed layer set is
   baked at capture time and a replay ignores the boundary. New
   PrefillGraphRecaptureRegistry is the registry cuda_graph_observer already
   asks for (captured_range + recapture), turning that observer from a refusal
   into a repair. After a flip every captured shape is re-recorded or
   DISCARDED (new discard_shape on the two backends that hold _graphs) -- the
   state this exists to make unreachable is "still captured under the old
   range and still replayable". hot_shapes buys back the cost: cold shapes go
   eager rather than being re-recorded. per_shape_seconds must be measured;
   cost_seconds refuses rather than inventing the term, because it is a term
   in decide(). prefill_cuda_graph_runner.capture() is split into capture() +
   _capture_session() so a recapture enters the IDENTICAL context stack rather
   than an approximation of it -- two of those four contexts change what gets
   recorded.

3. POOLS OVER THE UNION, AND THE UNION PRICED. The ~20 pool sites do not each
   decide the range; they all read the same three ModelRunner attributes. So
   union_layer_window() widens them at the one place they read from, for the
   duration of the build, and every reader follows without knowing it did.
   union_layout/union_kv_costs/require_union_fits then put a NUMBER on what
   that costs: pool tokens lost per card. Deliberately not converted to
   seconds -- that needs a measured recompute rate, and a fabricated term
   beside measured ones is the mixed-unit class derive_x_star refuses by name.
   decide() takes union_costs + required_context_tokens and refuses the layout
   PAIR up front when 262k no longer fits, and takes recapture_s as a real
   term charged to each leg that moves.

4. MOVER ROUTE. barlink capture_safety() already states that BAR1 send is
   capturable and recv is not (barlink_bar1_p2p.py:181). require_breakable_route
   refuses by name BEFORE the first op. The danger is not the raise but the
   HALF that records: a capture holding the deposits and not the collects
   replays as a mover that ships bytes into peer windows and never lands them,
   silently, on the fast path.

STAGE ORDER. Checked as asked, and the premise needed correcting twice.
(a) The PP stage number is WELDED to the rank number (server_args.py:10700,
parallel_state.py:3945-3962); the send/recv layer would follow a permuted rank
list (neighbours are positional, :1319-1350) but nothing constructs one. The
equivalent move is free and supported: permute the CARDS, --rank-gpu-id 0,2,1.
StagePermutation does it to all five rank-indexed vectors at once or refuses --
planner/flags.py:2619 says "rank 0 IS the 5090" and the tokvec (33,13,18) and
reserve (3000,2200,2200) were measured against that sentence, so a half-applied
permutation hands the 5090's KV share to a 3080 with no error at all.
(b) The gain is smaller than estimated. The swap does NOT speed up the slowest
LINK -- a 3080-to-3080 hop at 6.58 GB/s then becomes the worst one. And on the
64-layer geometry the switch goes 0.378 s -> 0.234 s, a 38 % cut, not the
halving: stage 1 also sheds four layers to stage 2, and once the 5090's hop is
fast that hop binds. placement_switch_seconds prices the bytes each hop
actually carries; slowest_hop_seconds is documented as the smaller question.

Two bugs were found by the new tests rather than reasoned about: the mirror
wrote start/end BEFORE the refusal check (a half-applied update the actuator's
rollback would not undo), and RecaptureReport reported the new range as the
old one.

Tests: tests/moe_offload/test_p_layout_inprocess_0920.py (39 cases, one per
blocker plus the stage order) and test_p_layout_whole_boot_0920.py (12 cases),
a CPU whole-boot probe standing up three fake PP ranks -- model, runner, KV
pool, capture ladder, BAR1 mover -- and driving P39 -> P32 -> P39. The ROUND
TRIP is the assertion: a one-way switch passes with a destructive narrowing, a
moved indexing base and a discarded graph ladder in it.

Gate tests/moe_offload: 43 failed (unchanged baseline), 638 -> 689 passed, 41
skipped (the morning's 657 counted the 19 test_layout_boundary_704 cases).
test_layout_boundary_704.py 19/19, test_p_layout_switch_0920.py 68/68.
Targeted graph-backend/capture suites 81 passed 1 skipped.
test_breakable_graph_error_masking 7 failed = baseline, verified against a
detached checkout of 4e11cd1 rather than assumed. black/isort/ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant