Repository navigation
feat: add e2e latency - #704
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cen121212
pushed a commit
to cen121212/sglang
that referenced
this pull request
Nov 10, 2025
* Update run_suite.py * Update test_retract_decode.py * Update test_reasoning_content.py
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… arena ladder, and that reprices it The actuator survey came back and it changes the design, so the design changes rather than the finding being reinterpreted. ## R1: there is no runtime cross-rank weight mover, and the absence is deliberate regime_stages.py:100 defines REACH_NO_WEIGHT_MOVER -- "no runtime actuator moves weights, switching arms needs a restart (sgl-project#354/sgl-project#357)". StagePlan.reachable (:156) excludes it, so such a stage cannot be selected, and regime_act.py:121-130 wires exactly kv/vram/phase. Corroborated at uneven_perf.py:7744-7751. So sgl-project#363's "weight mover" is a REFUSAL CODE, not an actuator -- the opposite of the counter-versus-actuator failure, and honest. It is not something to wire to. What exists is weights_arena.py + phase_flip_boot.py:361: a fixed-address VRAM arena refilled by a host->device memcpy from a boot-baked pinned image, checksum-verified, with a rollback arm, and containing no dist.* call at all. On this no-P2P rig that is the better primitive anyway -- a rank-to-rank transfer would stage through host memory regardless -- and the fixed address keeps captured CUDA graphs valid across a rung change. ## The consequence, which is not favourable The arena is sized for the DEEPEST rung and resident at EVERY rung: a rank does not get its weight bytes back when the ladder sits shallow. solve_arena_ladder() models this, and two things fall out. First, the shallow rungs get poorer. Over a rank0 span of [31,38] the top rung drops from 518,433 to 390,700 live-equivalent -- below the incumbent's observed 434,878. The ladder costs ~10% pool at the top to buy up to 1.231x pipelined prefill at the bottom. Standing alone that is a marginal trade, and this commit says so in the design rather than shipping the flattering number. Second, and structurally: with free memory pinned by the arena, a rung's pool depends only on its ATTENTION count. Two rungs sharing an attention profile price identically and the faster one strictly dominates. Under an arena the ladder's real axis is the attention-count vector, not the raw layer cut. That invalidates my own earlier Slice 1 pair. [33,15,16] <-> [35,14,15] was chosen because it moves zero attention layers, which is exactly what makes it useless AS A LADDER PAIR. It is retained only to prove the mover mechanically; a pair that actually trades must cross an attention boundary, so Slice 1b uses [35,14,15] <-> [38,13,13]. The alternative mover -- reallocate per rung, restoring the full per-rung pool at the cost of a CUDA graph recapture -- is named and left open. Both solvers are in the branch; Slice 2 measures the recapture cost and closes the fork. ## Part B repriced by its own survey The cross-rank LSE merge is real, N-way and rank-order deterministic (layers/dcp/comm.py:228-262, all_gather places rank i at index i), which RETIRES the merge-order risk. But two blockers are confirmed: DCP groups are subsets of one PP stage by construction (parallel_state.py:3142-3159) and under PP prefill no DCP group is instantiated at all (dcp_group_guard.py:1-42); and the KV pool is DIMENSIONED by weight ownership -- model_runner_kv_cache_mixin.py:2466-2470 filters attention layer ids to the rank's own [start_layer, end_layer) before the pool is built, so a rank has no row-space for a layer it does not own. Part B is construction, not assembly; slice 3 splits into 3a/3b/3c at 4-7 windows, with the pool allocation rework on the critical path. layers/dcp/test_weightless_kv_math.py is confirmed test-only: it imports no torch and reimplements the LSE formula locally. It is not evidence of a wired path. ## Test results 60 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean on all new files, codespell clean. - test_ladder_controller_704.py (10, new): oscillation falsifiers on adversarial fill traces. The anti-flap test carries a CAN-FAIL PROOF: with the hysteresis collapsed to a bare threshold the controller emits 400 moves under a 1-token jitter; with it intact, zero. An earlier version of that test probed an INACTIVE threshold and passed for both, i.e. it was vacuous -- fixed by exposing active_descend_threshold() and asserting a fixed point. - test_layout_ladder_704.py (25, +5): arena sizing, the equal-attention-profile domination property, the deeper-ladder-costs-pool trade, and an unaffordable arena refused with its shortfall. - test_pp_cut_family_pool_704.py (6) and the two sgl-project#702 suites (19): unchanged. The controller is a DECISION FUNCTION and says so in its docstring: it moves no bytes and nothing is wired to it yet. Stated because a decision function mistaken for an actuator is the exact confusion this survey was run to avoid.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…h reserve is the blocker Review-gate item 2. Every term from config or instruments, nothing fitted. KV term is byte-exact from config, not fitted: kv_cache_dtype fp8_e4m3 (1 byte) x num_key_value_heads 4 x head_dim 256 = 1024 B per token per ATTENTION layer for K, 2048 B for K+V. The boot log's per-rank K sizes at 436,766 tokens match on all three ranks (2.92 / 2.08 / 1.67 GiB predicted and logged). This falsifies the 4096 B bf16 constant in Slot-3's doc directly. Mamba residency derived from the allocation sites (memory_pool.py:583-608, :655-665, :693-705) with max_mamba_cache_size 12 (13 slots), max_running_requests 4 (5 spec slots), speculative_num_draft_tokens 4, temporal_state_shape (48,128,128), conv_dim 10240, win 3, bf16: temporal_state 13 x 48x128x128 x 2B = 19.50 MiB/GDN-layer conv_state 13 x 10240x3 x 2B = 0.762 intermediate_ssm_state_cache 5 x 4 x 48x128x128x2B = 30.00 intermediate_conv_window_cache 5 x 10240x6 x 2B = 0.586 total = 50.85 MiB/GDN-layer The gate's 19.5 MB/GDN-layer is the temporal component ALONE. The full residency is 50.85, dominated by the SPECULATIVE intermediate cache at 30.00, so a model charging only 19.5 under-charges GDN layers by 2.6x. Forward validation against the live [28,20,16] boot chain (avail-after-weights minus avail-after-pool): predicted 6.874 / 4.910 / 3.928 GiB against measured 6.91 / 4.95 / 3.97 -- under 1.1 percent on every rank. The residual is nearly CONSTANT at ~40 MiB rather than scaling with layers or attention, which is the evidence that the per-layer terms are complete: a missing per-layer term would have shown a slope. CORRECTION to my own blocker claim. I previously named the +-7 percent unbooted-floor uncertainty as what blocks bootable predictions. Wrong in emphasis. Retro-prediction runs the equation backwards and needs everything the sizer sets aside before the pool; every term is now config-derived except the GRAPH RESERVE, which the chain puts at 6,372 / 2,650 / 2,377 MiB per rank after subtracting the measured floors. Next to that the floor uncertainty is second-order. The fix is not to model the graph reserve independently but to have the pool solve consume the sizer's own reserve terms, as it must already consume the sgl-project#676 floor. Structural finding encoded (gate 3.5): no cut keeping rank2 = layers 48-63 can beat the incumbent pool. Rank2 is byte-identical across both boots (weights 10.40 GiB in each log) and binds at 436,766, a hard min-rule ceiling. Pool gains require shrinking rank2's ATTENTION count or Part B decoupling, not rank0/rank1 rebalancing. Also on record: the incumbent's binder is PP2 per the boot log (PP1 cap 463,406, PP2 436,766). My rev5 calibration assigned it to rank1 and solved a free constant from that assumption; this derivation needs no such fit and supersedes it. Docs only. Items 1 (canonical pp_cut convergence) and 3 (sgl-project#701 slice-3 rework) not started.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…take the four gate corrections The review gate adjudicated the attention-only divisor in this branch's favour -- white-box at the allocator, byte-exact against the boot log's K sizes -- and found four defects of mine. This lands the convergence and all four. ## Convergence: one canonical solver Merged c5afff7 (Slot-2 rev5), which adopts the attention-only rule and adds the per-layout arming floor and mamba terms. My duplicate FamilyPoolModel / stage_family_capacities / family_phase_pool are DELETED; rev5's PhasePoolModel is the only pool model. Slot-2 leads the solver; this branch owns layout_ladder, ladder_controller and the arena model, which now consume rev5 instead of reimplementing it. Two additions to pp_cut.py that do not duplicate rev5: - kv_mib_per_token_per_attn_layer_from_config() + kv_dtype_width_bytes() - decoupled_phase_pool(), the part B projection, rebased on PhasePoolModel ## E1: the KV cell was FITTED, and was wrong by 2x DESIGN_704 read the cell as 4096 B (bf16). The shipped config is fp8_e4m3 and the cell is 2048 B for K+V, 1024 B for K. Fitting it against an observed pool is what produced both the dtype error and the bogus "0.83 of observed" fudge -- they cancelled into something that looked calibrated. The cell is now consumed from config (2 x kv_heads x head_dim x dtype_width). Unknown dtypes raise rather than default, because a wrong default is a silent 2x on every pool number; 'auto' raises too, since it names no width. ## E2/E3: the binding rank, and the retraction I claimed the incumbent binds on rank1. The boot log says PP2 binds at 436,766. The functional form was right; my free-bytes vector was not, and that produced the retraction: [33,15,16] leaves rank2 holding layers 48-63 -- byte-identical to the incumbent's rank2, measured cap 436,766. Under the min-rule no cut keeping that rank2 can exceed it, so my claimed 457,604 over-predicted an UNCHANGED rank's measured capacity by 4.8%. Expected actual ~387k, about -11%. The arm is withdrawn. [32,16,16] had already failed its gate on metal at 416,796, the same error class. The "discriminating experiment" justification was void: the divisor was settled white-box, so the window would have bought nothing. What replaces it is stronger than the arm I lost: NO cut that keeps rank2 = layers 48-63 can beat the incumbent pool. Pool-positive rungs must shrink rank2's attention count, or wait for part B. That is structural, not a number needing re-measurement. ## E4: GDN residency was under-charged 2.6x, and it falsified a claim The full per-GDN-layer figure is 50.85 MiB, not 19.5: temporal_state 19.5 + speculative intermediate_ssm_state_cache 30.0 (5 spec slots x 4 draft tokens) + conv_state 0.762 + intermediate_conv_window 0.586. This falsified a structural claim of mine. §3.7 said two rungs sharing an attention profile price EXACTLY the same, so the deeper strictly dominates. That held only because the 30 MiB speculative term was missing. Corrected, such rungs differ by their GDN residency (~3,250 tokens at 8 attention layers): a weak real trade, not a domination. test_under_an_arena_equal_attention_profiles _have_equal_pool failed the moment the constant landed and is replaced by test_under_an_arena_same_attention_profile_differs_only_by_gdn_residency, which asserts the gap is explicable by GDN residency and nothing else. ## Part B defects D4/D5/D6 - D4: the byte-identity gate as written could never pass. "decoupled vs coupled" is A-vs-B -- an LSE merge sums in a different float order than monolithic attention -- so it would fail forever on correct code and be waived. Respecified: (1) decoupled-vs-decoupled determinism, byte-identical across runs and boots, CPU-sampled inputs; (2) agreement with the coupled reference within a tolerance fixed before the run. Gate 1 is never waived. - D5: "TP identity wins" was unconditional and cannot be. The measured TP vector puts ~43.8% of KV rows on rank0, the rank with the least free bytes at exactly the deep rungs part B exists to unlock, so prize 1 and prize 3 can contradict. TP identity now carries a per-rung feasibility bound, and the rung yields before the vector does. - D6: committing rung changes only at a fully-quiescent boundary can starve. 640 chunks per max-length prompt with overlapping admissions means the boundary may never arrive -- and ascent is needed precisely while chunked prefills are active. That is the ladder's own sgl-project#701-shaped wedge. Paired with an admission hold plus bounded drain and an urgency-derived deadline; asymmetric, since only a safety-seeking ascent may hold admission. ## Model discipline LadderInputs now REQUIRES an arming_floor_for(counts) provider and refuses construction without one -- a constant floor is the E3 error. Rev5's own docstring records the known gap: the sgl-project#676 solver derives the floor from a measured seam draw, so an unbooted cut has no solved floor, leaving ~±500 MiB (~±32,000 tokens, ~7%) of uncertainty on every unbooted rung. No rung's predicted pool is a boot gate on its own; the retro-prediction gate against all four measured points (434,878 / 435,822 / 436,766 / 416,796) comes first, and Slot-2 owns it. The withdrawn headline table is deleted rather than patched. Re-deriving those numbers before the retro gate passes would repeat the mistake. ## Test results 63 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. - test_kv_cell_from_config_704.py (7, replaces test_pp_cut_family_pool_704.py): reproduces the logged K sizes 2.92/2.08/1.67 GB from config alone with zero free parameters; pins that a bf16 reading misses the log by exactly 2x; refuses unknown dtypes and 'auto'; keeps the 0.47-bytes-per-element dimensional record that settled the dispute. - test_layout_ladder_704.py (25) and test_ladder_controller_704.py (12): ported to rev5's API. test_descend_only_happens_at_low_fill now states its thresholds RELATIVE to the solved ladder -- its absolute token constants silently encoded the KV dtype and became meaningless when the cell was corrected. - rev5's test_pp_cut_phase_pool_702.py + test_pp_cut_prefill_speed_702.py (19): green across the merge. The rig fixtures are labelled STRUCTURAL, NOT CALIBRATED PREDICTORS: they pin monotonicity, hysteresis and arena residency, which are invariant to the free vector. No pool value in them is bootable.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…age_pp_capacities Last reference to my deleted duplicate model. The refusal itself is rev5's and is unchanged: a stage with no attention layer carries no token-scaling KV, so pricing it as unbounded capacity is how a cut that cannot serve gets armed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…-0 amendment The arena argument establishes the SIGN of the trade, not its size. The specific 1.231x / 390,700 came from the fitted bf16 cell and went out with the rest; restating it in corrected-looking form before the retro-prediction gate passes is the same mistake in a smaller font.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
This is the instrument that unblocks the four-boot retro-prediction gate. The blocker was not the arming floor and not the graph reserve as a quantity. It was that the sizer builds budget_posts -- its own named decomposition of the KV budget -- on every boot and hands it to budget_exhausted_message ONLY when the budget runs out. The sizer names every term of its arithmetic exactly when it fails, and discards the naming when it succeeds. So the planner had no instrument for the reserve it must not double-count, and had to re-derive it from config in a different process, where it could not even be compared. Three independent re-derivations, three misses against the live [28,20,16] boot: subtracting derived_rank_auto_reserve_mib (4160 MiB, called directly) gave PP2 = 524,620 tokens against a measured 436,766, i.e. +20 percent; the non-budget slack path gave -3.8 percent; the configured-budget variant gave -12 percent. The boot actually takes the budget_mib branch because rank_gpu_memory_mib is set, which is a fourth path again. Re-deriving a number the sizer already computes is the shape this corpus has paid for repeatedly. So the fix is the one the gate's meta-lesson implies: create the instrument and consume it. One info line emits the posts and the final rest_memory on the success path, in the same GiB units the failure message already uses. The pool solve then consumes these exactly as it already consumes the sgl-project#676 arming floor, rather than recomputing them. Not included, and stated rather than implied: the consuming side. PhasePoolModel cannot require a sizer_reserve field until a boot has emitted one -- the posts do not exist in any log on disk today, which is precisely the defect. The next boot produces them, and the consuming field lands with real values instead of a placeholder. Verification: compiles, ruff clean, logger already in scope at module line 168. model_executor suite 594 passed / 15 failed, and the 15 are PRE-EXISTING -- verified by reverting this file, re-running, and getting the identical 594/15, then restoring. This commit adds one log statement and changes no control flow.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…RO weight bytes
DESIGN_704 specified the ladder's actuator as an H2D arena_refill per rung
change -- one contiguous copy of the target layout's host image, 451-901 MiB a
step -- because no cross-rank weight mover exists (regime_stages.py:100,
REACH_NO_WEIGHT_MOVER). Implementing it showed that is more work than the
existing primitives require.
plan_arena_layout fixes slot offsets by SORTED TENSOR NAME, and
bind_arena_views rebinds parameters to arena views without copying any bytes
(weights_arena.py:495-519). Plan the arena over the UNION of the rungs' tensors
rather than per rung and a tensor shared by two rungs sits at the same offset
in both -- so a rung change is a rebind plus a PP-boundary change, and no
weight byte moves at all.
## The union is doing real work, and there is a can-fail proof
With PER-RUNG layouts, 14 of 56 shared tensors land at DIFFERENT offsets,
because model.layers.28.* sorts lexically between ...27.* and ...3.* and
displaces every slot after it. A per-rung arena would therefore have to
relocate weights that are already resident and unchanged, purely because a name
sorted differently. Under the union, zero move. Without that check every
zero-copy assertion in the suite would pass vacuously.
## The price, and what is NOT claimed
Cost moves from bandwidth to residency: a layer that changes hands must be
resident on BOTH ranks, so the world holds 64 + (layers that move)
layer-images -- one extra layer (~451 MiB) for the slice-1a pair. That is the
same trade DESIGN_704 section 3.7 already prices for the arena, and on this rig
it strictly dominates the refill, since with no P2P a rank-to-rank transfer
stages through host anyway.
arena_refill is NOT replaced. It stays correct for the PHASE FLIP, where the
two layouts are genuinely different tensor sets (PP weights vs TP width-shards)
with no useful union. The union is for the LADDER, where consecutive rungs
differ by one or two whole layers.
Scope is carried in the data structure, not in a reader's memory: "zero bytes"
must not be read as "nothing to do". A GDN layer carries per-sequence recurrent
state (temporal_state ~19.5 MiB/layer + conv_state ~0.762) that travels with
the layer, so slice 1a flips only at QUIESCENCE. Every FlipDelta carries
requires_quiescence=True and an out_of_scope string, and a test asserts both.
Live-state transfer is slice 1b.
## Boot ticket, prepared and deliberately not self-arming
scripts/boot_704_1a_29_19_16.sh splits slice 1a into a zero-code half and a
code half:
1a-i two STATIC boots, [28,20,16] <-> [29,19,16], changing exactly one
variable (--pp-layer-ratio). Both cuts are attention profile (7,5,4)
and layer 28 is LINEAR, so no attention layer and no KV row moves.
1a-ii the union arena plus runtime wiring.
The ticket refuses without GO=1, a /spinning/gpu-arb/ claim, a clean tree, and
exactly one --pp-layer-ratio in the source argv -- a boot that changes zero or
two variables proves nothing. It stops before launching: F4-r4 owns the queue.
Falsifiable prediction, written to disk BEFORE the window as the review gate
requires: P1 pool 436,766 +- 2500; P2 PP2 binds; P3 K sizes 2.92/2.08/1.67 GB
unchanged. The reasoning is structural, not a rung-table re-derivation (which
is forbidden until Slot-2's retro-gate closes): rank2 keeps layers 48-63
byte-identically so cap2 is unchanged; rank1 sheds a layer so its cap rises;
rank0 gains ~502 MiB of weights plus GDN residency, dropping cap0 by ~36,600
tokens from >=550,000, still far above 436,766. If P1 fails, the corrected pool
model is wrong where it matters -- record and STOP, do not retune inside the
window.
Independent second prize: PrefillTiming.fixed_ms defaults to zero, so every
speedup in DESIGN_704 is an UPPER BOUND until a second measured cut exists.
This boot is that cut, and pins the per-layer slope against the fixed per-stage
intercept for the first time. Worth the window even if P1..P3 all hold.
## Test results
76 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean.
- test_weights_arena_union_704.py (13, new): the zero-copy offset invariant;
zero bytes_to_copy across a flip; residency overhead equal to exactly the
moved layer's tensors; binding mutates no arena byte; views alias rather than
copy; shape and dtype conflicts refused; unknown/single rung refused; an
undersized arena refused before binding; flip symmetry; the quiescence scope
flag; and the per-rung-layouts can-fail proof.
- 63 existing planner tests unchanged and green.
Boot-script gates verified by execution: refuses without GO, refuses without a
GPU claim, and the argv swap changes exactly one line.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
This is the instrument that unblocks the four-boot retro-prediction gate. The blocker was not the arming floor and not the graph reserve as a quantity. It was that the sizer builds budget_posts -- its own named decomposition of the KV budget -- on every boot and hands it to budget_exhausted_message ONLY when the budget runs out. The sizer names every term of its arithmetic exactly when it fails, and discards the naming when it succeeds. So the planner had no instrument for the reserve it must not double-count, and had to re-derive it from config in a different process, where it could not even be compared. Three independent re-derivations, three misses against the live [28,20,16] boot: subtracting derived_rank_auto_reserve_mib (4160 MiB, called directly) gave PP2 = 524,620 tokens against a measured 436,766, i.e. +20 percent; the non-budget slack path gave -3.8 percent; the configured-budget variant gave -12 percent. The boot actually takes the budget_mib branch because rank_gpu_memory_mib is set, which is a fourth path again. Re-deriving a number the sizer already computes is the shape this corpus has paid for repeatedly. So the fix is the one the gate's meta-lesson implies: create the instrument and consume it. One info line emits the posts and the final rest_memory on the success path, in the same GiB units the failure message already uses. The pool solve then consumes these exactly as it already consumes the sgl-project#676 arming floor, rather than recomputing them. Not included, and stated rather than implied: the consuming side. PhasePoolModel cannot require a sizer_reserve field until a boot has emitted one -- the posts do not exist in any log on disk today, which is precisely the defect. The next boot produces them, and the consuming field lands with real values instead of a placeholder. Verification: compiles, ruff clean, logger already in scope at module line 168. model_executor suite 594 passed / 15 failed, and the 15 are PRE-EXISTING -- verified by reverting this file, re-running, and getting the identical 594/15, then restoring. This commit adds one log statement and changes no control flow.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
Defect I introduced in 84abeb7, caught by its own first metal output. The instrument labelled each line with self.tp_rank. Under the flip's primary topology (tp=1, pp=N) every PP stage has tp_rank 0, so all three stages name stage 0's card. That is the sgl-project#201 defect, and the budget-exhausted path a few lines below already guards against it in a comment that says so verbatim: "tp_rank alone makes every PP stage name stage 0's card". I wrote the new line without applying the warning sitting next to it. Confirmed by the first boot to carry the line: PP0, PP1 and PP2 all emitted "[rank 0]". A second boot in the same log happens to show 0/1/2 because that configuration has distinct tp_ranks, which is exactly the kind of intermittent correctness that makes the label untrustworthy rather than merely ugly. Fixed to _rank_vector_index(), the accessor this module already uses for the same purpose (it resolves server_args.world_rank(pp_rank, tp_rank) and falls back to tp_rank only when server_args is stubbed). Field renamed to world_rank so consumers cannot silently read the old meaning. The already-captured data remains usable: the log prefix (PP0/PP1/PP2) carries the true stage, so the six lines in docs/dev/DATA_704_budget_posts.txt are correctly attributable despite the wrong bracket. Captured to the repo because /root/current_boot.log is a symlink that moves on the next boot and this is the retro-prediction gate's input data. First metal values, [28,20,16] budget_mib branch (the branch my external re-derivations kept missing): weights+runtime 15.688 / 9.826 / 10.449 GiB, mamba+spec+activation reserve 0.895 / 0.639 / 0.511, GGUF scratch 0.000, rest 14.472 / 7.894 / 8.375. Verification: compiles, ruff clean, model_executor 594 passed / 15 failed with the 15 already established as pre-existing by revert-and-rerun on the previous commit. Still one log statement, no control flow.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
Defect I introduced in 84abeb7, caught by its own first metal output. The instrument labelled each line with self.tp_rank. Under the flip's primary topology (tp=1, pp=N) every PP stage has tp_rank 0, so all three stages name stage 0's card. That is the sgl-project#201 defect, and the budget-exhausted path a few lines below already guards against it in a comment that says so verbatim: "tp_rank alone makes every PP stage name stage 0's card". I wrote the new line without applying the warning sitting next to it. Confirmed by the first boot to carry the line: PP0, PP1 and PP2 all emitted "[rank 0]". A second boot in the same log happens to show 0/1/2 because that configuration has distinct tp_ranks, which is exactly the kind of intermittent correctness that makes the label untrustworthy rather than merely ugly. Fixed to _rank_vector_index(), the accessor this module already uses for the same purpose (it resolves server_args.world_rank(pp_rank, tp_rank) and falls back to tp_rank only when server_args is stubbed). Field renamed to world_rank so consumers cannot silently read the old meaning. The already-captured data remains usable: the log prefix (PP0/PP1/PP2) carries the true stage, so the six lines in docs/dev/DATA_704_budget_posts.txt are correctly attributable despite the wrong bracket. Captured to the repo because /root/current_boot.log is a symlink that moves on the next boot and this is the retro-prediction gate's input data. First metal values, [28,20,16] budget_mib branch (the branch my external re-derivations kept missing): weights+runtime 15.688 / 9.826 / 10.449 GiB, mamba+spec+activation reserve 0.895 / 0.639 / 0.511, GGUF scratch 0.000, rest 14.472 / 7.894 / 8.375. Verification: compiles, ruff clean, model_executor 594 passed / 15 failed with the 15 already established as pre-existing by revert-and-rerun on the previous commit. Still one log statement, no control flow.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…nge, not by copying Slice 1a showed a rung change moves no weight BYTES (union arena). This wires the other half: changing which layers a rank EXECUTES, at runtime. ## The model already permits it Three facts, each verified at file:line, fix the design: - make_layers (utils/common.py:1970-2010) builds a ModuleList of length num_hidden_layers with PPMissingLayer placeholders outside the owned range. Layer indices are therefore GLOBAL on every rank, so a boundary change is not an index shift. - start_layer/end_layer are PROPERTIES over mutable _start_layer/_end_layer backing fields (models/qwen3_5.py:1452-1457). - The decoder forward iterates range(self.start_layer, self.end_layer) (qwen3_5.py:1483) and reads those properties on EVERY pass. So a real layer parked outside the active range is simply not executed, and the boundary change is a range mutation: no module swapping, no reallocation, no bytes. Composed with the union arena, a rung change moves nothing at all -- a test pins that every parameter keeps its data_ptr() across a flip while the executed set changes. Note qwen3_next.py is NOT the PP model: it asserts is_first_rank and is_last_rank at :1165. qwen3_5.py is the PP-capable one. ## Load wide, run narrow At boot a rank builds and loads real modules for the UNION of the ranges it may occupy, then runs whichever sub-range the ladder selects. The union must be in force during LOADING too, because weight loading is gated on the same range (qwen3_5.py:1563-1564, :1707-1708). ## Three failure modes, enforced rather than documented 1. Entering a non-resident range is SILENTLY WRONG, not loud. PPMissingLayer is a pass-through (layers/utils/common.py:109-127), so executing a range whose weights never loaded yields plausible output from a shallower model instead of an error. Every flip verifies residency first, and a hole inside the claimed union is caught at construction. 2. A half-applied boundary is worse than no flip. Dependent structures (the KV pool's full_attention_layer_ids filter at model_runner_kv_cache_mixin.py:2466-2470, the GDN state maps) are keyed by the owned range. Observers run INSIDE the flip; if one raises, the range is rolled back and the flip fails. 3. The third came out of a test failure rather than foresight. The actuator originally ASSUMED the model already sat at its current rung. Under load-wide it does not -- it arrives wide -- so the constructor now performs the narrowing step explicitly. Left as it was, the actuator's belief and the model's actual range would have disagreed from the first forward, silently executing a layer the rank did not own. Pinned by test_construction_performs_the_narrowing_step. World-level tiling is validated separately, because it cannot be checked locally: a gap silently DROPS layers and an overlap silently COMPUTES THEM TWICE, while every individual rank's range looks entirely sensible. ## Scope, unchanged Flips at QUIESCENCE only, and the refusal names why: a GDN layer's per-sequence recurrent state travels with the layer and slice 1a moves none of it, so an in-flight sequence would continue against state that is no longer there. Live-state transfer is slice 1b. arena_refill is untouched and remains correct for the phase flip, whose two layouts are disjoint tensor sets with no useful union. ## Test results 90 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. - test_layout_boundary_704.py (14, new): the active range selects what runs while an unowned layer stays resident; no storage moves across a flip; flip back restores exactly; refusal when not quiescent and when the target is not resident; a hole in the union caught at construction; unknown rung refused; same-rung flip is a no-op; world tiling rejects gap/overlap/short-cover; the narrowing step; observer notification with rollback on failure; and an end-to-end test that the union arena and the boundary actuator compose. - test_weights_arena_union_704.py (13) unchanged, including the can-fail proof that per-rung layouts displace 14 of 56 shared tensors. - 63 planner tests unchanged and green. Remaining for a GPU window: boot-time union load and registration of the dependent-structure observers. The actuator itself is complete.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… the budget post is short The operator flagged the metal mamba posts as ~14 percent below my derived 50.85 MiB/GDN-layer and suspected an over-charged spec-slot sub-term in my model. That suspicion is wrong, and settling it needed no new instrument: MambaPool already logs every sub-term at construction (memory_pool.py:756-765). Read off the live boots, per GDN layer, all three ranks: allocated 51.20 MiB against my derived 50.85 -- inside the log's own 2-decimal GiB rounding (four terms at 0.01 GiB each = +-1 MiB/layer). The individual terms match as well: ssm_state is exactly layers x 13 slots x 1.5 MiB (21 -> 0.400, 15 -> 0.286, 12 -> 0.229 GiB) and intermediate_ssm is exactly layers x 5 spec slots x 4 draft x 1.5 MiB (30.0 MiB/layer on all three). The spec-slot count of 5 is correct, not over-charged. What IS off is the sizer's own budget post. post/allocated = 0.852 on all three ranks -- exactly constant, so a systematic formula divergence rather than rounding or noise. The sizer hands out roughly 158 / 111 / 89 MiB more mamba memory than it charges to its budget. Consequence for the retro-prediction gate, and it changes the plan: the pool solve must consume the ALLOCATED figure from the Mamba Cache line, not the budget post. Feeding PhasePoolModel from the posts alone -- which is what I had planned -- would have baked a systematic 150 MiB/rank optimism into the model. Open sub-item, recorded rather than guessed: which term in handle_max_mamba_cache produces the 0.852. It is exactly constant, so it is one factor and not an accumulation, but naming it needs a read of that function rather than more arithmetic from outside. It does not block the gate, because the gate now consumes the allocation. Docs only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…brators, and one of them is none
PrefillTiming models a stage as fixed_ms[r] + ms_per_layer[r] * n_r, and one
measured cut cannot separate the two -- both fit the single point exactly.
prefill_timing_from_measurement therefore defaults fixed_ms to zero, the
OPTIMISTIC end of the family, which is exactly why every speedup in DESIGN_704
is an upper bound rather than a prediction. Two cuts resolve it per rank by
elimination.
The arithmetic is four lines. The reason this is a module is that the result
can be confidently wrong in two specific ways, and doing the sums BEFORE the
window changed what the boot should ask for.
## The slice-1a pair is three calibrators at once
rank0: 28 -> 29 (dn=+1) expected dt ~1.76 ms -- weak, the binding cost
rank1: 20 -> 19 (dn=-1) expected dt ~7.74 ms -- strong
rank2: 16 -> 16 UNCHANGED -- NOT CALIBRATED AT ALL
rank2 holds 16 layers in both cuts, so the pair carries no information about
its intercept. The solver refuses to report one rather than emitting the
fixed_ms=0 fallback, which at a call site is indistinguishable from a
measurement.
## Sample count is a precondition, not an afterthought
The slope is a difference of two means over the layer delta, so
stderr(s) = sqrt(2)*sigma/|dn|, and dn=1 puts the entire per-stage noise onto
the slope. Because rank0's slope is small it is far dearer to pin than rank1's:
per-chunk SD | chunks for 10% on rank0 (1.757) | on rank1 (7.740)
1 ms | 65 | 4
2 ms | 260 | 14
3 ms | 584 | 31
5 ms | 1620 | 84
One max-length prompt is 327680/512 = 640 chunks, which clears the 584 needed
at SD=3 ms -- so ONE full-length prefill per arm suffices for rank0 at 10%,
provided the per-chunk SD is measured and reported rather than assumed. The
boot ticket now demands mean, per-chunk SD, N and the standard error per stage;
without the SE the intercept cannot be gated and must not be published.
## Two refusals, because a confident wrong number is worse than none
- A NEGATIVE intercept is reported, not clamped. It means the linear model does
not hold across the pair (most likely a non-constant per-layer cost), and a
clamped fit looks calibrated while being wrong. Only floating-point dust is
rounded to zero.
- STRUCTURAL refusals precede data-dependent ones. Whether a pair can calibrate
a rank at all is a property of the two LAYOUTS; whether the numbers come out
sane is a property of the MEASUREMENT. Reporting the second while the first
is outstanding sends the reader to inspect timings when the pair was never
capable of the answer. Found by test failure: the rank2 refusal was being
masked by rank0's negative intercept.
## Also in this commit, per review
- DESIGN_704 sec 3.9 failure mode 3 is restated as a CONTRACT rather than a bug
anecdote: the actuator OWNS the active range and never infers it; there is
exactly one writer of _start_layer/_end_layer once an actuator exists, and any
other writer is a defect. The anecdote is kept in one parenthesis because the
next reader needs the rule, not the story.
- The model-class correction is carried explicitly: models/qwen3_next.py is NOT
the PP model (it asserts is_first_rank and is_last_rank at :1165);
models/qwen3_5.py is the PP-capable class. Deduplicated to one statement.
## Test results
99 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean,
boot script syntax-checked.
- test_timing_calibration_704.py (9, new): synthetic round-trip recovers a known
slope and intercept exactly; an unchanged rank is refused; the allowed
fallback is flagged and stays optimistic with stderr None (never 0.0);
slope uncertainty scales as 1/|dn|; the precision gate rejects rank0 and
passes rank1 at the same noise; samples_needed sizes the run and a larger
delta needs fewer; identical cuts, negative intercepts and mismatched rank
counts all refused.
- 90 existing tests unchanged and green.
No rung table re-derived; Slot-2's retro-gate still owns that. Boot ticket
remains non-self-arming and queued behind F4-r4.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…icit solver output Two independent calibrations converged on the same blind spot from the same structural cause, so it is now a first-class output rather than a footnote each of us rediscovers. Pool side (sgl-project#702 rev5): two metal points give two equations for three ranks. Rank1 binds at [28,20,16] and rank0 at [32,16,16]; rank2 binds at neither, so its free constant is bounded, not identified. Timing side (planner/timing_calibration.py): the slice-1a pair [28,20,16] <-> [29,19,16] is three different calibrators -- rank0 weak, rank1 strong, rank2 carrying ZERO information because its layer count does not move. Every candidate cut so far keeps rank2 at 16 layers, and the [33,15,16] discriminator does too. A window spent on it cannot identify rank2 on either axis. calibration_coverage() makes that visible BEFORE the window is spent. The key distinction the type encodes: a rank whose layer count does not move is UNIDENTIFIED, not weakly levered. Its regressor column is identically zero, so no sample size recovers it, and chunks_for_target() returns infinity rather than a large finite number that would invite someone to buy more chunks. The sample-demand formula is the two-sample standard error on a difference of boot means, N = (sqrt(2) * SD / (target * |dn| * ms_per_layer))^2, which reproduces the timing analysis independently: 583 chunks for rank0 and 30.0 for rank1 against its 584 and 31, at SD=3 ms and a 10 percent target. Agreement from two directions on numbers neither fitted to the other. publishable_intercept() gates an intercept before it reaches the solver: refused without a standard error, refused at N=0, and refused when |value| < 2 SE. Refusing beats emitting fixed_ms=0, because downstream a zero reads as a MEASUREMENT that the fixed cost is absent when in fact the pair could not see it. This is the same refusal the timing solver already makes, expressed as a shared gate so the two solvers cannot disagree about what counts as measured. suggest_rank_moving_arm() proposes arms in BOTH directions when a rank's term is load-bearing, since one direction confounds the term with any monotone drift. Tests: 9, red first. ruff clean. Nothing rig-specific: coverage is a property of the two layer vectors, sample demand a property of the lever and the noise.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… both become guards A boundary change is not finished when the range changes. Two structures follow the owned layer range and NEITHER raises on its own if it does not follow, so each is now an observer that refuses and rolls the flip back. ## (a) CUDA graphs bake the executed layer set -- and this corrects my own claim The decode graph captures the model's own forward (decode_cuda_graph_runner.py:1770), which iterates range(self.start_layer, self.end_layer). A CUDA graph records actual kernel launches, so the executed set is fixed AT CAPTURE TIME and a replay ignores any later change to _start_layer/_end_layer. After an unguarded flip the model reports the new range while every graph replay still runs the old one. That is the worst shape available here: the layer count silently reverts on exactly the fast path, and only under replay, so an eager smoke test would show the flip working. CORRECTION TO DESIGN_704 sec 3.7. I argued the arena's advantage over reallocation was that it "keeps captured CUDA graphs valid across a rung change". Wrong as stated. The arena keeps parameter ADDRESSES stable, which the graph's captured pointers do need, but the graph also bakes the executed layer SET -- so a rung change requires recapture EITHER WAY. The arena-vs-realloc fork narrows accordingly: recapture is common to both, and the arena's remaining advantages are avoiding the weight copy and avoiding reallocation. Slice 2 must therefore measure recapture as a cost of EVERY rung change, not as a penalty unique to the realloc design. ## (b) The KV and mamba pools are built for a span and indexed off its base Their layer id lists are filtered to the owned range at build time (model_runner_kv_cache_mixin.py:2460-2470) and rows are addressed as layer_id - pool.start_layer (memory_pool.py:1576, :2889). Two failures follow: - a layer newly activated outside the built span has NO ROWS AT ALL -- no KV for a full-attention layer, no mamba slot for a linear one; - the indexing BASE must not move. If a downstream rank's pool were rebuilt with the new start (rank1: 28 -> 29), every cached row would shift by one layer and be silently misattributed. So "load wide, run narrow" applies to the caches too: build the pools over the UNION, at a cost of one extra layer's rows per boundary that moves. In the slice-1a pair rank0 keeps start=0 and rank2 keeps start=48, so rank1 is the only rank whose base moves -- the one place a naive rebuild would corrupt silently. ## Test results 104 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. test_layout_boundary_704.py grows to 19. The graph guard carries a HAZARD-FIRST test: it demonstrates that a graph captured at rung A really does keep replaying rung A's layers after an unguarded flip, so the guard test below it cannot pass vacuously. Also: graphs with no recapture hook refuse the flip and roll back; a range escaping the pools' built span is refused; pools built over the union accept every rung; and the indexing-base arithmetic is pinned directly. Remaining for a GPU window is unchanged: boot-time union load (weights, KV and mamba pools) plus registration of these observers against the real registries.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ytes -> tokens Completes the instrument chain the retro-prediction gate needs. The chain is: budget - sum(budget_posts) = rest, then rest MINUS a per-rank reserve becomes available_bytes, then // cell_size becomes max_total_num_tokens. The first link I emitted in 84abeb7 and it is live on metal. This is the last one. The reserve between rest and available_bytes is the only term emitted nowhere, and it is not small or uniform. Backed out of the live [28,20,16] boot it is 8,848 / 3,818 / 5,164 MiB across the three stages -- a 2.3x spread. It tracks per-rank CUDA-graph capture, which depends on the shard, so config cannot predict it; my three external re-derivations missed by +20, -3.8 and -12 percent respectively. planner/plan.py's auto_reserve_mib docstring states the value is one "which the boot itself derives and logs". The deriving is real; the logging was not. A 172 MB boot log contains only the server_args echo (rank_auto_reserve_mib= 'auto', rank_user_reserve_mib=1024) and no derived per-rank figure. That is the third instance this session of a term the sizer computes and discards, after the budget posts and the lumped mamba post. With this line the reserve is recoverable as (rest - available_bytes) with no fourth re-derivation, and cell_size is published alongside so the bytes-to-tokens conversion can be checked rather than assumed. Independently confirmed en route: rest = budget - sum(posts) reproduces the logged rest exactly on all three ranks (14.472 / 7.894 / 8.376 GiB), so that link needs no further validation. Verification: compiles, logger already in scope at line 72, model_executor 594 passed / 15 failed with the 15 established pre-existing earlier by revert-and-rerun. The 3 ruff findings in this file are also pre-existing -- they sit at lines 372/375/380, ahead of this edit, and reproduce identically on HEAD. One log statement, no control flow change.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ytes -> tokens Completes the instrument chain the retro-prediction gate needs. The chain is: budget - sum(budget_posts) = rest, then rest MINUS a per-rank reserve becomes available_bytes, then // cell_size becomes max_total_num_tokens. The first link I emitted in 84abeb7 and it is live on metal. This is the last one. The reserve between rest and available_bytes is the only term emitted nowhere, and it is not small or uniform. Backed out of the live [28,20,16] boot it is 8,848 / 3,818 / 5,164 MiB across the three stages -- a 2.3x spread. It tracks per-rank CUDA-graph capture, which depends on the shard, so config cannot predict it; my three external re-derivations missed by +20, -3.8 and -12 percent respectively. planner/plan.py's auto_reserve_mib docstring states the value is one "which the boot itself derives and logs". The deriving is real; the logging was not. A 172 MB boot log contains only the server_args echo (rank_auto_reserve_mib= 'auto', rank_user_reserve_mib=1024) and no derived per-rank figure. That is the third instance this session of a term the sizer computes and discards, after the budget posts and the lumped mamba post. With this line the reserve is recoverable as (rest - available_bytes) with no fourth re-derivation, and cell_size is published alongside so the bytes-to-tokens conversion can be checked rather than assumed. Independently confirmed en route: rest = budget - sum(posts) reproduces the logged rest exactly on all three ranks (14.472 / 7.894 / 8.376 GiB), so that link needs no further validation. Verification: compiles, logger already in scope at line 72, model_executor 594 passed / 15 failed with the 15 established pre-existing earlier by revert-and-rerun. The 3 ruff findings in this file are also pre-existing -- they sit at lines 372/375/380, ahead of this edit, and reproduce identically on HEAD. One log statement, no control flow change.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ady for the boot Derives nothing. Reads the four instrumented terms and refuses what it cannot read. That discipline is the session's most expensive lesson: three external re-derivations of the sizer's arithmetic missed the measured boot by +20, -3.8 and -12 percent, because the reserve tracks per-rank CUDA-graph capture and config cannot see it. RankInstruments carries one PP stage as emitted numbers: budget, the three budget posts, rest, mamba allocated, available_bytes, cell_size, tokens, the per-layout arming floor and the layer split. verify_sizing_chain asserts the sizer's own identity, rest = budget - sum(posts). It reconciles on all three live stages within 0.9 MiB against a 3 MiB tolerance set by the posts' three-decimal-GiB rounding. This is the one link validatable without a new boot, and it pins that the emission is COMPLETE: an unemitted post would surface here as a residue. recover_reserve_mib recovers the reserve as rest - available_bytes rather than modelling it: 8,848 / 3,818 / 5,164 MiB on the live boot, a 2.3x spread. Two traps encoded as behaviour rather than comments: * mamba_charge_mib returns the ALLOCATION, not the budget post. The post under-charges by a constant 0.852 on every rank, so a solve fed from posts carries ~150 MiB/rank of systematic optimism -- which was the original plan for this field before the discrepancy was resolved. * the arming floor is NOT subtracted separately. It is held back after the profiler, so it already sits inside the recovered reserve; charging it again would understate every pool by ~2 GiB. A test pins that the floor fits inside the reserve and that the prediction still lands on the boot. predict_tokens_for_cut takes no default reserve and raises when it is missing, because the reserve does not transfer between layouts. An unbooted cut is an extrapolation and the call site has to say so. world_pool_tokens applies the PP min-rule and reports the binding stage. Retro-prediction status: the incumbent reproduces from instruments alone (436,766, chain reconciled). The remaining three boots close the four-boot gate the moment a boot carrying 2a6305d emits available_bytes directly -- until then that field is pinned from tokens x cell_size, which the instrument will confirm or refute rather than being assumed correct. Tests: 10, red first. Planner suite 2643 passed, 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean. Prior queue item for the record: the sgl-project#701 slice-3 rework landed earlier as 845ac92 (five gate holes, 24 tests, cross-pass falsifiers against a real PrefillAdder).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ntics inherited Slot-3's LadderInputs already consumes PhasePoolModel and requires an arming_floor_for provider, but it builds the model itself in pool_model_for. That is two solvers which must agree and are computed twice -- the shape this corpus keeps paying for. solve_rung_pool is the single surface the rung table and the boundary actuator call instead. Inherited rather than restated: * NO DEFAULT RESERVE. The per-rank reserve tracks CUDA-graph capture and does not transfer between layouts -- 3,818 / 5,164 / 8,848 MiB across three stages of one boot, a 2.3x spread. A rung with no reserve source raises, naming the spread, instead of returning a number that would read as measured. * provenance is a FIELD, so an extrapolated rung cannot be mistaken for a measured one further down the ladder; the extrapolated path also carries an explicit caveat string. * calibration coverage is returned whenever a reference cut is given, and every unidentified rank produces a caveat naming it. Relevant immediately: every candidate cut so far, [33,15,16] included, keeps rank2 at 16 layers, so rank2 is unidentified on both the pool and timing axes. Two metal-derived corrections are built in rather than left to the caller: * mamba is charged at the ALLOCATION, not the budget post -- the post under-charges by a constant 0.852 on all three ranks, worth ~150 MiB/rank of systematic optimism. * the arming floor is NOT subtracted here. It is held back after the profiler so it already sits inside the recovered reserve; charging it twice would understate every rung by ~2 GiB. It is still resolved and required, because a rung whose floor cannot be produced is a rung nobody has sized. Providers accept either a callable or a mapping so this can match existing call sites without a second surface appearing to serve them. One test caught itself: the unbooted-rung case originally tripped on a missing rest before reaching the reserve check, so it was not testing what it named. Fixed to supply rest and withhold only the reserve. Tests: 9, red first. Planner suite 2652 passed, 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean. Coordination note: I could not reach Slot-3 by name from this session to agree the signature first. The providers are deliberately callable-or-mapping and the field names are positional-free, so his call sites can be matched without a breaking change; if he wants per_stage keyed by stage rather than positional I will adapt rather than have him adapt.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ive requirements FOUR-BOOT GATE VERDICT: pinned available_bytes CONFIRMED at the binder, REFUTED at the non-binders, and the refutation is fully explained. PP0 pinned 6,261,477,376 metal 8,526,565,376 -26.57% PP1 pinned 4,472,483,840 metal 4,741,928,960 -5.68% PP2 pinned 3,577,987,072 metal 3,575,365,632 +0.07% <- binder I had pinned available_bytes = pool x cell for EVERY stage, which assumes every stage is exactly saturated. Under the min-rule only the BINDER is; the others carry slack. So the pin was determinate only at PP2 and is confirmed there to 0.07 percent. Same pattern in the reserve: mine 5,163.8 MiB vs metal 5,166.3 at PP2 (-0.05 percent), but +7.2 and +32.3 percent at PP1/PP0. The method is validated where it was decidable and the error elsewhere is the min-rule, not a modelling defect. World pool 436,446, binding rank PP2, both as the boot reports. BONUS SETTLED: cell_size is 14336 / 10240 / 8192 = exactly 7 / 5 / 4 attention layers x 2048 B. The sgl-project#702 attention-divisor now rests on the running system's ABSOLUTE values, not on a ratio, so the multiple-of-4 degeneracy that made [28,20,16] and [32,16,16] non-discriminating no longer matters. CORRECTION to my 0.852 claim, on F4-r4's challenge, which was right. The budget post is "mamba state pool + speculative intermediate state + prefill activation reserve" while the allocated line sums conv + ssm + intermediate_ssm + intermediate_conv. Those are not the same quantity, so "the mamba post under-charges mamba by 14.8 percent" is WITHDRAWN. What survives is stronger in kind: the post covers a nominal SUPERSET of the allocated line and is nevertheless SMALLER by 0.155 / 0.111 / 0.089 GiB, which cannot be legitimate. The gap is real and at least that size; which sub-term carries it needs the post's three components emitted separately -- the same missing-instrument fix a third time. The solver charges the ALLOCATION either way, because that is what the allocator took. Reserve figures corrected throughout to metal: 6,688 / 3,561 / 5,166 MiB, a 1.88x spread (my earlier 8,848 / 3,818 / 5,164 back-out was the non-binder artifact above). ENTRY POINT reshaped to Slot-3's five requirements: 1. attn_counts DERIVED from layer_families via PoolContext, never passed -- an inconsistent pair is now unrepresentable, and it is the sgl-project#702 divisor. 2. arming_floor_for required, no default (E3), and still NOT subtracted: it lives inside the recovered reserve and charging it twice would understate every rung by ~2 GiB. 3. binding stage and per-stage caps RETURNED, so the controller does not recompute them in a second code path. 4. PoolContext is built once by the caller; each solve is pure arithmetic with no NVML, census or config parsing, so an enumeration over thousands of cuts is cheap. 5. KV cell from kv_mib_per_token_per_attn_layer_from_config, converged VERBATIM from feat/704-prefill-ladder into pp_cut together with kv_dtype_width_bytes and its width table, so there is one implementation rather than two that must agree. It yields exactly 2048 B, matching metal. Optional explain=True returns per-stage weights/mamba/floor/free-for-KV terms. Tests: 21 across the two suites, red first. Planner suite 2654 passed, 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean. Slot-3 was not reachable by name from this session; providers accept callable or mapping so his call sites can be matched without a breaking change.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
Builds the instrument this ticket has now needed three times (budget posts, then available_bytes, now the post's own parts), and closes the recurring reconciliation blocker. The post has been labelled "mamba state pool + speculative intermediate state + prefill activation reserve" all along while emitting ONE number. That lump is why the illegitimate gap could not be attributed: the post nominally covers a SUPERSET of the "Mamba Cache is allocated" line -- it adds the prefill activation reserve -- yet measures 0.155 / 0.111 / 0.089 GiB LESS on PP0/PP1/PP2. A term covering more cannot legitimately measure less, but with a single number there is no way to say which of the three carries it. Next boot there will be. decompose_mamba_budget_post is a pure function, so it is testable without a boot. Two properties are pinned separately: the parts SUM to the lump exactly (a decomposition that changed the total would silently move the KV budget), and the residual is NAMED as the state pool rather than left anonymous. A negative residual is reported, not clamped -- a negative state pool is precisely the error this instrument exists to surface, and clamping would hide it. With no components the lump is returned unchanged, so the other budget branches are untouched. Wired at all five sites that actually move the number: four speculative- intermediate subtractions across the branches, plus the prefill activation reserve. One self-inflicted defect caught and fixed before push. The accumulator started as a bound method, which broke eleven ceiling-fit tests immediately: their SimpleNamespace doubles bind handle_max_mamba_cache as a plain function and cannot resolve self._note_mamba_component. That is the sgl-project#624 stub-drift class with me on the wrong side of it. Fixed by making the helper module-level and duck-typed -- production must not demand more of a stand-in than the behaviour under test needs -- and the reason is recorded in its docstring so it does not get "tidied" back into a method. Tests: 6, red first. model_executor suite 600 passed / 15 failed, and the 15 are the SAME pre-existing set as before this change (594 + the 6 new = 600). ruff clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…t#706 lanes) into the tool-call argument-loss fix Brings the branch onto the commit the 30030 tree currently holds so the deploy is a fast-forward. No conflicts with the adapter or detector changes -- the incoming commits touch KV sizing and mamba slicing.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… the instrument to fix it Desk model of the per-rank reserve, on existing metal. Verdict: UNDETERMINED on current data, but two things ARE settled and the recommended next step is not a fit. Correction to my own gate-close reasoning first. I had said only the BINDER's reserve is determinate. That is true of the PINNED available_bytes (derived as pool x cell) and false of the EMITTED one: the boot publishes each rank's own capacity -- 594,766 / 463,079 / 436,446 tokens, all different -- and the world pool is the min taken afterwards. So reserve = rest - available_bytes is determinate on EVERY rank. Three points, not one: PP0 28 layers / 7 attn / 21 gdn / 31,800 MiB budget -> 6,687.8 MiB PP1 20 / 5 / 15 / 18,800 -> 3,561.2 PP2 16 / 4 / 12 / 19,800 -> 5,166.3 FALSIFIED, from three points and without any fit: single-factor positive scaling in layers. PP1 and PP2 are both 3080s; PP1 carries MORE layers (20 vs 16) and has LESS reserve (3,561 vs 5,166). Reserve is non-monotone in layers, so no a + b*layers with b > 0 can produce the triple. Same for attn and gdn, which are proportional here. CONFOUNDED, exactly rather than statistically: on the incumbent attn = layers/4 on every rank (28/7, 20/5, 16/4 all exactly 4.0) and gdn = 3*attn. The regressors are perfectly collinear, so the three hypotheses give IDENTICAL residuals to the digit (+452.6 / -1357.9 / +905.3, max 1358). Budget-only fits better but still poorly (max 734). Two-factor forms have three parameters for three points, fit exactly with zero information, and are unidentifiable -- they must not be published as models. This is the same degeneracy that made [28,20,16] and [32,16,16] non-discriminating for the pool divisor; there it was broken by an absolute value, and no absolute value exists here because the reserve is not emitted decomposed. RECOMMENDATION: the fourth instrument, not a curve. Every previous blocker in this ticket dissolved the same way -- the sizer computed the term and did not emit it (budget posts, available_bytes, the mamba post's components). derived_rank_auto_reserve_mib returns 4,160 MiB UNIFORMLY when called with this boot's arguments while metal shows 6,688 / 3,561 / 5,166, so the held-back amount is not that function's output alone and fitting three points would be modelling a black box the process already knows. Emitting what is actually held back, per rank, settles the form in ONE boot. If a boot must discriminate instead, it needs a non-multiple-of-4 cut. [33,15,16] qualifies (ratios 4.125 / 3.75 / 4.0); per-hypothesis predictions are published so it becomes a free calibration point either way -- PP0 separates the forms by 165-455 MiB, PP1 by up to 219. PP2 is predicted identical by three of four forms, which is the rank2 blind spot again. Note [33,15,16] is no longer needed for the pool divisor (settled on absolute grounds by cell_size), so this is now its only reason to boot. Discipline retained: this note does not license carrying a reserve across layouts. It quantifies how wrong that is -- up to ~1.6 GiB per rank between two ranks of the SAME card type in a single boot, larger than most rung-to-rung pool deltas, which is precisely why extrapolated rungs cannot gate a window. Docs only.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
The fourth instrument, and the one the reserve analysis asked for instead of a curve fit. The quantity between the profiler's rest and what the configurator actually receives lands on metal at 6,688 / 3,561 / 5,166 MiB, while derived_rank_auto_reserve_mib returns 4,160 UNIFORMLY for the same boot's arguments. So the holdback is not that function's output. It is also not fittable from what exists: DESIGN_704 showed single-factor layer models are falsified outright by non-monotonicity (PP1 carries more layers than PP2 yet holds back less), and that layers/attn/gdn are EXACTLY collinear on any multiple-of-4 cut, giving identical residuals to the digit. Fitting three collinear points would have been modelling a black box the process already knows. Emitted at _config_from_budget, whose own comment calls it "the one funnel every sizing path reaches" -- so this catches every sizing path rather than one branch. The holdback is precisely the delta across _seam_adjusted_budget there, which is why no external re-derivation reproduced it (+20, -3.8, -12 percent on three attempts): it is a seam adjustment, not a graph-capture reserve, and I had been modelling the wrong quantity entirely. budget_holdback_mib and budget_holdback_fraction are pure, so they are tested without a boot, and they REPORT rather than sanitise. A negative holdback means the seam handed budget back -- a real and loud state -- and clamping it to zero would hide exactly the accounting-error class this ticket has spent four instruments chasing. The fraction helper returns None on a zero budget rather than dividing, so no caller has to guard it. A test pins the three metal reserves against the helper, so the instrument and the analysis cannot drift apart. Tests: 6, red first. model_executor suite 606 passed / 15 failed -- the same pre-existing set (600 + the 6 new). ruff clean. One log statement and two pure functions; no control flow change.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ently disabled Defect I introduced in 5f3e61f, caught in pre-boot review. budget_exhausted_message derives its --max-running-requests-ceiling advice by summing the posts named MAMBA_BUDGET_POST. Once the post is emitted as three named parts that sum matches nothing, so the hint vanished from the refusal message EXACTLY on the boots that carry the new instrument -- the worst possible correlation, since those are the boots someone would be debugging. Fixed with mamba_post_total_gb, which accepts both shapes (the lump and the parts) so callers stop matching a name, plus MAMBA_POST_PART_NAMES as the one declaration of what the decomposition emits. decompose_mamba_budget_post now takes the residual's name from that tuple, so the emitter and the summers cannot drift apart -- drift is precisely how the hint broke. A test pins that the emitted names equal the declared set, and another that unrelated posts (weights, GGUF scratch) are not swept into the total. Tests: 3 more, red first (9 in the file). model_executor 609 passed / 15 failed -- the same pre-existing set (606 + 3). ruff clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
Builds the instrument this ticket has now needed three times (budget posts, then available_bytes, now the post's own parts), and closes the recurring reconciliation blocker. The post has been labelled "mamba state pool + speculative intermediate state + prefill activation reserve" all along while emitting ONE number. That lump is why the illegitimate gap could not be attributed: the post nominally covers a SUPERSET of the "Mamba Cache is allocated" line -- it adds the prefill activation reserve -- yet measures 0.155 / 0.111 / 0.089 GiB LESS on PP0/PP1/PP2. A term covering more cannot legitimately measure less, but with a single number there is no way to say which of the three carries it. Next boot there will be. decompose_mamba_budget_post is a pure function, so it is testable without a boot. Two properties are pinned separately: the parts SUM to the lump exactly (a decomposition that changed the total would silently move the KV budget), and the residual is NAMED as the state pool rather than left anonymous. A negative residual is reported, not clamped -- a negative state pool is precisely the error this instrument exists to surface, and clamping would hide it. With no components the lump is returned unchanged, so the other budget branches are untouched. Wired at all five sites that actually move the number: four speculative- intermediate subtractions across the branches, plus the prefill activation reserve. One self-inflicted defect caught and fixed before push. The accumulator started as a bound method, which broke eleven ceiling-fit tests immediately: their SimpleNamespace doubles bind handle_max_mamba_cache as a plain function and cannot resolve self._note_mamba_component. That is the sgl-project#624 stub-drift class with me on the wrong side of it. Fixed by making the helper module-level and duck-typed -- production must not demand more of a stand-in than the behaviour under test needs -- and the reason is recorded in its docstring so it does not get "tidied" back into a method. Tests: 6, red first. model_executor suite 600 passed / 15 failed, and the 15 are the SAME pre-existing set as before this change (594 + the 6 new = 600). ruff clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
The fourth instrument, and the one the reserve analysis asked for instead of a curve fit. The quantity between the profiler's rest and what the configurator actually receives lands on metal at 6,688 / 3,561 / 5,166 MiB, while derived_rank_auto_reserve_mib returns 4,160 UNIFORMLY for the same boot's arguments. So the holdback is not that function's output. It is also not fittable from what exists: DESIGN_704 showed single-factor layer models are falsified outright by non-monotonicity (PP1 carries more layers than PP2 yet holds back less), and that layers/attn/gdn are EXACTLY collinear on any multiple-of-4 cut, giving identical residuals to the digit. Fitting three collinear points would have been modelling a black box the process already knows. Emitted at _config_from_budget, whose own comment calls it "the one funnel every sizing path reaches" -- so this catches every sizing path rather than one branch. The holdback is precisely the delta across _seam_adjusted_budget there, which is why no external re-derivation reproduced it (+20, -3.8, -12 percent on three attempts): it is a seam adjustment, not a graph-capture reserve, and I had been modelling the wrong quantity entirely. budget_holdback_mib and budget_holdback_fraction are pure, so they are tested without a boot, and they REPORT rather than sanitise. A negative holdback means the seam handed budget back -- a real and loud state -- and clamping it to zero would hide exactly the accounting-error class this ticket has spent four instruments chasing. The fraction helper returns None on a zero budget rather than dividing, so no caller has to guard it. A test pins the three metal reserves against the helper, so the instrument and the analysis cannot drift apart. Tests: 6, red first. model_executor suite 606 passed / 15 failed -- the same pre-existing set (600 + the 6 new). ruff clean. One log statement and two pure functions; no control flow change.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ently disabled Defect I introduced in 5f3e61f, caught in pre-boot review. budget_exhausted_message derives its --max-running-requests-ceiling advice by summing the posts named MAMBA_BUDGET_POST. Once the post is emitted as three named parts that sum matches nothing, so the hint vanished from the refusal message EXACTLY on the boots that carry the new instrument -- the worst possible correlation, since those are the boots someone would be debugging. Fixed with mamba_post_total_gb, which accepts both shapes (the lump and the parts) so callers stop matching a name, plus MAMBA_POST_PART_NAMES as the one declaration of what the decomposition emits. decompose_mamba_budget_post now takes the residual's name from that tuple, so the emitter and the summers cannot drift apart -- drift is precisely how the hint broke. A test pins that the emitted names equal the declared set, and another that unrelated posts (weights, GGUF scratch) are not swept into the total. Tests: 3 more, red first (9 in the file). model_executor 609 passed / 15 failed -- the same pre-existing set (606 + 3). ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…-pointed This was assigned as a pin-only line-number fix, on the strength of a diagnosis I gave in the previous report. That diagnosis was WRONG and is withdrawn: I said the pinned assert had moved to server_args.py:13693-13694 and changed shape to view.disable_overlap_schedule. That site is _validate_mamba_no_buffer, an unrelated mamba assert. I read a bare disable_overlap_schedule grep hit as the pinned guard without reading its context -- the same infer-from-a-hit error sgl-project#635 named ("a constant is evidence about the code that reads it and nothing else"). Reading the real guard shows the row was not merely mis-pointed. The single conjunction the row describes has been SPLIT INTO TWO statements that went in OPPOSITE directions: server_args.py:17583 if self.pp_size > 1: server_args.py:17584-17586 assert self.disable_overlap_schedule server_args.py:17598 assert self.speculative_algorithm is None or self.enable_phase_flip 1. The speculation half is still a hard assert, but it GREW AN EXEMPTION: "or self.enable_phase_flip". Per the comment at :17587-17597, a phase-flip instance runs PP for prefill with no draft worker built at all and arms speculation on the TP decode stack at cutover, so the combination is enforced by construction instead of refused. 2. The overlap half is now effectively an auto-disable. arg_groups/ overrides.py:2163 _pipeline_parallel_overlap_disable sets disable_overlap_schedule=True and logs a warning in a post-process pass that runs BEFORE the assert, so that assert can no longer fire. So the row's claim -- "a hard assert, not an auto-disable ... every PP number is therefore a no-spec number" -- was wrong in both halves, and wrong in the direction that matters to this strand: it forbade on paper the exact configuration sgl-project#704 is built on (PP=3 prefill flipping to TP=3+EAGLE decode). Updating only the line number would have re-pointed a false verdict at the code that falsifies it. Row corrected: BLOCKED for PLAIN pipeline parallelism, EXEMPT under --enable-phase-flip, with the auto-disable of the sibling half recorded so the register does not imply both halves still refuse. Level unchanged at BLOCKED; whether a phase-flip-exempt row should keep that level is a register-owner call, not mine. Tests: the two reds were ready-made falsifiers, but they pinned the OLD single conjunction, so they could not express the split. Rewritten to pin both halves plus the exemption, each with its own falsifier proven by breakage: test_evidence_cites_land_on_BOTH_halves_of_the_guard red when the evidence carries the stale :16240-16245 cite test_the_register_records_the_PHASE_FLIP_EXEMPTION red when the row text is reverted to the pre-#704b wording test_the_overlap_half_is_now_an_AUTO_DISABLE red when the row stops naming disable_overlap_schedule. Note the weaker form of this check passed against the stale text: "auto-disable" appears there too, inside "not a quiet auto-disable" -- the opposite claim. Naming the field is what makes it a real check. test/registered/unit/planner/test_rejected_evidence_pins.py 2 failed, 2 passed -> 6 passed test/registered/unit/planner full suite, no remaining failures Hermetic, CUDA_VISIBLE_DEVICES="", interpreter /spinning/htsglang-gpu/.venv/bin/python3 (system python3 cannot collect these suites; see 2c0542e). Register text and its pins only; no engine behaviour touched. Register rows carry a length budget (test_wizard_v2.py:482-490: why <= 400 chars, gain/cost <= 120) -- "a paragraph in the gain field is how this becomes the wall it replaced". My first pass at the corrected row blew the why budget at ~600 chars and turned that test red; the detail moved into evidence, which carries no budget. why is now 295. Full planner suite: 2805 passed, 0 failed (2 failed at the parent commit).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
… the sizer must NOT claim it VERDICT: no dead reservation found. The gap is the arming floor (deliberate) plus min-rule residue (structural), so this is the labeled accounting table the brief asks for in that case, not a forced fix. Per-rank, from the sgl-project#704/sgl-project#707 instrument chain rather than modelled: rank budget weights mamba rest holdback allowed used slack PP0 31800 16064.5 916.5 14819.5 6690.1 8129.5 5964.7 2164.8 PP1 18800 10061.8 654.3 8083.4 3562.7 4520.7 4260.5 260.2 PP2 19800 10699.8 523.3 8576.3 5167.9 3408.4 3408.4 0.0 Total slack 2,425 MiB, and the shape is the finding: it is almost all on PP0, and the BINDER PP2 has exactly zero. HOLDBACK is deliberate, established in sgl-project#707: the pool is capped so the resting free column still holds the arming floor. Sizing below it is the boot that holds the corridor and never flips (sgl-project#656 E/G). Not claimable. SLACK is structural and still not claimable BY THE SIZER. The pool is one global token count, so a non-binding rank cannot spend its extra allowance without the binder moving -- PP0's 2,164.8 MiB is not booked-and-untouched, it is unreachable at this cut. A sizer that claimed it would hand the pool tokens PP2 cannot back, and PP2 is the rank that OOMs. That is the sgl-project#593-family direction the brief warns about, so I wrote NO red-first test for it: the falsifier would have to assert a wrong answer. WHERE THE CAPACITY ACTUALLY IS: the slack is a property of the CUT, and that is already solved on another strand. sgl-project#702 rev5 found the incumbent [28,20,16] is not pool-optimal -- the binding rank switches, so moving layers OFF the binder raises the pool, and [30,18,16] gives +20 percent AND 1.11x pipelined prefill. Claim it by rebalancing the cut, not by relaxing a reserve. WHAT I DO NOT CLAIM: the ticket's 2.0 / 5.7 / 3.7 GiB per card is a DIFFERENT operating point from the instrumented boot, and the instruments that would attribute it per-term only exist on boots carrying 2a6305d / 5f3e61f / f55c1a8. Mapping this table onto those three numbers is the cross-boot arithmetic this strand has already retracted twice, so I did not. Ask for the next boot to capture the three instrument lines at that operating point and the table re-runs against it directly -- a window item. One thing that CAN be said without it: against the measured arming floors (1728 / 1825 / 2467 MiB) the per-card free exceeds the ~1024 MiB corridor target BY CONSTRUCTION, because on a flip-enabled boot the arming floor is the binding level, not the corridor. Reading the gap against 1024 overstates it on every rank. Docs only. No sizer change, deliberately.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…le, and what the phase matrix owes Docs only. No code changed, nothing built or wired. ## sgl-project#702 decision table (docs/DECISION_702_cut_table.md) Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool reads 436,275 here against the 436,766 in older notes; all percentages use 436,275 so they are internally consistent. cut attn pool speed noise(14.1%) gate [28,20,16] (7,5,4) 436,275 1.0000 -- is the gate [30,18,16] (7,5,4) 436,275 (+0.0%) 1.1111 BELOW IT PASSES [33,15,16] (8,4,4) 415,859 (-4.7%) 1.3299 above FAILS Switch cost 1575 ms from the incumbent either way (distance-independent, per #704a); payback 15.75 s and 6.35 s respectively. THE TRADE: the two candidates fail in OPPOSITE places, which is the decision. [30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is offered; the pick is the user's. [30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the (7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and [517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two layers between PP0 and PP1 cannot move a pool PP2 sets. CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool, which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut including the incumbent) and therefore not attributable to any cut. SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates, per lead depth, ONLY the tail split that minimises pipelined time. At n0=30 that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated -- yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275 (+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a roomier, slightly slower cut at the same lead depth cannot appear on it. [30,18,16] strictly dominates the incumbent (equal pool, more speed) and was invisible for exactly that reason, which is why it never reached the sgl-project#702 block. The fix would be to keep the Pareto set over (pool, speed) per lead depth instead of the speed-argmin; other pool-preserving cuts may also be hidden. Validation-boot evidence is specified in falsification order, and includes the one that matters most: for [30,18,16] the prefill A/B is expected to be INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its own noise, and that result must be rejected rather than celebrated. Appended to the morning plan file as a dated block per its amendment convention; head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). That file is not version-controlled, hence this copy. ## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md) Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law 2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485 are the same ticket, not a number collision: the matrix doctrine plus its prefill-column solver, and the seam/memory certification gating it. VERDICT: PARTIALLY DELIVERED, do not close. DELIVERED prefill column for the attention-vs-GDN/linear boundary (c08f613, b675630, b3c9802, e645aa7), with a second axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503 447249a). Its own feasibility facet (407b382) is RED: the threshold is not reachable at the certification pool. SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED reasoning -- against the honest proportional (uneven-TP) baseline the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round. But this closes ONE CANDIDATE CELL (concentration), not the column. Named tension: the directive says single-family arms are DIAGNOSTIC only and must never be phrased as a phase-level verdict, and that commit is worded as one. The narrow reading is adopted. OPEN O1 no (family x phase) structure exists anywhere -- pp_cut.py solves prefill only, family_split.py returns by_family with NO phase argument or dimension; the "matrix" is prose over two single-phase solvers. O2 the decode solver is DEAD CODE: solve_family_placement has exactly two references, its definition (family_split.py:141) and its test, zero call sites. O3 vocab, experts, nonlinear kernels and per-quant-lane linear layers were never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR and training tenants -- entirely untouched, and the largest open surface. O5 the prefill column is not arm-able while Facet B is red. NOT sgl-project#485 sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming machinery. They consume the prefill-column solve as an input and must not be counted toward the matrix; counting them would inflate its delivery with work on a different axis. Status in one line: one of two columns, one of many families, one of several workloads -- and the table is still prose. Every load-bearing claim spot-verified against the tree rather than taken from the survey: directive text read at source, solve_family_placement call sites grepped, d937d5f's baselines read from the commit body, [30,18,16] priced directly rather than inferred from frontier membership. Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py). Desk only, no GPU, deploy tree untouched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 18, 2026
…ne consumer Boot v7pp8 booted clean with the ledger no longer refusing, and still died: PP2 priced at weights 6.248 + mamba 0.384 + pool 11.923 GiB against an 18.55 GiB budget, 0.15 GB free, dead on a 32 MiB decode allocation. The pool was not mis-priced. There was nothing left for the transients a price does not cover. rank_user_reserve_mib is the corridor headroom and it had exactly ONE consumer in the tree, phase_flip_runtime.py:473. A plain PP boot driven by --rank-gpu-memory-mib therefore spent its budget to the last MiB, and no auto-reserve line appears anywhere in the boot log to make up for it. Held back as a NAMED post, so the planner reads it off the same 'KV budget posts' line it already consumes rather than re-deriving it -- the discipline sgl-project#704 established when it made that line the instrument. GATED TO THE GAPPED PATH. Every shipped configuration was solved against today's arithmetic; moving the reserve into the posts globally would shrink each of their KV pools by a gigabyte per card, and no boot here has evidence for changing solved, measured configurations. The gapped cut is the new path and the one with the demonstrated OOM, so it is the only sizing that moves. The corridor is a target and not a floor, so the holdback never drives the pool negative and says so when it declines. TESTS. 23 passing, including that a contiguous set and a boot with no layer set price EXACTLY as before. One test earned its keep immediately: the 'or 1024' idiom copied from the flip path conflates an explicit 0 with unset, so the reserve could not be switched off. Fixed in the code.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
… while the peers spin
BOOT-PROVEN (boot 22, boot_855_1033b_0840f82601_0831_131955, 13:24-13:26).
The first TP forward after a pp_to_tp cutover touches a shape specialisation
never loaded in this process. Rank 0's crash-time stack (log line 53248 ff.):
_dcp_write_scatter (flashinfer_backend.py:2574)
run (triton/runtime/jit.py:743) -> _init_handles (compiler.py:466)
loadBinary -> cuModuleLoadData (libcuda.so) + 11 libcuda frames
NOT the compiler -- cuModuleLoadData. The module is built; LOADING it into the
CUDA context blocks, because the load needs the device and the device is
saturated by the peers' barlink BAR1 spin kernels, which are waiting in
all_gather for this very rank. The cycle closes and the spin deadline fires:
Bar1CollectiveAborted (ranks 1/3 and 2/3, group flip_dcp:0) -> SIGQUIT.
WHY THE EXISTING MECHANISM MISSED IT, measured: 352 build-window lines in that
boot, ZERO in 13:24-13:26, all 352 carrying 'full cuda-graph capture warmup'.
cold_build_window had exactly three production callers (barlink BAR1 build,
sampling warmup #603b, capture warmup) and none is on the path a cutover
re-dispatches into. sgl-project#640 on a path sgl-project#615 never saw. The fix makes that path a
caller rather than building a second mechanism beside it.
A STALE COMMENT NEARLY REFUTED THIS FIX, and the correction ships with it.
sampler_warmup.py said wrapping a lazy build in cold_build_window "does NOT
work ... the window is PROCESS-LOCAL". True when written (8bddb93,
2026-08-06); falsified ONE DAY LATER by sgl-project#615 (38ec4fb, 2026-08-07), which
hooked publication into cold_build_window so "every existing call site
therefore becomes group-visible without moving". Never revised, both ancestors
of this pin, verified with git log -S rather than assumed. The paragraph is
annotated in place rather than deleted, because it is still right about ITS
module (warm-at-boot + barrier REMOVES the race; a window only EXTENDS it).
WHY A WINDOW OVER THE REAL FORWARDS, NOT AN ENUMERATED WARM SET. The warm set
is not knowable by inspection -- it varies with direction, spec-decode, the
sgl-project#887 one-chunk grant and any backend swap -- and missing one member reproduces
the wedge exactly, at the next first-loader. Whatever loads, loads under the
window. Residual stated in the code: this EXTENDS deadlines (900 s cap) rather
than removing the race; boot 22's stall was ~150 s, so the cap is not binding.
FIRST-LOADER CENSUS, as a standing table: devtools/CENSUS_1033c_first_loaders.md
-- every first-loader site x triggering state change x covering window x
collective proximity. Rows 1-6 verified at file:line (row 5 on metal); rows 7
(runtime recapture / drafter switch) and 8 (rung change sgl-project#704, resume-restore
sgl-project#89) are written UNVERIFIED, so the table is a lower bound on the covered set
and never a proof that nothing else is exposed. It also names the inverse
column (armed windows that may cover paths that no longer first-load -- the
352 are a count of WARMUPS, not of BUILDS, and the window carries no
modules-loaded counter, so live and dead coverage are indistinguishable from
the log) and answers the upstream-minimal question: there are not four window
mechanisms but two plus a front door, with one site (barlink_device.py:865)
bypassing the front door. Named, not rebuilt.
#1033d, same boot, INDEPENDENT DEFECT, minimal hardening only: the prefetch IO
aux thread caught only Empty, so a page whose geometry did not match the
incoming binding (mha.py:556 reshape, '[2,16,1,4,256]'=32768 against a 16384
page) ended the thread three seconds before the cutover -- killing no process,
setting no exit code, appearing in no health probe, and leaving storage
prefetch dead for the rest of the boot. It now fails the OPERATION (host slots
released, so the requester gets a refusal instead of an unreachable
completion), logs loudly with a counter, and stays alive. The broad except is
defensible only because the alternative is silent thread death with no
supervisor above the loop; the underlying two-geometry host pool across a flip
(sgl-project#718/sgl-project#719/sgl-project#875 family, and the fork's own 'sgl-project#939 RE-HOME VIA RE-READ ... source
page 16384 elems vs destination page 32768 elems' line names the same mismatch
two lines earlier while handling it correctly) is NOT fixed here and is its own
posten.
DESK CHECKS, matched to each edit's failure class:
* devtools/check_1033c_cutover_window.py -- 19 cases, hermetic, driving the real
run_batch wrapper: RED-FIRST arm reproducing the boot-22 condition (counter
unarmed -> no window -> nothing published), first forward covered AND
published, window closes (the sgl-project#431 open-without-close shape), budget finite
and steady state byte-identical, window closed and budget spent even when the
forward raises, and an AST check that the arming sits under no `if` (the naive
grep form of that assertion failed on a COMMENT -- fixed to ask the AST).
* AST check on the aux thread: broad handler present, logs at ERROR, releases
the failed operation's host slots, and continues.
* Method-split check: wrapper delegates, window only on the armed path, lazy
import off the fast path, inner body intact.
ruff F401/F821/UP037 clean.
DESK-PROVEN. Metal proof is boot 23.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 20, 2026
The morning's module decides WHETHER to switch and what it costs. It could not be booted: four pieces of the running process do not follow a boundary change, and every one of them fails QUIETLY -- none raises, all produce plausible output from the wrong layer set. Each falls here with a hermetic test of its own. 1. ONE SOURCE OF TRUTH FOR THE RANGE. ModelRunner snapshots the range at init (model_runner.py:1185-1186) and derives three further things from the snapshot: num_effective_layers (:1200-1212), which is the KV pool's layer_num= at nine call sites, and the hybrid-SWA layer id lists (:1729-1745). sgl-project#704's actuator moves only model._start_layer, so all three stayed at their boot values. New ModelRunnerRangeMirror pulls them along, registered as an observer so a failure rolls the flip back. The SWA narrowing is DESTRUCTIVE -- adjust_hybrid_swa_layers_for_pp assigns the filtered list back over the attribute it filtered -- so re-running it after a widening intersects with an already-narrowed list and can never recover a dropped layer id. The mirror therefore snapshots the PRISTINE lists and re-derives every rung from those. It is built at the UNION and checks that it was, because only there are they pristine. The inclusive `end_layer + 1` bound is reproduced, not silently fixed: a mirror that corrected it would make the first flip change the lists for a reason unrelated to the boundary. Under SGLANG_PP_LAYER_SET the move is a refusal by name, not a recount -- the env names one fixed set per stage. 2. GRAPH RECAPTURE. A captured prefill graph records the decoder loop over owned_layer_ids(start, end) (qwen3_5.py:1720), so the executed layer set is baked at capture time and a replay ignores the boundary. New PrefillGraphRecaptureRegistry is the registry cuda_graph_observer already asks for (captured_range + recapture), turning that observer from a refusal into a repair. After a flip every captured shape is re-recorded or DISCARDED (new discard_shape on the two backends that hold _graphs) -- the state this exists to make unreachable is "still captured under the old range and still replayable". hot_shapes buys back the cost: cold shapes go eager rather than being re-recorded. per_shape_seconds must be measured; cost_seconds refuses rather than inventing the term, because it is a term in decide(). prefill_cuda_graph_runner.capture() is split into capture() + _capture_session() so a recapture enters the IDENTICAL context stack rather than an approximation of it -- two of those four contexts change what gets recorded. 3. POOLS OVER THE UNION, AND THE UNION PRICED. The ~20 pool sites do not each decide the range; they all read the same three ModelRunner attributes. So union_layer_window() widens them at the one place they read from, for the duration of the build, and every reader follows without knowing it did. union_layout/union_kv_costs/require_union_fits then put a NUMBER on what that costs: pool tokens lost per card. Deliberately not converted to seconds -- that needs a measured recompute rate, and a fabricated term beside measured ones is the mixed-unit class derive_x_star refuses by name. decide() takes union_costs + required_context_tokens and refuses the layout PAIR up front when 262k no longer fits, and takes recapture_s as a real term charged to each leg that moves. 4. MOVER ROUTE. barlink capture_safety() already states that BAR1 send is capturable and recv is not (barlink_bar1_p2p.py:181). require_breakable_route refuses by name BEFORE the first op. The danger is not the raise but the HALF that records: a capture holding the deposits and not the collects replays as a mover that ships bytes into peer windows and never lands them, silently, on the fast path. STAGE ORDER. Checked as asked, and the premise needed correcting twice. (a) The PP stage number is WELDED to the rank number (server_args.py:10700, parallel_state.py:3945-3962); the send/recv layer would follow a permuted rank list (neighbours are positional, :1319-1350) but nothing constructs one. The equivalent move is free and supported: permute the CARDS, --rank-gpu-id 0,2,1. StagePermutation does it to all five rank-indexed vectors at once or refuses -- planner/flags.py:2619 says "rank 0 IS the 5090" and the tokvec (33,13,18) and reserve (3000,2200,2200) were measured against that sentence, so a half-applied permutation hands the 5090's KV share to a 3080 with no error at all. (b) The gain is smaller than estimated. The swap does NOT speed up the slowest LINK -- a 3080-to-3080 hop at 6.58 GB/s then becomes the worst one. And on the 64-layer geometry the switch goes 0.378 s -> 0.234 s, a 38 % cut, not the halving: stage 1 also sheds four layers to stage 2, and once the 5090's hop is fast that hop binds. placement_switch_seconds prices the bytes each hop actually carries; slowest_hop_seconds is documented as the smaller question. Two bugs were found by the new tests rather than reasoned about: the mirror wrote start/end BEFORE the refusal check (a half-applied update the actuator's rollback would not undo), and RecaptureReport reported the new range as the old one. Tests: tests/moe_offload/test_p_layout_inprocess_0920.py (39 cases, one per blocker plus the stage order) and test_p_layout_whole_boot_0920.py (12 cases), a CPU whole-boot probe standing up three fake PP ranks -- model, runner, KV pool, capture ladder, BAR1 mover -- and driving P39 -> P32 -> P39. The ROUND TRIP is the assertion: a one-way switch passes with a destructive narrowing, a moved indexing base and a discarded graph ladder in it. Gate tests/moe_offload: 43 failed (unchanged baseline), 638 -> 689 passed, 41 skipped (the morning's 657 counted the 19 test_layout_boundary_704 cases). test_layout_boundary_704.py 19/19, test_p_layout_switch_0920.py 68/68. Targeted graph-backend/capture suites 81 passed 1 skipped. test_breakable_graph_error_masking 7 failed = baseline, verified against a detached checkout of 4e11cd1 rather than assumed. black/isort/ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.