Repository navigation
docs: init readthedocs support - #783
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
…ect#783) * Update test_utils.py * Update test_utils.py * Update test_ascend_debug_tensor_input_file.py * Update run_suite.py * Update run_suite.py * Change lint job to run on linux-amd64-cpu-1 * Update lint.yml * Refactor lint workflow for improved configuration * Update daily-build-test-npu-innersource.yml * Fix syntax error in merged PR benchmark workflow * Update pr-test-npu-debug.yml * Clean up blank lines in activation.py Removed unnecessary blank lines in activation.py. * Fix indentation in attention layer methods * Update io_struct.py * Update dbrx.py * Update phi.py * Update test_utils.py * Clean up unused imports in test_vlm_utils.py Removed unused imports from the test file. * Add blank line before main execution block * Add model path to TestLing class * Update test_ascend_llm_models_phi_4_multimodal.py * Update test_ascend_llm_models_smollm_1_7b.py * Update test_ascend_sleep_on_idle.py * Update test_ascend_llm_models_Qwen3_32B.py * Update test_serving_chat.py * Update run_suite.py * Update test_server_args.py * Update daily-build-test-npu-innersource.yml * Update pr-test-npu-debug.yml * Update activation.py * Update torch_native_backend.py * Update io_struct.py * Update phi.py * Update test_utils.py * Update test_ascend_llm_models_Stablelm_2_1_6b.py * Update test_ascend_llm_models_ling.py * Update test_ascend_llm_models_smollm_1_7b.py * Update merged-pr-benchmark.yml * Update run_suite.py * Update daily-build-test-npu-innersource.yml
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
The prefix cache was dark instance-wide: 2892 of 2892 prefill batches on
boot_735_standS read `#cached-token: 0`, and three identical ~6k-token
prompts each came back with no cached tokens. The read path was the
suspect, but the tree was simply empty -- nothing was ever retained.
MambaComponent.prepare_for_caching_req answers with a cache_len, and
UnifiedRadixCache.cache_finished_req min()s every component's answer into
one shared effective_cache_len. When a request's end did not land on the
--mamba-checkpoint-interval grid, the component answered 0. That 0 is a
statement about mamba's OWN anchor -- it has no on-grid state to file --
but as a shared minimum it collapsed the whole insert and threw away the
full-attention KV as well, which does not depend on the mamba grid at all.
With an interval coarser than the traffic the veto is total: the rig ran
8192 against ~6k-token prompts, so EVERY end is off-grid, every insert
cached nothing, and every match therefore returned zero. The branch the
rig actually ran (no --enable-mamba-extra-buffer) was also silent, which
is why an entirely dark cache looked like idle traffic for 2892 batches.
The tree already documents the intended behaviour in two places: "A leaf
survives on its Full value alone; auxiliary components (SWA, Mamba) may
legitimately hold only a tombstone" (unified_radix_cache.py), and, for
int8 pool exhaustion, "the node is then inserted without a mamba value
... so the KV stays cached and only the mamba resume point is lost".
The off-grid path contradicted both.
Declining the anchor now returns None ("no constraint from me") instead
of 0, so the node is retained at full length with a mamba tombstone, and
commit_insert_component_data treats a missing state as that tombstone
rather than asserting on it -- the assert also made the documented int8
exhaustion path a latent crash. Retention and resume-eligibility stay
separate decisions: a tombstoned node carries no state, so the match
validator still refuses it as a resume anchor and no request can resume
on a prefix whose recurrent state is absent. Declines are counted and
logged, so a grid coarser than the traffic announces itself.
Tests (hermetic, CUDA_VISIBLE_DEVICES=""):
TestUnifiedRadixCacheMambaCheckpointGridRetention
test_off_grid_end_retains_full_kv red before, green after
(before: AssertionError: 0 != 64, "an off-grid mamba end must
not discard the full-attention KV"); also asserts matched == 0,
pinning that a tombstone is never served as a resume anchor
test_on_grid_end_retains_and_matches green before and after
test_grid_disabled_retains_and_matches green before and after
The two green-before tests are the can-fail control: the same
assertions pass whenever the grid is reachable, so the red one
discriminates the defect rather than the harness.
Full test_unified_radix_cache_unittest.py, before vs after: 212 errors
and 648 skips in both runs, identical failure sets, zero new failures
(the 212 are pre-existing CPU-environment errors on a hermetic host).
ruff clean on mamba_component.py (the 6 findings in the test file are
pre-existing, lines 1637-2182); codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
…ey units Second defect on the same dark-cache path, and the one that actually governs this rig: with --speculative-algorithm NEXTN the radix key is an EAGLE bigram view, where 64 raw tokens are 63 bigrams. The checkpoint grid was then tested in two different unit systems. retention (prepare_for_caching_req) token_ids_len RAW tokens insert backstop (commit_insert_...) len(params.key) KEY units match walk (create_match_validator) cum_tokens KEY units So an anchor had to satisfy `interval | n` on the retention side and `interval | (n - 1)` on the other two. For any interval above 1 that pair is unsatisfiable, at every request length: n on the grid means n-1 is not, and the backstop refused the anchor retention had just certified; n-1 on the grid means retention declined first. EAGLE plus --mamba-checkpoint-interval therefore could not produce a single prefix hit, independent of prompt length or of how reachable the grid was. Measured before the fix (desk, FULL+MAMBA, interval 64, eagle): n=64 retained 63 matched 0 n=65 retained 64 matched 0 n=128 retained 127 matched 0 n=129 retained 128 matched 0 after: n=64 retained 63 matched 63 n=65 retained 64 matched 0 n=128 retained 127 matched 127 n=129 retained 128 matched 0 The off-grid lengths still match 0 on purpose: retained, no anchor, so not resumable. The grid is a statement about how many tokens the recurrent state has consumed, so raw tokens are the correct unit and the two key-unit sites convert through `_raw_token_pos` (identity outside EAGLE, so the non-EAGLE lineage is byte-identical). Tests: test_eagle_on_grid_end_matches new, red before / green after test_eagle_off_grid_end_retains_without_anchor new Can-fail control: disabling ONLY the `+1` in `_raw_token_pos` reddens test_eagle_on_grid_end_matches with "AssertionError: 0 != 63: an on-grid EAGLE anchor must be accepted on match" and leaves the other four green, so the test discriminates this fix and not the harness. Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips before and after, identical failure sets, zero new failures (the 212 are pre-existing CPU-environment errors on a hermetic host). ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
…sides The two fixes on this branch make prefixes reusable again, and the failure mode that has to stay shut while doing so is sgl-project#767: a resume whose KV and recurrent state belong to DIFFERENT positions, which surfaced as a foreign request's text continuing into a fresh prompt. Widening a match is exactly how that comes back, so the guard is now asserted, not argued. test_resume_stops_at_the_position_the_state_was_taken builds a path where deeper KV genuinely exists past the anchor -- otherwise the assertion would be vacuous: A: 128 raw tokens, on-grid at interval 64 -> files an anchor at raw 128 B: 168 raw tokens extending A, off-grid -> KV retained, NO anchor The path then carries 167 bigrams of KV while the deepest snapshotted state is at raw 128, and the test requires both: that the tail really was retained (167), and that a query for B resumes at 127 bigrams (raw 128), never 167. Can-fail control: making is_resume_candidate accept anchorless nodes reproduces the sgl-project#767 widening exactly -- "AssertionError: 167 != 127: the resume point must be the raw position the state was taken at" and simultaneously reddens the two tombstone guards with "AssertionError: 64 != 0: a tombstoned node must never serve as a resume anchor" so all three assertions discriminate the rule rather than the harness. Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips, identical failure set to the pre-sgl-project#783 baseline, zero new failures. ruff clean on the added code (the 6 findings in this file are pre-existing, lines 1637-2182).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 20, 2026
First boot of this branch died on the pool ledger:
ValueError: pool memory leak detected! [full] total=161378,
available=40952, evictable=130, protected=0, session_held=0,
uncached=0, withheld=120418
40952 + 130 + 120418 = 161500 against a total of 161378: 122 slots
unaccounted, and the log named them -- "off-grid end 122 (interval 512)".
Cause, and it was mine. `cache_unfinished_req` inserts and then IMMEDIATELY
re-matches, transferring ownership on the strength of that match
(`req.cache_protected_len = len(new_indices)`). An anchorless node is
deliberately unmatchable, so the round trip comes back empty: the tree took
121 of the request's 122 slots while the request kept using -- and later
freeing -- the same slots, with cache_protected_len falling back to 0.
Retention is therefore correct only where ownership actually transfers:
on FINISH, where the request is over and the tree takes the KV outright.
Mid-flight steps keep the pre-existing "cache nothing this step" behaviour;
their KV stays with the request and is retained when it finishes. The two
answers now come from one documented helper, `_decline_retention`, instead
of an unqualified `return None`.
This does not weaken the original fix. The repro is a repeated prompt, and
what has to survive is the COMPLETED request's KV -- that is the finish
path. On-grid chunk ends are unaffected either way: they file an anchor,
the re-match returns indices, and ownership transfers normally.
Tests:
test_off_grid_unfinished_step_conserves_pool_slots new, green
Asserts OWNERSHIP, not allocator conservation: mid-flight the slots are
legitimately still held by the live request (the running ledger counts
them as `uncached`), so what must hold is that the tree did not ALSO
claim them.
Can-fail control: making _decline_retention return None unconditionally
reddens it with "AssertionError: 121 != 0: an off-grid mid-flight step
must cache nothing", which is the double claim the boot reported.
The first version of this test passed under that same control because
full_untruncated_fill_ids was never populated, so get_fill_ids() was
empty and the step did nothing at all; it only counts now that the
control can redden it.
Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips,
identical failure set to the pre-sgl-project#783 baseline, zero new failures.
ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…ark transport batches W37-G (boot_w37g_0825_1326.log, pin 0be4ff4) proved the #861j doors open on metal -- SEAM TRANSPORT ADMITTED x3, TP decode formed at bs 5-6 with accept ~2.4-2.7, per-rid tokens strictly rising -- and exposed the next gap: all 27 TP transport batches recomputed cold (#cached-token: 0, the sgl-project#783 dark-radix shape defeating the premise at runtime) while the conformance count stayed 0, because layout_conformance.work_layout_verdict -- built after W37-D's 258 identical batches -- had ZERO callers. Desk-written- never-executed, in the instrument meant to falsify exactly this shape. * metrics_reporter: _note_work_layout wired into report_prefill_stats before the logging-rank gate (counters are rank-local); phase from the same authority as the phase label, strict from purity_of, fully try/except'd. Decode side deliberately not wired -- named in the docstring, not omitted silently. * scheduler: the admission that rides the exemption stamps batch.is_seam_transport from the same transport_only local that spends the seam stamps, so the claim cannot drift from the admission. * layout_conformance: the restore excuse is now cached_tokens > 0 (was AND new_tokens <= 0, which would have flagged every legitimate read-through -- a real restore still computes >=1 token per request). The defect signature is cached == 0, W37-D and W37-G both. Red-proven in a temp worktree at HEAD: the cold-transport batch through the REAL report_prefill_stats counts 0 ("the detector for exactly this shape has no caller") and the build-site pin finds no transport mark; green on this tree. Can-fails: genuine partial restore stays silent, ordinary prefill-in-tp still flags, prefill-in-pp never flags. Tests: test_tp_decode_formation_861j.py 13/13 (dark-radix arm added), scoped impact 130/130 (conformance/economy/band/forward-pass/exemption/ servability), ruff delta 0, codespell delta 0. Full serial managers battery: 8 failed / 4320 passed / 2 skipped / 356 subtests in 722.41s (/tmp/861k_gate2.log), cards VISIBLE -- the Tier-2 managers gate is not hermetic and CVD="" flips its verdict in both directions. Failure set identical BY NAME to the standing 8: staging_reserve_631 x4, rank_prefill_log::TestCollectiveClock x1, hisparse_unit x2, draft_cuda_graph_removal x1. Unexpected names: none. WURZEL. Klasse: desk-written-never-executed -- an instrument built, unit- tested against hand-made inputs, and never wired to the path it judges, so it reports green by never running; test-only coverage is what HIDES it. Geschwister, swept tree-wide by AST over python/sglang/srt (143 instrument- shaped defs, references counted outside the def line), sweep validated in both directions (it lists work_layout_verdict with metrics_reporter.py masked and goes silent with it wired): 10 members with zero production callers. Three are load-bearing and all in managers -- * progress_liveness.py: the WHOLE sgl-project#699 module has zero production importers (only its test file), including thrash_verdict (#861e), which has exactly one reference repo-wide, its own def line, and no test at all. That is the detector for "decode advances, completions do not" -- the W37-D/W37-G shape and the sgl-project#857 acceptance discriminator. * corridor_guard.phase_corridor_verdict (sgl-project#784): 7 test call sites, zero production callers; the constants are imported widely, the verdict is not. Zukunfts-Check: OWED and specified, not built here -- the sweep becomes a registered test that fails when an instrument-shaped def has no production caller, with an allowlist carrying an owner and a reason per dark entry. Filed as #861l so this commit stays the wiring change. Full list and file:lines in /spinning/gpu-arb/NOTE_861fg.md.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…althy fence W37-G (boot_w37g_0825_1326.log): 39 cutovers in 5.5 min, ZERO completions, and `#cached-token: 0` on all 209 prefill batch lines. The inherited framing for sgl-project#783 was "the radix tree holds rows, match returns 0" -- a dark READ path. That is falsified in both halves. Every fence of the entire boot: 33 x eligible=0 staged=0 already_staged=0 acked=0 outstanding=0 elapsed=0.000s/2.000s 3 x eligible=3 staged=0 already_staged=3 acked=0 outstanding=2 elapsed=0.000s/2.000s 3 x eligible=1 staged=0 already_staged=1 acked=0 outstanding=1 elapsed=0.000s/2.000s `acked=0` in every one. Nothing was ever confirmed into the canonical store. `eligible = len(_hashed_nodes(tree_cache))`, so eligible=0 means the device tree carried no hashed node at fence time -- and the tree drop independently returned 0 rows on 36 of 39 cutovers, including cutovers that retracted 6 and 7 requests. Two independent measures agree the tree was EMPTY. A read path cannot be the root when there is nothing to read. WHY NOTHING SAW IT. The seam's guard is headed "sgl-project#703 FENCE COVERAGE, ASSERTED RATHER THAN ASSUMED" and warns only when `_writeback_fence_ms(...) is None`. That helper returns `elapsed_s`, and an empty fence returns 0.0, not None -- silent on all 33. `FlipWritebackReport.complete` is `outstanding == 0`, so those same 33 also logged at INFO as COMPLETE. Two instruments, both correct, both blind to the one state that matters: a cheap fence and a healthy fence are indistinguishable BY COST, because the healthy case is also cheap. `test_writeback_fence_ms_856.py` names this exact outcome in its own docstring -- "a defaulted zero would report such a flip as fully fenced while nothing was persisted" -- and then pins the wrong variable. It guards a DEFAULTED zero; W37-G produced a GENUINE zero from an empty tree and reached that outcome by a path the test never considered. THIS COMMIT IS THE DETECTOR ONLY. `FlipWritebackReport.persisted_nothing` asks the question the read-through actually depends on -- is anything retrievable after the tree is dropped -- from counts the report already carried and nothing read. Deliberately narrow: true only when NEITHER route survives (no storage ack AND no pre-existing host copy), so the 6 lapsed fences that already draw the "deadline reached with backups still in flight" warning are not double-reported. Wired into the seam so it is not another desk-written instrument. Tests: red confirmed before the fix (AttributeError on `persisted_nothing`). The wiring is proven BY EXECUTION, not by `inspect.getsource` as all four existing tests of `_release_residents_for_cutover` do -- source inspection cannot tell a guard that runs from one that is unreachable. Both directions pinned (empty fence warns / healthy fence silent / lapsed fence silent), and an `if False` mutant on the guard kills the wiring test. 79 passed across the seam neighbourhood plus the #861l reachability gate. black/isort/ruff (--select=F401,F821,UP037, the repo gate) clean on the changed files; codespell and UP037 hits in phase_flip_runtime.py are pre-existing on HEAD and untouched. Also included: isort reordered two pre-existing imports in phase_flip_runtime.py. Gate-mandated, unrelated to sgl-project#783, named here rather than left silent. NOT FIXED HERE. The root is a closed loop: retention is finish-only since ff5651a, the flip retracts every resident before it finishes, `retract_all` -> `release_req(offload_kv=False)` frees that KV without tree insert or writeback, so the tree stays empty, the fence has nothing eligible, the store is never written, re-admission finds neither tier and recomputes in full, which does not finish before the next cutover. `build_cutover_release` justifies `offload_kv=False` as "the FENCE has already persisted the prefixes ... so a second device->host copy would pay twice" -- a premise W37-G falsifies, and the same false premise is load-bearing in two places.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…not by handover THE DEFECT. The sgl-project#856 cutover retracts every resident and keeps nothing, so the state a phase transition must preserve simply does not survive it: * PHASE-FLIP-GDN moved 0 slot(s) on ALL 228 flips of the W37 series. The GDN mover reads `flip_mamba_slots` = resident mamba slots UNION tree checkpoints, and BOTH are destroyed at phase_flip_runtime.py:10829, one seam step before the mover runs at :11139. * The sgl-project#703 fence walked an empty tree: eligible=0 on 78 of 85 cutovers, acked=0 on every fence of the boot. * #cached-token: 0 on 403 of 403 prefill batch lines (W37-H arm A, the control on this tree). WHY NOT AN INSERT. Persisting at that instant by inserting into the tree was tried and measured: W37-H arm B died 33 s after ready, all three ranks, `pool memory leak detected! total=468981, available=108565, evictable=1, withheld=360437` -- 108565+1+360437 = 469003, i.e. 22 rows claimed twice. `readmit_seam_residents` brings the population straight back ("W31: RE-ADMIT THEM") and it resumes on rows the tree had taken. Ownership transfer is structurally wrong for a population that returns; `cache_finished_req` is the finish CODE PATH, not a finished request. THE FIX, IN TWO HALVES THAT ONLY WORK TOGETHER. Half 1 -- copy at the cutover retraction. `release_req(..., copy_state=False)` calls `offload_kv_cache` BEFORE `release_kv_cache`, while the rows still hold live bytes. Row ownership is untouched, so the arm B double claim cannot recur (`is_insert` stays False). Threaded through `retract_all` and set True at EXACTLY ONE line, `build_cutover_release._retract`. That single-site property is load-bearing: `retract_all`'s only other caller is the decode-pressure path, whose rate is load-dependent, and the host budget (0.585 GiB/cutover, ~57 MiB/s aggregate) is priced at the flip cadence. "Ungated by design" means no flag -- the condition is structural. Half 2 -- restore where the rows come back. NOT in `readmit_seam_residents`, which only re-queues: by then `reset_for_retract()` has cleared `req_pool_idx`. `restore_seam_state` runs in `prepare_for_extend`'s per-req loop, after `alloc_for_extend` (rows exist) and before `is_retracted` is cleared -- the same shape as the proven disagg path, `_pre_alloc` then `load_kv_cache`. The guard is the PRESENCE OF THE COPY, not `is_retracted`: the flag is also true for pressure retractions that never copied, so keying on it would request a restore that does not exist. The copy is consumed, so a second pass cannot re-apply stale bytes. Enabling change: the mamba half of the copy did not exist on this rig's pool. `Req.offload_kv_cache` promised "copies over both the kv cache and mamba state" and passed `mamba_indices`, which `UnifiedSWAKVPool` accepts and never reads -- correctly, since a KV pool with no mamba pool has nothing to copy. Only `HybridLinearKVPool` honours it, because it owns one. So the capability is now DECLARED (`supports_mamba_cpu_copy`, default False on `KVCache`, True on `HybridLinearKVPool`, forwarded by the allocator to its `_kvcache`) and `Req` dispatches on the declaration, covering exactly the pools that have no owner to delegate to. Exactly one mover per payload, enforced rather than promised. The form reuses the house idiom (`BasePrefixCache.supports_mamba` / `supports_swa`); the allocator tree had none of its own. The declaration must take the SAME hop as the payload: allocators implement `get_cpu_copy` as `self._kvcache.get_cpu_copy(...)`, so a declaration that did not forward would let `Req` read the base False while the pool underneath copies mamba -- copying it twice. The getattr default is kept there because `UnifiedKVPool` and `DeepSeekV4UnifiedKVPool` declare no base class and genuinely cannot answer; it is removed at the `Req` callsite, where every allocator inherits the method and a missing attribute is a real defect. TESTS test_seam_state_copy_783 4 copy fires at the cutover and nowhere else; pressure retraction unchanged test_seam_state_restore_783 6 restore round-trips, is consumed, and is wired after alloc / before the clear test_seam_ownership_ledger_783 8 drives the PRODUCTION invariant (SchedulerInvariantChecker._check_pool_ invariant) over a claim ledger; gates the COMBINATION, and its can-fail shows half 1 alone balanced while the pair leaks -- the arm B shape, at the desk test_unified_mamba_cpu_copy_783 8 the declaration is pinned, including a pool that declares True and lies Each file carries at least one assertion that passes without the change, so a broken fixture cannot masquerade as a finding (it caught three of mine). Serial managers battery, cards visible (this gate is not hermetic): 4358 collected = 4348 passed + 8 failed + 2 skipped, 356 subtests, 721.20s Failure set identical BY NAME to the standing 8: staging_reserve_631 x4, rank_prefill_log x1, hisparse_unit x2, draft_cuda_graph_removal x1 Zero unexpected names, zero standing failures missing. Collected == ran, so the run is complete -- including test_pp_retracted_pass_void_797, which wedged in an earlier run while arm A was still winding down. black/isort applied, ruff gate (F401,F821,UP037) clean, codespell clean. NOT PROVEN HERE: that the prefix actually survives a flip on metal. That is the boot, and its acceptance lines are #cached-token > 0 after a flip against 403/403 zeros, and PHASE-FLIP-GDN moved > 0 against 228 zeros.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…laration mechanics W38-A (pin eeed2bc) reached /health and died 14 s into the load, all three ranks: prepare_for_extend -> restore_seam_state -> load_kv_cache -> HybridLinearKVPool.load_cpu_copy -> memory_pool.py:3295 IndexError: list index out of range `load_cpu_copy` walks `range(0, len(indices), chunk_size)` and indexes the saved chunk list, so it requires the restore's extent to be no larger than the copy's. It was larger. ROOT: the restore borrowed the disaggregation shape (`_pre_alloc(req)` then `load_kv_cache`), which works there because the request is PRE-ALLOCATED IN FULL. The normal prefill path admits CHUNKED: `prepare_for_extend` allocates rows for the current chunk only, so `req_to_token[req_pool_idx, :seqlen-1]` at restore time does not describe the extent that existed at copy time. I checked WHERE the rows exist and never WHAT EXTENT they cover. NOT PATCHED WITH A min(). The two extents describe different things; clamping one into the other would write a prefix's KV into the wrong rows -- a wrong ANSWER instead of a crash. The IndexError is the better failure and the extent question is answered before this is rebuilt. BOTH HALVES GO, not just half 2. Half 1 alone would copy on every cutover retraction with nobody consuming it: pure host cost (0.585 GiB per cutover, ~57 MiB/s aggregate) for no benefit. "Both halves or none" cuts against a partial revert here. REVERTED release_req / retract_all `copy_state` parameter and the copy call build_cutover_release._retract `copy_state=True` prepare_for_extend the `restore_seam_state` hook schedule_batch `restore_seam_state` itself test_seam_state_copy_783 / test_seam_state_restore_783 -- they pinned code that no longer exists KEPT, because it is sound and was expensive to establish `supports_mamba_cpu_copy` on KVCache (False) / HybridLinearKVPool (True) / the allocator forwarding hop, and Req's dispatch on it. W38-A's traceback proves the rig runs HybridLinearKVPool -- the pool with the COMPLETE implementation -- so the dispatch correctly let the pool own the mamba copy and skipped Req's. It behaved as designed on a pool nobody had predicted would be the one in play. Note this also retires the earlier "the rig's pool cannot copy mamba" framing: it always could. What D adds is the explicit single-owner declaration, not a closed capability gap. test_seam_ownership_ledger_783, test_unified_mamba_cpu_copy_783. Tests after revert: 26 passed (ledger + mamba-cpu-copy + fence), 87 passed across the seam regression set (seam_order_856, seam_readmission_w31, tree_drop_returns_rows_856, uniform_retract_count_583, pp_retracted_pass_void_797). ruff gate clean. Also carried, and named rather than left silent: black reformatted two pre-existing `assert` statements in schedule_batch.py during this work. Gate output, unrelated to sgl-project#783.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…fuses instead of indexing Second attempt at carrying the cutover's state across the flip. The first (eeed2bc, reverted in dc5d557) died 14 s into W38-A's load with `IndexError: list index out of range` at memory_pool.py:3295, three ranks. ROOT, and it is one sentence: A LOGICAL LENGTH WAS USED AS A PHYSICAL EXTENT. `Req.seqlen` is `len(origin_input_ids) + len(output_ids)` -- the request's final logical length, which says nothing about how much of `req_to_token` has been written. During chunked prefill the row is filled only to `kv_allocated_len` (== `extend_range.end`), strictly less. Both `offload_kv_cache` and `load_kv_cache` index by `seqlen - 1`, so both are sound only once prefill is COMPLETE. A restore whose extent had grown walked off the end of the saved chunk list. The vulnerability was LATENT, not pre-existing-and-tripped-over: both methods had exactly one caller each, and both were on fully-preallocated decode-disaggregation paths. My seam wiring was the first caller to make the chunked path reachable, so this commit owns the reachability it created. THE TREE HAD ALREADY DECIDED THIS QUESTION -- AGAINST what the first attempt built. `kv_session_offload` refuses chunked admission outright rather than restoring into it: ":445 return False # would be CHUNKED -> needs PS3", and the assert at :4496 naming "PS3 (host-prefix extend read)". PS3 has four mentions in this tree, all comments: unimplemented. This commit matches that answer instead of quietly building the unimplemented thing. THE CONTRACT Req.kv_cache_cpu_extent recorded at capture; a copy states what it covers seam_copy_state() DECLINES a mid-chunk request -- no full extent restore_seam_state() REFUSES on extent drift and DROPS the stale copy _seam_prefill_is_complete `kv_allocated_len >= seqlen - 1`, i.e. the physical extent must have caught up with the logical length REFUSAL, NEVER A CLAMP. The two extents describe different things; a min() would write a prefix's KV into the wrong rows -- a wrong ANSWER instead of a crash. A refusal costs a recompute, which is merely slow. It lives at the CALLER because that is the only layer holding both numbers: every pool-level `load_cpu_copy` merely forwards indices, and its lone assert (:3298) is per-chunk, on the wrong axis entirely. There is no backstop below. SCOPE, deliberately bounded: decode-phase residents -- the population the cutover actually retracts, and the one that loses real session state -- are covered. A mid-chunk request loses no session state by recomputing; it repeats unfinished work. PS3 is named, not half-built. OBSERVABILITY IS PART OF THE FIX, not an addendum. The first attempt's restore logged nothing at all, which is the desk-written shape this branch has spent its time closing. [sgl-project#783 seam-state] SEAM RESTORE rid=.. extent=N ... (copied/declined/ restored/refused) [sgl-project#783 seam-state] SEAM RESTORE REFUSED rid=.. covers X, needs Y [sgl-project#783 seam-state] SEAM COPY DECLINED rid=.. prefill still chunked `declined` is also the MEASUREMENT of how often a mid-chunk request meets the seam -- so the PS3 question gets a number rather than a guess. WIRING, one site each: `copy_state=True` only at `build_cutover_release._retract` (the decode-pressure path never sets it, which is what keeps the host budget priced at the flip cadence); the restore in `prepare_for_extend` after `alloc_for_extend` and before `is_retracted` is cleared. TESTS test_seam_extent_contract_783, 11 passed, including TWO controls that pass without the change -- one of them asserts the stub pool can REPRODUCE the W38-A IndexError, without which the reds would say nothing about the real pool. Regression: 103 passed (extent + ledger + mamba-cpu-copy + fence + seam_order_856 + seam_readmission_w31 + uniform_retract_count_583 + pp_retracted_pass_void_797). Serial managers battery, cards visible: 4359 collected = 4349 passed + 8 failed + 2 skipped, 356 subtests, 723.32s Failure set identical BY NAME to the standing 8: staging_reserve_631 x4, hisparse_unit x2, rank_prefill_log x1, draft_cuda_graph_removal x1. Zero unexpected, zero missing. Collected == accounted, so the run is complete. Passed delta vs the last battery is exactly -10 (two removed plumbing test files) +11 (this one) = +1. black/isort applied; ruff gate clean on the changed files. NOT PROVEN HERE: that the prefix survives a flip on metal.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 25, 2026
…both axes
THE SILENT HALF IS THE DEFECT. `-1` is simultaneously the classic stale-row
sentinel and valid torch indexing, so a stale id does not fault: it writes the
LAST row of the pool. Wrong KV -- or, on the mamba axis, wrong recurrent state
-- under a prefix the radix tree reports as RESTORED. No crash, no log line, on
a shipped path, refused nowhere until this commit. Two new tests pin it as a
fact about the code as it stood:
test_a_negative_row_does_NOT_fault_and_hits_the_last_row (MHA, dim 0)
test_a_negative_slot_does_NOT_fault_and_hits_the_last_slot (Mamba, dim 1)
Both PASS without the fix. That is the point of them.
The illegal memory access was the LOUD half, and it is the only reason we
looked. W40b, 2026-08-25 19:52:52, three PP ranks, instance down five seconds
later:
prepare_for_extend -> restore_seam_state (schedule_batch.py:2748, :2091)
-> Req.load_kv_cache (:1778) -> load_cpu_copy (memory_pool.py:3303)
-> current_platform.synchronize()
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
ROOT. sgl-project#783's extent contract (25e7849) compares ONE SCALAR -- the row COUNT a
copy covers against the count the request now needs -- and then forwards
`req_to_token[req_pool_idx, :seqlen-1]` to the pools as ids that nothing
validates. EQUAL LENGTH IS NOT EQUAL VALIDITY. The pools index their buffers
directly and the only assert on the path (MHA :3298) compares the saved host
chunk's SHAPE against the chunk LENGTH -- a different axis, which that commit
already names: "per-chunk, on the wrong axis entirely. There is no backstop
below." This is that backstop. It sits BESIDE :3298, not in place of it: one
checks the saved payload lines up with the index count, the other checks the
index VALUES exist. Neither implies the other.
A MODULE FUNCTION, NOT A METHOD, AND THE BOUND IS AN ARGUMENT.
`check_cpu_copy_rows(indices, rows, direction, axis)`. The pools that need it do
not share a base -- `MHATokenToKVPool` and `MLATokenToKVPool` are `KVCache`
subclasses and `MambaPool` is not -- so a guard installed on one ancestor would
be silently inert on the other. The bound and axis are passed in because they
genuinely differ: MHA is bounded by KV buffer rows on dim 0, `MambaPool` by its
slot count on dim 1 (`conv[:, indices]`, layout [num_layers, num_slots, ...]).
An earlier draft of this guard read `k_buffer[0].shape[0]` off `self`; wiring
Mamba to it would have looked done and done nothing. A guard that cannot fire is
the failure mode this whole sweep exists to remove.
REFUSAL, NEVER A CLAMP -- the parent commit's rule for the parent commit's
reason: the ids and the pool describe different things, so dropping or wrapping
the offending id writes state into rows belonging to someone else. A refusal
costs a recompute, which is merely slow. Checked BEFORE the first store, with a
test asserting the pool is byte-identical after a refusal: a guard that fires
halfway through is a louder corruption, not a fix.
THE SWEEP -- 14 `get/load_cpu_copy` definitions in memory_pool.py alone, not
one. The first pass at this assumed a single site.
WIRED MHATokenToKVPool :3262/:3284 the W40b site, and what this rig's
HybridLinearKVPool delegates its KV half to (:4390, :4401)
WIRED MambaPool :1180/:1192 dim 1, bound mamba_cache.conv[0].shape[1];
reached here via HybridLinearKVPool -> mamba_pool.load_cpu_copy(
_mamba_translate(...)), and that translation is a second staleness
source
n/a HybridLinearKVPool :4389/:4399 pure delegator, covered by both above
n/a KVCache ABC :2319/:2322 NotImplementedError stubs
FILED PageMajorMHATokenToKVPool :4006/:4012, MLATokenToKVPool :4674/:4689,
DSATokenToKVPool :5023/:5048, dsa_cache_layer_split.py, and the
forwarders in allocator/{base,token,paged,swa}.py,
swa_memory_pool.py:285-330, unified_memory_pool.py:1223-1237
The 12 unwired sites are not on this rig's path. They are enumerated with their
file:line in docs/dev/NOTE_783b_cpu_copy_row_bounds.md rather than wired on
faith.
PRIOR EVIDENCE THAT THIS BELONGS AT THE POOL BOUNDARY, not per-caller: two
forwarders already filter their indices -- swa_memory_pool.py:295
(`swa_indices[swa_mask]`) and unified_memory_pool.py:1227 (`swa_phys[valid]`) --
the SWA half ONLY, with the unfiltered `full_phys` sitting directly beside them.
The need was seen once and not generalised. Putting the check where every caller
must pass through it is the answer to that, rather than trusting each caller to
remember.
OBSERVABILITY, because the emitter could not witness its own event.
`restore_seam_state`'s log line sat AFTER `req.load_kv_cache`, so a restore that
does not return is exactly what it cannot report: W40a logged 18 seam lines,
W40b crashed INSIDE the call and logged ZERO, which is why the traceback names
`synchronize()` and not the restore. An ATTEMPT line now goes before, rate
limited on the same cadence as the success line.
TESTS test_seam_row_bounds_783b, 17 passed. Hermetic, mocks only: the REAL pool
methods bound to CPU buffers, because a test that reimplemented the copy would
encode the defect's own assumption -- the shape 25e7849 corrected in three of
its own mocks. A/B against this tip (fc24204): 9 failed / 8 passed WITHOUT
the change, 17 passed WITH it. That A/B is the mutation proof; the 8 that pass
either way are controls, including the two silent-half pins above and the
over-refusal guards (valid rows still load, an empty index set is a no-op).
Hermetic suites, run SEPARATELY per sgl-project#749 against fc24204, count gate
(^FAILED + ^SUBFAILED + ^ERROR == summary) PASSED on all three before any
comparison:
mem_cache 2 failed / 1774 passed / 1658 skipped / 361 subtests
gate 2 == 2. Baseline on this tip was 1757 passed; +17 is exactly
this commit's tests. The 2 are the standing
test_acceptance_emitters_758 pair, "RuntimeError: No CUDA GPUs
are available" -- measured pre-existing earlier tonight by
reverting to the tip and re-running, not assumed.
managers 15 failed / 4343 passed / 356 subtests gate 15 == 15
IDENTICAL to the post-#858b baseline, by count and by name:
test_arena_high_water_631 (7), test_phase_flip_rotation_wiring_809
(4), test_restore_never_rebuild_677 (4), all hermetic-environment
CUDA failures.
distributed 95 failed / 3017 passed / 1170 subtests gate 92+3+0 == 95
IDENTICAL to baseline. The 3 SUBFAILED are the
dcp_size=2/3/4 subtests; summary extracted BY PATTERN because
these arms print UCX teardown warnings after pytest's summary.
Zero new failures on any arm. ruff: memory_pool.py 0 before and after,
schedule_batch.py 41 before and 41 after (untouched pre-existing), the new test
6 x E402 from the house `register_cpu_ci()`-between-imports convention, the same
class the existing test_seam_extent_contract_783.py carries (4). codespell
clean. black/isort deliberately NOT run over these files: the venv's black is
OLDER than the one this tree was formatted with and isort relocates a
deliberately-late `# noqa: E402` import in memory_pool_host.py -- every added
line was checked against the 88-column limit by hand instead.
STILL OPEN, AND NOT CLOSED BY GUESSING: why did W40a's nine restores succeed
while W40b's first one did not? Unknown. No trigger has been written into this
commit to make the story tidy. The guard lands anyway because it is the
INSTRUMENT that will answer it -- it converts an asynchronous IMA attributed to
an unrelated `synchronize()` into a deterministic host-side refusal naming the
id, the bound and the axis. CUDA_LAUNCH_BLOCKING=1 belongs in the sgl-project#857
acceptance arm so an IMA is attributed to its own kernel.
NOT PROVEN HERE: that a seam restore survives a flip on metal. This commit makes
the failure loud and specific instead of asynchronous or silent; making it WORK
is the acceptance boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…o the guard can arm sgl-project#718/sgl-project#847 built every part of the rebind except one: the 'tp' staging pin was assembled with a SINGLE entry, PoolName.KV. On a hybrid model the live tier carries KV *and* MAMBA, so `check_pool_coverage` computed `missing={MAMBA}` and refused -- correctly, on every cutover. A refused rebind leaves the sgl-project#718 device tier DISARMED, `load()` returns None, every read-through misses, and every prefix a cutover retracted is recomputed in full. Measured on the W40 sgl-project#857 acceptance boot (boot_w40_857strict_0825_2342.log): 60 `sgl-project#719 HiCache rebind refused`, ZERO arms, and `#cached-token: 0` on all 243 prefill batch lines -- one bucket, no exceptions. THE PRECONDITION WAS ALREADY WRITTEN DOWN, in the guard's own docstring: "A phase host tier has to be built with the FULL POOL SET before this rebind can arm; until then the sgl-project#718 disarm is the correct state and a read-through miss is the correct cost." This is that precondition, met. Not a new finding -- sgl-project#856 answers whether the retraction is necessary (it is: carry moves Req scheduling metadata, never KV bytes, and sgl-project#856 deliberately retired the movers), and sgl-project#718/sgl-project#847 already named this remedy. REFUSAL CONVERSION, NOT GUARD DELETION. `check_pool_coverage` is untouched. It must stop firing because its precondition is MET, never because it was removed, and the test asserts BOTH directions -- a full pool set arms, a narrowed one still refuses. Neither assertion alone can tell a fix from a disarm. DERIVED FROM THE BOUND TIER, not from the model config: the set that must be covered is whatever the READER names, which is the same quantity the guard compares. Reading the config would be a second opinion about one fact. MIRRORS `build_hybrid_mamba_stack` rather than re-deriving it -- same primitives, same layer mappings, same transfer_layer_num rule. The controller is deliberately NOT reused: this pin needs a host VIEW, and a second HybridCacheController would be a second writer against one device pool. Both entries are rebuilt in the hybrid case. The KV-only pin used an identity map over range(layers), which is right while KV is the only entry and wrong the moment a second pool shares the transfer index space -- the two maps collide at index 0. SIZING IS PER-SLOT, NOT PER-GB, and it is the one place the mamba half must not copy the KV half. MambaPoolHost reads host_size in GB only when > 0, else `device_pool.size * ratio`. The KV pin's GB figure is derived from a token count; mamba is allocated per request slot. Ratio 1.0 with host_size 0 mirrors the device pool, which is what a phase-matched staging pin means. The mamba half is a NAMED HOST-LEDGER POST, priced from what was allocated rather than from the intention. An unpriced pinned pool is what the ledger exists to prevent. A pool set this builder cannot mirror (SWA, indexer, DeepSeek) is logged as an ERROR at the cause and still refused by the guard -- named, not swallowed. THE CHECK (sgl-project#871, third scope item): `advance_fence_blind_streak` + FENCE_BLIND_STREAK=4, aggregating the EXISTING `persisted_nothing` instrument. One empty fence is legitimate; every empty fence means the canonical store can never populate, and that shows up only as latency. Gated on `released`, mirroring sgl-project#719's busy gate at the stale-gate streak: a fence over an empty tree is correct to persist nothing, so counting quiet cutovers would build a crying-wolf alarm out of the instrument written to replace one. NO SECOND COUNTER -- a parallel "recomputed prefix tokens" counter would measure what this and #cached-token already measure between them. Extracted as a pure function so it is falsifiable without booting. A guard whose logic can only be exercised by booting is a guard that ships unexercised, which is the failure mode this ticket is about. CLASS (unchanged, carried from the sweep): a store whose only writer is a lifecycle event another mechanism systematically preempts. Retention is finish-only; the cutover retracts before finish; the store can never populate, so every recovery path reading it can never fire. TEST RESULTS test_phase_tier_full_pool_set_871.py (new): 15 passed. FALSIFIED IN THREE DIRECTIONS BEFORE THE GREEN WAS CLAIMED, count gate held on each: guard removed (disarm) -> 3 failed, and they are exactly the refusal tests. 3 extracted == 3 in summary. guard fires unconditionally -> 6 failed, exactly the arming tests. 6 == 6. streak loses its `released` gate -> 1 failed, the idle test. 1 == 1. restored -> 15 passed. Targeted regression set (7 files: sgl-project#847 writer, sgl-project#783 fence x3, sgl-project#856 empty wave, counters, and the new file): 97 passed, 0 failed, 2 subtests passed. Count gate 0 extracted. Hermetic, CUDA_VISIBLE_DEVICES verified EMPTY at the PROCESS (/proc/<pid>/environ), not at the command. AN EXISTING TEST CAUGHT A REAL BUG OF MINE and the first run of that set was 3 failed: the new HOST-LEDGER line read `tp_host.entry_map` directly, but that writer is driven in tests by stand-ins where HostPoolGroup itself is patched. Fixed with the getattr discipline this module states at its other probes -- an instrument may never be the thing that breaks a boot. Lint: ruff 0 before and after on phase_flip_runtime.py and the new test. phase_flip_boot.py reads 1 before AND after -- a pre-existing F401 on `pack_into_arena`, present in HEAD, not mine. codespell clean. Flag-off path verified byte-identical: without --phase-flip-rebind-hicache the writer still returns {} and allocates nothing. NOT CLAIMED, AND DELIBERATELY NOT GUESSED: no metal run. Whether the kv+mamba pin can actually be ALLOCATED on this box, what its real HOST-LEDGER post comes to against the 16G floor, and whether #cached-token becomes non-zero are decidable only on hardware. They belong to a boot window. The sgl-project#857 acceptance instance was left running and untouched throughout -- it is the standing proof and the operator declared it taboo.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…ct#868's 15, and one keyword wide sgl-project#864. The question was whether the 16 standing failures are the same hermeticity form as the 15 that sgl-project#868 resolved. They are its MIRROR IMAGE, and the mirror is the direction that hurts: hermetic cards the desk gate shows sgl-project#868's 15 FAIL pass a red nobody can act on sgl-project#864's 16 SKIP FAIL a GREEN over a standing red Same class -- the invocation decides the verdict -- but sgl-project#868's announce themselves on every desk run, while these announce themselves nowhere anybody looks: the machine that can see them has cards, and the gate people read does not. CAUSE, proved without a card. hiradix_cache.py:266-278 calls all_reduce(tensor, op=op, group=group, async_op=True) unconditionally. The double at test_unified_radix_cache_unittest.py:2382 is fake(tensor, op=None, group=None). All 16 die with a TypeError inside mock's dispatch BEFORE their first assertion, so they are not evidence about SWA prefetch adoption in either direction. Checked the other half of the call too rather than assuming: the double returns None, and hicache_collective.py:190-196 documents None as a completed no-op -- so the drift is exactly one keyword wide and a one-line fix is the whole fix. TEST-ONLY. Zero live defects. sgl-project#783 gets nothing from them. Corrects the prior art it rests on: NOTE_861fg's "16 passed" hermetic is a numeric COINCIDENCE, not the same 16. Measured: -k tp_swa_prefetch under CVD="" gives 38 skipped / 0 passed / 0 failed (8+8 for want of a card, 11+11 non-SWA parametrisations; tally 38 == summary 38). They are inside the 1506 skips, not the 16 passes. sgl-project#862. mem_cache hermetic: 2 failed / 1774 passed / 1658 skipped, both tally axes OK (names 2 == summary 2; extracted 1658 == summary 1658). Both failures DEVICE-caused; REAL defects 0. The 1658 split by reason: HARDWARE 996 | DEPENDENCY 12 | CONFIG-not-darkness 645 | BY ACCIDENT 0 UNCLASSIFIED 5 So the darkness is BY DESIGN -- BY ACCIDENT is zero and the conftest mechanism explains itself in its own docstring. The ticket is still right for a finer reason: 1008 of those would run on a card and 645 would never run anywhere, and the summary line renders those identical facts. My own classifier was wrong first and the data said so: two buckets put 636 of 1658 in neither. Folding them into "dark" would have inflated the figure by 62 %. All 36 survivors were read individually and encoded; 5 remain unclassified and are reported as such rather than tuned away. Two more corrections to prior art: mem_cache DOES run directory-wide on CPU here (3396 collected, no abort), so the recorded collection-time abort does not reproduce on this tree -- reported, not bisected. And the escape behind it is real but at RUNTIME and two tests wide: rotation_executor.py:344 allocates with pin_memory=True, a device door that never calls get_device(), which is the one door the conftest guard patches. THE CHECK, in the CPU lane on purpose. New test_double_signature_drift_864.py (2 tests, 0.11 s, register_cpu_ci) parses the double and the production call sites and asserts the first can bind the second -- no device, no ranks, no import of a module that needs them. It also asserts production still passes async_op, so the guard reports a change instead of silently testing a call that no longer exists. Can-fail proven: re-introducing the drift turns it red with the exact diagnostic. Regression: mem_cache after the change is 2 failed / 1776 passed / 1658 skipped -- same failures, same skips, +2 from the new guard. NOT done and recorded: the 16 were never run WITH cards, so the fix is proved to remove the drift and not proved to make them pass. Every number here is a CVD="" number, verified on the process; nvidia-smi compute-apps showed only the other runner's serving ranks throughout. (cherry picked from commit a428907)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
`--verify` named exactly one new module,
test/registered/unit/mem_cache/test_pool_invariant_live_double_912b.py, and
no sha256 among the 184 already in the table moved. `test_seam_extent_
contract_783.py` DID change bytes, but it sits in the managers table's
long-standing unclassified set, so no fast-lane proof went stale with it --
PARALLEL stayed at 270 there, checked rather than assumed.
MEASURED on this tip, hermetic (CUDA_VISIBLE_DEVICES=""):
* fresh serial reference, 185 modules, one process, one order:
2 failed / 2761 passed / 927 skipped, 164.8s. 2761 = 2745 + exactly the
16 the new module contributes; the 2 failures are the same two
test_acceptance_emitters_758.py::RefillTiming names, compared as a LIST.
* the 184 solo logs carried forward, plus one new solo probe.
Demotions: ZERO, counted by diffing the table's own rows before and after.
before PARALLEL 171 RANKS 13 UNCLASSIFIED 1
after PARALLEL 172 RANKS 13 SERIAL 0 EXCLUDED 0 -> VERIFY OK
gate_partition.tsv (managers) again NOT rebuilt, for the reason recorded in
Train-2c: its 35-module gap has no surviving solo logs, so a rebuild would
demote 270 proven rows to make two new ones look better. sgl-project#920's module and
sgl-project#917's before it therefore run in the serial lane by the default-for-the-
unknown rule, both named in the runner's report, and the lane's arithmetic
shows it: 617 -> 630 passed, which is exactly the new module's 13. The
adapted sgl-project#783 contract contributes 0 to that delta -- 11 passed on this tip
and 11 on 3502a9a, measured on both, so pinning the DECISION instead of
the literal changed what the tests assert without changing how many there
are.
gate_partition_scheduler.tsv untouched: seam_census_double.py is a harness,
not a test_*.py, so it carries no table row.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…omote, and the silently-wrong pair ONE build closing the publish -> cached_tokens chain, plus the health-probe confound that made every measurement of it ambiguous. THE TWO BINDING TERMS (both desk-proven at the tree, both root, not symptom): 1. The promote `pending -> effective` existed at exactly ONE site, inside `_event_loop_pp_body`. The warm prefill on this rig runs boot-measured in the TP phase (the instance flags it itself: `sgl-project#838 work_in_wrong_layout class=prefill phase=tp`, 9 genuine violations), where that body never runs. PP0 published into `pending` and nobody ever promoted it: `sgl-project#968 LOAD-BACK PROMOTED` = 0/0/0 over three whole boots, and that is a REAL zero -- its cadence counts promotions, so the first would have printed. Fix: `promote_pp_load_back_pending()`, idempotent, called from both loop bodies AND from `_get_new_batch_prefill_raw` (the phase-neutral consumer), above every early return. 2. `init_pp_loop_state` nulled `_pp_load_back_pending` UNCONDITIONALLY while the line directly below guarded `_pp_load_back_effective` with `hasattr`. The unconditional null was deliberate ("only pending is a per-pass quantity") and is only fatal BECAUSE the promote was phase-bound. Fix 1 is the root; this is the guard that carries the fact across the cutover. REGRESSION IN e0bc960, FOUND AND FIXED: that commit moved the offer write to the load-back site but kept only the `_hit > 0` half, deleting the only deleter -- the old `= _hit if _hit > 0 else None`. The comment it removed states the lost property verbatim ("goes to None on its own"). The offer then outlived its own truth: stale-HIGH re-publish -> a peer told an extent its host tier no longer covers -> `EXTENT UNHONOURABLE` -> PPScheduleRefused, which voids the pass for the WHOLE GROUP. Deleter restored at the spend. STAGE A: the stamp population and the entry population were two different sets -- the stamp sits behind eight `continue` branches (`already_in_batch` is the sgl-project#946 shape: `add_chunked_req` appends the continuation to `can_run_list` BEFORE the loop, which then skips it), while the decision is built from `can_run_list`. Boot-measured in the gap: `loop_skips( prefetch_pending=1(first=bce57cf8...))`, 3x, the same 4618-token rid. The key is now stamped over `can_run_list` itself, so the two populations are identical by construction. This also closes the NEXT link, which was already visible: the publish filter needs `load_back_len AND load_back_key`, so an entry with a live offer and no key would still have published 0. SILENTLY-WRONG PAIR, mandatory in this build because fixes 1+2 are what arm the path -- once live it returns wrong tokens rather than crashing: * The adopted GDN state is the node's END state, at `_applied`; the S1 clamp cuts KV to `told` and cannot cut a transfer that already ran. The scan would resume at `told` from a state that has consumed `told.._applied-1`. No assert can fire; the KV geometry is self-consistent at `told`. And it is the EXPECTED relation, not a corner (a4c9a7a: "stale-low by construction", measured 1215 -> 1216 -> 2114). The whole load-back now falls instead; the state is NEVER trimmed, per the hazard the code already spells out. The slot is returned before the raise, because the exception bypasses the sgl-project#991 revert site. * Mamba slot starvation used to skip the restore while the KV prefix grew to `told` -- warm-booked prefix over a state never loaded. It now fails the whole load-back. KV extent and GDN state stand or fall together, one rule. HEALTH ISOLATION (sgl-project#942, standing user order): a /health_generate probe is a liveness instrument, not work. Upstream only dispatches one when the box is `is_fully_idle`, so at idle the probe IS the entire backlog, and under purity `tp_threshold` is 0 -- its single token decided the layout of the whole instance. Excluded from all FOUR backlog terms via `_pending_prefill_tokens(include_health=False)` and `_admissible_prefill_tokens(include_health=False)`, at the economy readers only. THE SPLIT IS AT THE CALL SITE, AND THAT IS THE WHOLE POINT. A blanket exclusion is the obvious form and is wrong in the silent direction: `tp_compute_fits_in_one_chunk` grants only while `0 < pending < chunk`, so subtracting the probe there too collapses the sgl-project#887 grant and the probe is served by nothing at all. Service readers keep the default; economy readers exclude. No new tag: upstream already stamps the rid at the /health_generate source, and its own predicate reads `.rid` via getattr, so it applies unchanged to Req, TokenizedGenerateReqInput and PPAdmissionEntry. MEASURABILITY (without these the next boot cannot read its own result): * `cached_tokens` was structurally blind for exactly the population the warm acceptance lands on: the booking sits under `if not req.retracted_stain` and the stain is an ever-flag, while the cutover design retracts and re-admits everything. Now booked per admission by assignment (never `+=`, so the double count the stain guards stays impossible) and the stain is kept, because two other consumers read it. * The publish probe counted PASSES: with fewer than 64 publishes the `%64` arm never fired and the whole budget went to the boot's cold head, leaving every warm opportunity unobserved. Now event-cadenced, and health is its own column rather than mixed into the denominator. * `sgl-project#783` named a term that was not the one firing (it printed two identical numbers); each of the three terms now names itself. * The `sgl-project#998` sample printed `items()[:3]` of a never-pruned dict, i.e. the OLDEST entries forever -- one genuine event echoed 35k+ times, which reads as a request that stayed resident all boot. Now `[-3:]`. Ordering guard at `truncate_prefix_to`: it zeroes `host_hit_length`, which the offer is captured from. Dormant only because sgl-project#1015 deleted the receiving end; whoever restores that wire must re-check this order first. Tests: test/srt/test_health_isolation_942.py, 7 tests against the production functions, all green; can-fail proven by mutation (exclusion -> no-op gives 3 failures, incl. "a health probe armed the tp-ward flip"). Cross-module exception identity for the new refusal verified as identical. AST checks on signature/call-site agreement (9 sites) and on control-flow ordering (35/35, 24 failures against pristine HEAD). ruff 108 = 108 vs HEAD, no new findings. Not proven on metal at the symptom -- that is the next boot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.