Skip to content

docs: init readthedocs support - #783

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:read
Jul 28, 2024
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:read

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 28, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@zhyncs
zhyncs merged commit 9486257 into sgl-project:main Jul 28, 2024
@zhyncs
zhyncs deleted the read branch July 28, 2024 06:50
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025
…ect#783)

* Update test_utils.py

* Update test_utils.py

* Update test_ascend_debug_tensor_input_file.py

* Update run_suite.py

* Update run_suite.py

* Change lint job to run on linux-amd64-cpu-1

* Update lint.yml

* Refactor lint workflow for improved configuration

* Update daily-build-test-npu-innersource.yml

* Fix syntax error in merged PR benchmark workflow

* Update pr-test-npu-debug.yml

* Clean up blank lines in activation.py

Removed unnecessary blank lines in activation.py.

* Fix indentation in attention layer methods

* Update io_struct.py

* Update dbrx.py

* Update phi.py

* Update test_utils.py

* Clean up unused imports in test_vlm_utils.py

Removed unused imports from the test file.

* Add blank line before main execution block

* Add model path to TestLing class

* Update test_ascend_llm_models_phi_4_multimodal.py

* Update test_ascend_llm_models_smollm_1_7b.py

* Update test_ascend_sleep_on_idle.py

* Update test_ascend_llm_models_Qwen3_32B.py

* Update test_serving_chat.py

* Update run_suite.py

* Update test_server_args.py

* Update daily-build-test-npu-innersource.yml

* Update pr-test-npu-debug.yml

* Update activation.py

* Update torch_native_backend.py

* Update io_struct.py

* Update phi.py

* Update test_utils.py

* Update test_ascend_llm_models_Stablelm_2_1_6b.py

* Update test_ascend_llm_models_ling.py

* Update test_ascend_llm_models_smollm_1_7b.py

* Update merged-pr-benchmark.yml

* Update run_suite.py

* Update daily-build-test-npu-innersource.yml
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
The prefix cache was dark instance-wide: 2892 of 2892 prefill batches on
boot_735_standS read `#cached-token: 0`, and three identical ~6k-token
prompts each came back with no cached tokens. The read path was the
suspect, but the tree was simply empty -- nothing was ever retained.

MambaComponent.prepare_for_caching_req answers with a cache_len, and
UnifiedRadixCache.cache_finished_req min()s every component's answer into
one shared effective_cache_len. When a request's end did not land on the
--mamba-checkpoint-interval grid, the component answered 0. That 0 is a
statement about mamba's OWN anchor -- it has no on-grid state to file --
but as a shared minimum it collapsed the whole insert and threw away the
full-attention KV as well, which does not depend on the mamba grid at all.

With an interval coarser than the traffic the veto is total: the rig ran
8192 against ~6k-token prompts, so EVERY end is off-grid, every insert
cached nothing, and every match therefore returned zero. The branch the
rig actually ran (no --enable-mamba-extra-buffer) was also silent, which
is why an entirely dark cache looked like idle traffic for 2892 batches.

The tree already documents the intended behaviour in two places: "A leaf
survives on its Full value alone; auxiliary components (SWA, Mamba) may
legitimately hold only a tombstone" (unified_radix_cache.py), and, for
int8 pool exhaustion, "the node is then inserted without a mamba value
... so the KV stays cached and only the mamba resume point is lost".
The off-grid path contradicted both.

Declining the anchor now returns None ("no constraint from me") instead
of 0, so the node is retained at full length with a mamba tombstone, and
commit_insert_component_data treats a missing state as that tombstone
rather than asserting on it -- the assert also made the documented int8
exhaustion path a latent crash. Retention and resume-eligibility stay
separate decisions: a tombstoned node carries no state, so the match
validator still refuses it as a resume anchor and no request can resume
on a prefix whose recurrent state is absent. Declines are counted and
logged, so a grid coarser than the traffic announces itself.

Tests (hermetic, CUDA_VISIBLE_DEVICES=""):
  TestUnifiedRadixCacheMambaCheckpointGridRetention
    test_off_grid_end_retains_full_kv        red before, green after
        (before: AssertionError: 0 != 64, "an off-grid mamba end must
         not discard the full-attention KV"); also asserts matched == 0,
         pinning that a tombstone is never served as a resume anchor
    test_on_grid_end_retains_and_matches     green before and after
    test_grid_disabled_retains_and_matches   green before and after
  The two green-before tests are the can-fail control: the same
  assertions pass whenever the grid is reachable, so the red one
  discriminates the defect rather than the harness.

Full test_unified_radix_cache_unittest.py, before vs after: 212 errors
and 648 skips in both runs, identical failure sets, zero new failures
(the 212 are pre-existing CPU-environment errors on a hermetic host).
ruff clean on mamba_component.py (the 6 findings in the test file are
pre-existing, lines 1637-2182); codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…ey units

Second defect on the same dark-cache path, and the one that actually
governs this rig: with --speculative-algorithm NEXTN the radix key is an
EAGLE bigram view, where 64 raw tokens are 63 bigrams. The checkpoint
grid was then tested in two different unit systems.

  retention (prepare_for_caching_req)  token_ids_len   RAW tokens
  insert backstop (commit_insert_...)  len(params.key) KEY units
  match walk (create_match_validator)  cum_tokens      KEY units

So an anchor had to satisfy `interval | n` on the retention side and
`interval | (n - 1)` on the other two. For any interval above 1 that pair
is unsatisfiable, at every request length: n on the grid means n-1 is not,
and the backstop refused the anchor retention had just certified; n-1 on
the grid means retention declined first. EAGLE plus
--mamba-checkpoint-interval therefore could not produce a single prefix
hit, independent of prompt length or of how reachable the grid was.

Measured before the fix (desk, FULL+MAMBA, interval 64, eagle):
  n=64  retained 63  matched 0      n=65  retained 64  matched 0
  n=128 retained 127 matched 0      n=129 retained 128 matched 0
after:
  n=64  retained 63  matched 63     n=65  retained 64  matched 0
  n=128 retained 127 matched 127    n=129 retained 128 matched 0
The off-grid lengths still match 0 on purpose: retained, no anchor, so
not resumable.

The grid is a statement about how many tokens the recurrent state has
consumed, so raw tokens are the correct unit and the two key-unit sites
convert through `_raw_token_pos` (identity outside EAGLE, so the
non-EAGLE lineage is byte-identical).

Tests:
  test_eagle_on_grid_end_matches            new, red before / green after
  test_eagle_off_grid_end_retains_without_anchor  new
  Can-fail control: disabling ONLY the `+1` in `_raw_token_pos` reddens
  test_eagle_on_grid_end_matches with "AssertionError: 0 != 63: an
  on-grid EAGLE anchor must be accepted on match" and leaves the other
  four green, so the test discriminates this fix and not the harness.

Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips before
and after, identical failure sets, zero new failures (the 212 are
pre-existing CPU-environment errors on a hermetic host). ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
…sides

The two fixes on this branch make prefixes reusable again, and the failure
mode that has to stay shut while doing so is sgl-project#767: a resume whose KV and
recurrent state belong to DIFFERENT positions, which surfaced as a foreign
request's text continuing into a fresh prompt. Widening a match is exactly
how that comes back, so the guard is now asserted, not argued.

test_resume_stops_at_the_position_the_state_was_taken builds a path where
deeper KV genuinely exists past the anchor -- otherwise the assertion would
be vacuous:
  A: 128 raw tokens, on-grid at interval 64 -> files an anchor at raw 128
  B: 168 raw tokens extending A, off-grid   -> KV retained, NO anchor
The path then carries 167 bigrams of KV while the deepest snapshotted state
is at raw 128, and the test requires both: that the tail really was retained
(167), and that a query for B resumes at 127 bigrams (raw 128), never 167.

Can-fail control: making is_resume_candidate accept anchorless nodes
reproduces the sgl-project#767 widening exactly --
  "AssertionError: 167 != 127: the resume point must be the raw position
   the state was taken at"
and simultaneously reddens the two tombstone guards with
  "AssertionError: 64 != 0: a tombstoned node must never serve as a resume
   anchor"
so all three assertions discriminate the rule rather than the harness.

Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips, identical
failure set to the pre-sgl-project#783 baseline, zero new failures. ruff clean on the
added code (the 6 findings in this file are pre-existing, lines 1637-2182).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 20, 2026
First boot of this branch died on the pool ledger:

  ValueError: pool memory leak detected! [full] total=161378,
  available=40952, evictable=130, protected=0, session_held=0,
  uncached=0, withheld=120418

40952 + 130 + 120418 = 161500 against a total of 161378: 122 slots
unaccounted, and the log named them -- "off-grid end 122 (interval 512)".

Cause, and it was mine. `cache_unfinished_req` inserts and then IMMEDIATELY
re-matches, transferring ownership on the strength of that match
(`req.cache_protected_len = len(new_indices)`). An anchorless node is
deliberately unmatchable, so the round trip comes back empty: the tree took
121 of the request's 122 slots while the request kept using -- and later
freeing -- the same slots, with cache_protected_len falling back to 0.

Retention is therefore correct only where ownership actually transfers:
on FINISH, where the request is over and the tree takes the KV outright.
Mid-flight steps keep the pre-existing "cache nothing this step" behaviour;
their KV stays with the request and is retained when it finishes. The two
answers now come from one documented helper, `_decline_retention`, instead
of an unqualified `return None`.

This does not weaken the original fix. The repro is a repeated prompt, and
what has to survive is the COMPLETED request's KV -- that is the finish
path. On-grid chunk ends are unaffected either way: they file an anchor,
the re-match returns indices, and ownership transfers normally.

Tests:
  test_off_grid_unfinished_step_conserves_pool_slots   new, green
    Asserts OWNERSHIP, not allocator conservation: mid-flight the slots are
    legitimately still held by the live request (the running ledger counts
    them as `uncached`), so what must hold is that the tree did not ALSO
    claim them.
    Can-fail control: making _decline_retention return None unconditionally
    reddens it with "AssertionError: 121 != 0: an off-grid mid-flight step
    must cache nothing", which is the double claim the boot reported.
    The first version of this test passed under that same control because
    full_untruncated_fill_ids was never populated, so get_fill_ids() was
    empty and the step did nothing at all; it only counts now that the
    control can redden it.

Full test_unified_radix_cache_unittest.py: 212 errors / 648 skips,
identical failure set to the pre-sgl-project#783 baseline, zero new failures.
ruff clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…ark transport batches

W37-G (boot_w37g_0825_1326.log, pin 0be4ff4) proved the #861j doors open
on metal -- SEAM TRANSPORT ADMITTED x3, TP decode formed at bs 5-6 with
accept ~2.4-2.7, per-rid tokens strictly rising -- and exposed the next gap:
all 27 TP transport batches recomputed cold (#cached-token: 0, the sgl-project#783
dark-radix shape defeating the premise at runtime) while the conformance
count stayed 0, because layout_conformance.work_layout_verdict -- built
after W37-D's 258 identical batches -- had ZERO callers. Desk-written-
never-executed, in the instrument meant to falsify exactly this shape.

* metrics_reporter: _note_work_layout wired into report_prefill_stats
  before the logging-rank gate (counters are rank-local); phase from the
  same authority as the phase label, strict from purity_of, fully
  try/except'd. Decode side deliberately not wired -- named in the
  docstring, not omitted silently.
* scheduler: the admission that rides the exemption stamps
  batch.is_seam_transport from the same transport_only local that spends
  the seam stamps, so the claim cannot drift from the admission.
* layout_conformance: the restore excuse is now cached_tokens > 0 (was
  AND new_tokens <= 0, which would have flagged every legitimate
  read-through -- a real restore still computes >=1 token per request).
  The defect signature is cached == 0, W37-D and W37-G both.

Red-proven in a temp worktree at HEAD: the cold-transport batch through the
REAL report_prefill_stats counts 0 ("the detector for exactly this shape
has no caller") and the build-site pin finds no transport mark; green on
this tree. Can-fails: genuine partial restore stays silent, ordinary
prefill-in-tp still flags, prefill-in-pp never flags.

Tests: test_tp_decode_formation_861j.py 13/13 (dark-radix arm added),
scoped impact 130/130 (conformance/economy/band/forward-pass/exemption/
servability), ruff delta 0, codespell delta 0. Full serial managers
battery: 8 failed / 4320 passed / 2 skipped / 356 subtests
in 722.41s (/tmp/861k_gate2.log), cards VISIBLE -- the Tier-2 managers gate
is not hermetic and CVD="" flips its verdict in both directions. Failure
set identical BY NAME to the standing 8: staging_reserve_631 x4,
rank_prefill_log::TestCollectiveClock x1, hisparse_unit x2,
draft_cuda_graph_removal x1. Unexpected names: none.

WURZEL. Klasse: desk-written-never-executed -- an instrument built, unit-
tested against hand-made inputs, and never wired to the path it judges, so
it reports green by never running; test-only coverage is what HIDES it.
Geschwister, swept tree-wide by AST over python/sglang/srt (143 instrument-
shaped defs, references counted outside the def line), sweep validated in
both directions (it lists work_layout_verdict with metrics_reporter.py
masked and goes silent with it wired): 10 members with zero production
callers. Three are load-bearing and all in managers --
  * progress_liveness.py: the WHOLE sgl-project#699 module has zero production
    importers (only its test file), including thrash_verdict (#861e), which
    has exactly one reference repo-wide, its own def line, and no test at
    all. That is the detector for "decode advances, completions do not" --
    the W37-D/W37-G shape and the sgl-project#857 acceptance discriminator.
  * corridor_guard.phase_corridor_verdict (sgl-project#784): 7 test call sites, zero
    production callers; the constants are imported widely, the verdict is
    not.
Zukunfts-Check: OWED and specified, not built here -- the sweep becomes a
registered test that fails when an instrument-shaped def has no production
caller, with an allowlist carrying an owner and a reason per dark entry.
Filed as #861l so this commit stays the wiring change. Full list and
file:lines in /spinning/gpu-arb/NOTE_861fg.md.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…althy fence

W37-G (boot_w37g_0825_1326.log): 39 cutovers in 5.5 min, ZERO completions,
and `#cached-token: 0` on all 209 prefill batch lines. The inherited framing
for sgl-project#783 was "the radix tree holds rows, match returns 0" -- a dark READ
path. That is falsified in both halves.

Every fence of the entire boot:

    33 x eligible=0 staged=0 already_staged=0 acked=0 outstanding=0 elapsed=0.000s/2.000s
     3 x eligible=3 staged=0 already_staged=3 acked=0 outstanding=2 elapsed=0.000s/2.000s
     3 x eligible=1 staged=0 already_staged=1 acked=0 outstanding=1 elapsed=0.000s/2.000s

`acked=0` in every one. Nothing was ever confirmed into the canonical store.
`eligible = len(_hashed_nodes(tree_cache))`, so eligible=0 means the device
tree carried no hashed node at fence time -- and the tree drop independently
returned 0 rows on 36 of 39 cutovers, including cutovers that retracted 6 and
7 requests. Two independent measures agree the tree was EMPTY. A read path
cannot be the root when there is nothing to read.

WHY NOTHING SAW IT. The seam's guard is headed "sgl-project#703 FENCE COVERAGE,
ASSERTED RATHER THAN ASSUMED" and warns only when `_writeback_fence_ms(...)
is None`. That helper returns `elapsed_s`, and an empty fence returns 0.0,
not None -- silent on all 33. `FlipWritebackReport.complete` is
`outstanding == 0`, so those same 33 also logged at INFO as COMPLETE. Two
instruments, both correct, both blind to the one state that matters: a cheap
fence and a healthy fence are indistinguishable BY COST, because the healthy
case is also cheap.

`test_writeback_fence_ms_856.py` names this exact outcome in its own
docstring -- "a defaulted zero would report such a flip as fully fenced while
nothing was persisted" -- and then pins the wrong variable. It guards a
DEFAULTED zero; W37-G produced a GENUINE zero from an empty tree and reached
that outcome by a path the test never considered.

THIS COMMIT IS THE DETECTOR ONLY. `FlipWritebackReport.persisted_nothing`
asks the question the read-through actually depends on -- is anything
retrievable after the tree is dropped -- from counts the report already
carried and nothing read. Deliberately narrow: true only when NEITHER route
survives (no storage ack AND no pre-existing host copy), so the 6 lapsed
fences that already draw the "deadline reached with backups still in flight"
warning are not double-reported. Wired into the seam so it is not another
desk-written instrument.

Tests: red confirmed before the fix (AttributeError on `persisted_nothing`).
The wiring is proven BY EXECUTION, not by `inspect.getsource` as all four
existing tests of `_release_residents_for_cutover` do -- source inspection
cannot tell a guard that runs from one that is unreachable. Both directions
pinned (empty fence warns / healthy fence silent / lapsed fence silent), and
an `if False` mutant on the guard kills the wiring test. 79 passed across the
seam neighbourhood plus the #861l reachability gate. black/isort/ruff
(--select=F401,F821,UP037, the repo gate) clean on the changed files;
codespell and UP037 hits in phase_flip_runtime.py are pre-existing on HEAD
and untouched.

Also included: isort reordered two pre-existing imports in
phase_flip_runtime.py. Gate-mandated, unrelated to sgl-project#783, named here rather
than left silent.

NOT FIXED HERE. The root is a closed loop: retention is finish-only since
ff5651a, the flip retracts every resident before it finishes,
`retract_all` -> `release_req(offload_kv=False)` frees that KV without tree
insert or writeback, so the tree stays empty, the fence has nothing eligible,
the store is never written, re-admission finds neither tier and recomputes in
full, which does not finish before the next cutover. `build_cutover_release`
justifies `offload_kv=False` as "the FENCE has already persisted the
prefixes ... so a second device->host copy would pay twice" -- a premise
W37-G falsifies, and the same false premise is load-bearing in two places.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…not by handover

THE DEFECT. The sgl-project#856 cutover retracts every resident and keeps nothing, so
the state a phase transition must preserve simply does not survive it:

  * PHASE-FLIP-GDN moved 0 slot(s) on ALL 228 flips of the W37 series. The
    GDN mover reads `flip_mamba_slots` = resident mamba slots UNION tree
    checkpoints, and BOTH are destroyed at phase_flip_runtime.py:10829, one
    seam step before the mover runs at :11139.
  * The sgl-project#703 fence walked an empty tree: eligible=0 on 78 of 85 cutovers,
    acked=0 on every fence of the boot.
  * #cached-token: 0 on 403 of 403 prefill batch lines (W37-H arm A, the
    control on this tree).

WHY NOT AN INSERT. Persisting at that instant by inserting into the tree was
tried and measured: W37-H arm B died 33 s after ready, all three ranks,
`pool memory leak detected! total=468981, available=108565, evictable=1,
withheld=360437` -- 108565+1+360437 = 469003, i.e. 22 rows claimed twice.
`readmit_seam_residents` brings the population straight back ("W31: RE-ADMIT
THEM") and it resumes on rows the tree had taken. Ownership transfer is
structurally wrong for a population that returns; `cache_finished_req` is the
finish CODE PATH, not a finished request.

THE FIX, IN TWO HALVES THAT ONLY WORK TOGETHER.

Half 1 -- copy at the cutover retraction. `release_req(..., copy_state=False)`
calls `offload_kv_cache` BEFORE `release_kv_cache`, while the rows still hold
live bytes. Row ownership is untouched, so the arm B double claim cannot
recur (`is_insert` stays False). Threaded through `retract_all` and set True
at EXACTLY ONE line, `build_cutover_release._retract`. That single-site
property is load-bearing: `retract_all`'s only other caller is the
decode-pressure path, whose rate is load-dependent, and the host budget
(0.585 GiB/cutover, ~57 MiB/s aggregate) is priced at the flip cadence.
"Ungated by design" means no flag -- the condition is structural.

Half 2 -- restore where the rows come back. NOT in `readmit_seam_residents`,
which only re-queues: by then `reset_for_retract()` has cleared
`req_pool_idx`. `restore_seam_state` runs in `prepare_for_extend`'s per-req
loop, after `alloc_for_extend` (rows exist) and before `is_retracted` is
cleared -- the same shape as the proven disagg path, `_pre_alloc` then
`load_kv_cache`. The guard is the PRESENCE OF THE COPY, not `is_retracted`:
the flag is also true for pressure retractions that never copied, so keying
on it would request a restore that does not exist. The copy is consumed, so
a second pass cannot re-apply stale bytes.

Enabling change: the mamba half of the copy did not exist on this rig's pool.
`Req.offload_kv_cache` promised "copies over both the kv cache and mamba
state" and passed `mamba_indices`, which `UnifiedSWAKVPool` accepts and never
reads -- correctly, since a KV pool with no mamba pool has nothing to copy.
Only `HybridLinearKVPool` honours it, because it owns one. So the capability
is now DECLARED (`supports_mamba_cpu_copy`, default False on `KVCache`, True
on `HybridLinearKVPool`, forwarded by the allocator to its `_kvcache`) and
`Req` dispatches on the declaration, covering exactly the pools that have no
owner to delegate to. Exactly one mover per payload, enforced rather than
promised. The form reuses the house idiom (`BasePrefixCache.supports_mamba`
/ `supports_swa`); the allocator tree had none of its own.

The declaration must take the SAME hop as the payload: allocators implement
`get_cpu_copy` as `self._kvcache.get_cpu_copy(...)`, so a declaration that
did not forward would let `Req` read the base False while the pool underneath
copies mamba -- copying it twice. The getattr default is kept there because
`UnifiedKVPool` and `DeepSeekV4UnifiedKVPool` declare no base class and
genuinely cannot answer; it is removed at the `Req` callsite, where every
allocator inherits the method and a missing attribute is a real defect.

TESTS
  test_seam_state_copy_783          4  copy fires at the cutover and nowhere
                                      else; pressure retraction unchanged
  test_seam_state_restore_783       6  restore round-trips, is consumed, and
                                      is wired after alloc / before the clear
  test_seam_ownership_ledger_783    8  drives the PRODUCTION invariant
                                      (SchedulerInvariantChecker._check_pool_
                                      invariant) over a claim ledger; gates
                                      the COMBINATION, and its can-fail shows
                                      half 1 alone balanced while the pair
                                      leaks -- the arm B shape, at the desk
  test_unified_mamba_cpu_copy_783   8  the declaration is pinned, including a
                                      pool that declares True and lies
Each file carries at least one assertion that passes without the change, so a
broken fixture cannot masquerade as a finding (it caught three of mine).

Serial managers battery, cards visible (this gate is not hermetic):
  4358 collected = 4348 passed + 8 failed + 2 skipped, 356 subtests, 721.20s
  Failure set identical BY NAME to the standing 8:
    staging_reserve_631 x4, rank_prefill_log x1, hisparse_unit x2,
    draft_cuda_graph_removal x1
  Zero unexpected names, zero standing failures missing. Collected == ran, so
  the run is complete -- including test_pp_retracted_pass_void_797, which
  wedged in an earlier run while arm A was still winding down.
black/isort applied, ruff gate (F401,F821,UP037) clean, codespell clean.

NOT PROVEN HERE: that the prefix actually survives a flip on metal. That is
the boot, and its acceptance lines are #cached-token > 0 after a flip against
403/403 zeros, and PHASE-FLIP-GDN moved > 0 against 228 zeros.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…laration mechanics

W38-A (pin eeed2bc) reached /health and died 14 s into the load, all three
ranks:

    prepare_for_extend -> restore_seam_state -> load_kv_cache
      -> HybridLinearKVPool.load_cpu_copy -> memory_pool.py:3295
    IndexError: list index out of range

`load_cpu_copy` walks `range(0, len(indices), chunk_size)` and indexes the
saved chunk list, so it requires the restore's extent to be no larger than the
copy's. It was larger.

ROOT: the restore borrowed the disaggregation shape (`_pre_alloc(req)` then
`load_kv_cache`), which works there because the request is PRE-ALLOCATED IN
FULL. The normal prefill path admits CHUNKED: `prepare_for_extend` allocates
rows for the current chunk only, so `req_to_token[req_pool_idx, :seqlen-1]` at
restore time does not describe the extent that existed at copy time. I checked
WHERE the rows exist and never WHAT EXTENT they cover.

NOT PATCHED WITH A min(). The two extents describe different things; clamping
one into the other would write a prefix's KV into the wrong rows -- a wrong
ANSWER instead of a crash. The IndexError is the better failure and the extent
question is answered before this is rebuilt.

BOTH HALVES GO, not just half 2. Half 1 alone would copy on every cutover
retraction with nobody consuming it: pure host cost (0.585 GiB per cutover,
~57 MiB/s aggregate) for no benefit. "Both halves or none" cuts against a
partial revert here.

REVERTED
  release_req / retract_all      `copy_state` parameter and the copy call
  build_cutover_release._retract `copy_state=True`
  prepare_for_extend             the `restore_seam_state` hook
  schedule_batch                 `restore_seam_state` itself
  test_seam_state_copy_783 / test_seam_state_restore_783 -- they pinned code
  that no longer exists

KEPT, because it is sound and was expensive to establish
  `supports_mamba_cpu_copy` on KVCache (False) / HybridLinearKVPool (True) /
  the allocator forwarding hop, and Req's dispatch on it. W38-A's traceback
  proves the rig runs HybridLinearKVPool -- the pool with the COMPLETE
  implementation -- so the dispatch correctly let the pool own the mamba copy
  and skipped Req's. It behaved as designed on a pool nobody had predicted
  would be the one in play. Note this also retires the earlier "the rig's pool
  cannot copy mamba" framing: it always could. What D adds is the explicit
  single-owner declaration, not a closed capability gap.
  test_seam_ownership_ledger_783, test_unified_mamba_cpu_copy_783.

Tests after revert: 26 passed (ledger + mamba-cpu-copy + fence), 87 passed
across the seam regression set (seam_order_856, seam_readmission_w31,
tree_drop_returns_rows_856, uniform_retract_count_583,
pp_retracted_pass_void_797). ruff gate clean.

Also carried, and named rather than left silent: black reformatted two
pre-existing `assert` statements in schedule_batch.py during this work. Gate
output, unrelated to sgl-project#783.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…fuses instead of indexing

Second attempt at carrying the cutover's state across the flip. The first
(eeed2bc, reverted in dc5d557) died 14 s into W38-A's load with
`IndexError: list index out of range` at memory_pool.py:3295, three ranks.

ROOT, and it is one sentence: A LOGICAL LENGTH WAS USED AS A PHYSICAL EXTENT.
`Req.seqlen` is `len(origin_input_ids) + len(output_ids)` -- the request's
final logical length, which says nothing about how much of `req_to_token` has
been written. During chunked prefill the row is filled only to
`kv_allocated_len` (== `extend_range.end`), strictly less. Both
`offload_kv_cache` and `load_kv_cache` index by `seqlen - 1`, so both are sound
only once prefill is COMPLETE. A restore whose extent had grown walked off the
end of the saved chunk list.

The vulnerability was LATENT, not pre-existing-and-tripped-over: both methods
had exactly one caller each, and both were on fully-preallocated
decode-disaggregation paths. My seam wiring was the first caller to make the
chunked path reachable, so this commit owns the reachability it created.

THE TREE HAD ALREADY DECIDED THIS QUESTION -- AGAINST what the first attempt
built. `kv_session_offload` refuses chunked admission outright rather than
restoring into it: ":445 return False  # would be CHUNKED -> needs PS3", and
the assert at :4496 naming "PS3 (host-prefix extend read)". PS3 has four
mentions in this tree, all comments: unimplemented. This commit matches that
answer instead of quietly building the unimplemented thing.

THE CONTRACT
  Req.kv_cache_cpu_extent   recorded at capture; a copy states what it covers
  seam_copy_state()         DECLINES a mid-chunk request -- no full extent
  restore_seam_state()      REFUSES on extent drift and DROPS the stale copy
  _seam_prefill_is_complete `kv_allocated_len >= seqlen - 1`, i.e. the physical
                            extent must have caught up with the logical length

REFUSAL, NEVER A CLAMP. The two extents describe different things; a min()
would write a prefix's KV into the wrong rows -- a wrong ANSWER instead of a
crash. A refusal costs a recompute, which is merely slow. It lives at the
CALLER because that is the only layer holding both numbers: every pool-level
`load_cpu_copy` merely forwards indices, and its lone assert (:3298) is
per-chunk, on the wrong axis entirely. There is no backstop below.

SCOPE, deliberately bounded: decode-phase residents -- the population the
cutover actually retracts, and the one that loses real session state -- are
covered. A mid-chunk request loses no session state by recomputing; it repeats
unfinished work. PS3 is named, not half-built.

OBSERVABILITY IS PART OF THE FIX, not an addendum. The first attempt's restore
logged nothing at all, which is the desk-written shape this branch has spent
its time closing.
  [sgl-project#783 seam-state] SEAM RESTORE rid=.. extent=N ... (copied/declined/
                    restored/refused)
  [sgl-project#783 seam-state] SEAM RESTORE REFUSED rid=.. covers X, needs Y
  [sgl-project#783 seam-state] SEAM COPY DECLINED rid=.. prefill still chunked
`declined` is also the MEASUREMENT of how often a mid-chunk request meets the
seam -- so the PS3 question gets a number rather than a guess.

WIRING, one site each: `copy_state=True` only at
`build_cutover_release._retract` (the decode-pressure path never sets it, which
is what keeps the host budget priced at the flip cadence); the restore in
`prepare_for_extend` after `alloc_for_extend` and before `is_retracted` is
cleared.

TESTS  test_seam_extent_contract_783, 11 passed, including TWO controls that
pass without the change -- one of them asserts the stub pool can REPRODUCE the
W38-A IndexError, without which the reds would say nothing about the real pool.
Regression: 103 passed (extent + ledger + mamba-cpu-copy + fence +
seam_order_856 + seam_readmission_w31 + uniform_retract_count_583 +
pp_retracted_pass_void_797).

Serial managers battery, cards visible:
  4359 collected = 4349 passed + 8 failed + 2 skipped, 356 subtests, 723.32s
  Failure set identical BY NAME to the standing 8: staging_reserve_631 x4,
  hisparse_unit x2, rank_prefill_log x1, draft_cuda_graph_removal x1.
  Zero unexpected, zero missing. Collected == accounted, so the run is
  complete. Passed delta vs the last battery is exactly -10 (two removed
  plumbing test files) +11 (this one) = +1.
black/isort applied; ruff gate clean on the changed files.

NOT PROVEN HERE: that the prefix survives a flip on metal.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 25, 2026
…both axes

THE SILENT HALF IS THE DEFECT. `-1` is simultaneously the classic stale-row
sentinel and valid torch indexing, so a stale id does not fault: it writes the
LAST row of the pool. Wrong KV -- or, on the mamba axis, wrong recurrent state
-- under a prefix the radix tree reports as RESTORED. No crash, no log line, on
a shipped path, refused nowhere until this commit. Two new tests pin it as a
fact about the code as it stood:

    test_a_negative_row_does_NOT_fault_and_hits_the_last_row      (MHA, dim 0)
    test_a_negative_slot_does_NOT_fault_and_hits_the_last_slot    (Mamba, dim 1)

Both PASS without the fix. That is the point of them.

The illegal memory access was the LOUD half, and it is the only reason we
looked. W40b, 2026-08-25 19:52:52, three PP ranks, instance down five seconds
later:

    prepare_for_extend -> restore_seam_state (schedule_batch.py:2748, :2091)
      -> Req.load_kv_cache (:1778) -> load_cpu_copy (memory_pool.py:3303)
      -> current_platform.synchronize()
    torch.AcceleratorError: CUDA error: an illegal memory access was encountered

ROOT. sgl-project#783's extent contract (25e7849) compares ONE SCALAR -- the row COUNT a
copy covers against the count the request now needs -- and then forwards
`req_to_token[req_pool_idx, :seqlen-1]` to the pools as ids that nothing
validates. EQUAL LENGTH IS NOT EQUAL VALIDITY. The pools index their buffers
directly and the only assert on the path (MHA :3298) compares the saved host
chunk's SHAPE against the chunk LENGTH -- a different axis, which that commit
already names: "per-chunk, on the wrong axis entirely. There is no backstop
below." This is that backstop. It sits BESIDE :3298, not in place of it: one
checks the saved payload lines up with the index count, the other checks the
index VALUES exist. Neither implies the other.

A MODULE FUNCTION, NOT A METHOD, AND THE BOUND IS AN ARGUMENT.
`check_cpu_copy_rows(indices, rows, direction, axis)`. The pools that need it do
not share a base -- `MHATokenToKVPool` and `MLATokenToKVPool` are `KVCache`
subclasses and `MambaPool` is not -- so a guard installed on one ancestor would
be silently inert on the other. The bound and axis are passed in because they
genuinely differ: MHA is bounded by KV buffer rows on dim 0, `MambaPool` by its
slot count on dim 1 (`conv[:, indices]`, layout [num_layers, num_slots, ...]).
An earlier draft of this guard read `k_buffer[0].shape[0]` off `self`; wiring
Mamba to it would have looked done and done nothing. A guard that cannot fire is
the failure mode this whole sweep exists to remove.

REFUSAL, NEVER A CLAMP -- the parent commit's rule for the parent commit's
reason: the ids and the pool describe different things, so dropping or wrapping
the offending id writes state into rows belonging to someone else. A refusal
costs a recompute, which is merely slow. Checked BEFORE the first store, with a
test asserting the pool is byte-identical after a refusal: a guard that fires
halfway through is a louder corruption, not a fix.

THE SWEEP -- 14 `get/load_cpu_copy` definitions in memory_pool.py alone, not
one. The first pass at this assumed a single site.

  WIRED   MHATokenToKVPool :3262/:3284  the W40b site, and what this rig's
          HybridLinearKVPool delegates its KV half to (:4390, :4401)
  WIRED   MambaPool :1180/:1192  dim 1, bound mamba_cache.conv[0].shape[1];
          reached here via HybridLinearKVPool -> mamba_pool.load_cpu_copy(
          _mamba_translate(...)), and that translation is a second staleness
          source
  n/a     HybridLinearKVPool :4389/:4399 pure delegator, covered by both above
  n/a     KVCache ABC :2319/:2322 NotImplementedError stubs
  FILED   PageMajorMHATokenToKVPool :4006/:4012, MLATokenToKVPool :4674/:4689,
          DSATokenToKVPool :5023/:5048, dsa_cache_layer_split.py, and the
          forwarders in allocator/{base,token,paged,swa}.py,
          swa_memory_pool.py:285-330, unified_memory_pool.py:1223-1237

The 12 unwired sites are not on this rig's path. They are enumerated with their
file:line in docs/dev/NOTE_783b_cpu_copy_row_bounds.md rather than wired on
faith.

PRIOR EVIDENCE THAT THIS BELONGS AT THE POOL BOUNDARY, not per-caller: two
forwarders already filter their indices -- swa_memory_pool.py:295
(`swa_indices[swa_mask]`) and unified_memory_pool.py:1227 (`swa_phys[valid]`) --
the SWA half ONLY, with the unfiltered `full_phys` sitting directly beside them.
The need was seen once and not generalised. Putting the check where every caller
must pass through it is the answer to that, rather than trusting each caller to
remember.

OBSERVABILITY, because the emitter could not witness its own event.
`restore_seam_state`'s log line sat AFTER `req.load_kv_cache`, so a restore that
does not return is exactly what it cannot report: W40a logged 18 seam lines,
W40b crashed INSIDE the call and logged ZERO, which is why the traceback names
`synchronize()` and not the restore. An ATTEMPT line now goes before, rate
limited on the same cadence as the success line.

TESTS  test_seam_row_bounds_783b, 17 passed. Hermetic, mocks only: the REAL pool
methods bound to CPU buffers, because a test that reimplemented the copy would
encode the defect's own assumption -- the shape 25e7849 corrected in three of
its own mocks. A/B against this tip (fc24204): 9 failed / 8 passed WITHOUT
the change, 17 passed WITH it. That A/B is the mutation proof; the 8 that pass
either way are controls, including the two silent-half pins above and the
over-refusal guards (valid rows still load, an empty index set is a no-op).

Hermetic suites, run SEPARATELY per sgl-project#749 against fc24204, count gate
(^FAILED + ^SUBFAILED + ^ERROR == summary) PASSED on all three before any
comparison:

  mem_cache    2 failed / 1774 passed / 1658 skipped / 361 subtests
               gate 2 == 2. Baseline on this tip was 1757 passed; +17 is exactly
               this commit's tests. The 2 are the standing
               test_acceptance_emitters_758 pair, "RuntimeError: No CUDA GPUs
               are available" -- measured pre-existing earlier tonight by
               reverting to the tip and re-running, not assumed.
  managers     15 failed / 4343 passed / 356 subtests   gate 15 == 15
               IDENTICAL to the post-#858b baseline, by count and by name:
               test_arena_high_water_631 (7), test_phase_flip_rotation_wiring_809
               (4), test_restore_never_rebuild_677 (4), all hermetic-environment
               CUDA failures.
  distributed  95 failed / 3017 passed / 1170 subtests  gate 92+3+0 == 95
               IDENTICAL to baseline. The 3 SUBFAILED are the
               dcp_size=2/3/4 subtests; summary extracted BY PATTERN because
               these arms print UCX teardown warnings after pytest's summary.

Zero new failures on any arm. ruff: memory_pool.py 0 before and after,
schedule_batch.py 41 before and 41 after (untouched pre-existing), the new test
6 x E402 from the house `register_cpu_ci()`-between-imports convention, the same
class the existing test_seam_extent_contract_783.py carries (4). codespell
clean. black/isort deliberately NOT run over these files: the venv's black is
OLDER than the one this tree was formatted with and isort relocates a
deliberately-late `# noqa: E402` import in memory_pool_host.py -- every added
line was checked against the 88-column limit by hand instead.

STILL OPEN, AND NOT CLOSED BY GUESSING: why did W40a's nine restores succeed
while W40b's first one did not? Unknown. No trigger has been written into this
commit to make the story tidy. The guard lands anyway because it is the
INSTRUMENT that will answer it -- it converts an asynchronous IMA attributed to
an unrelated `synchronize()` into a deterministic host-side refusal naming the
id, the bound and the axis. CUDA_LAUNCH_BLOCKING=1 belongs in the sgl-project#857
acceptance arm so an IMA is attributed to its own kernel.

NOT PROVEN HERE: that a seam restore survives a flip on metal. This commit makes
the failure loud and specific instead of asynchronous or silent; making it WORK
is the acceptance boot.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…o the guard can arm

sgl-project#718/sgl-project#847 built every part of the rebind except one: the 'tp' staging pin was
assembled with a SINGLE entry, PoolName.KV. On a hybrid model the live tier
carries KV *and* MAMBA, so `check_pool_coverage` computed `missing={MAMBA}`
and refused -- correctly, on every cutover. A refused rebind leaves the sgl-project#718
device tier DISARMED, `load()` returns None, every read-through misses, and
every prefix a cutover retracted is recomputed in full.

Measured on the W40 sgl-project#857 acceptance boot (boot_w40_857strict_0825_2342.log):
60 `sgl-project#719 HiCache rebind refused`, ZERO arms, and `#cached-token: 0` on all 243
prefill batch lines -- one bucket, no exceptions.

THE PRECONDITION WAS ALREADY WRITTEN DOWN, in the guard's own docstring:
"A phase host tier has to be built with the FULL POOL SET before this rebind
can arm; until then the sgl-project#718 disarm is the correct state and a read-through
miss is the correct cost." This is that precondition, met. Not a new finding --
sgl-project#856 answers whether the retraction is necessary (it is: carry moves Req
scheduling metadata, never KV bytes, and sgl-project#856 deliberately retired the movers),
and sgl-project#718/sgl-project#847 already named this remedy.

REFUSAL CONVERSION, NOT GUARD DELETION. `check_pool_coverage` is untouched. It
must stop firing because its precondition is MET, never because it was removed,
and the test asserts BOTH directions -- a full pool set arms, a narrowed one
still refuses. Neither assertion alone can tell a fix from a disarm.

DERIVED FROM THE BOUND TIER, not from the model config: the set that must be
covered is whatever the READER names, which is the same quantity the guard
compares. Reading the config would be a second opinion about one fact.

MIRRORS `build_hybrid_mamba_stack` rather than re-deriving it -- same
primitives, same layer mappings, same transfer_layer_num rule. The controller
is deliberately NOT reused: this pin needs a host VIEW, and a second
HybridCacheController would be a second writer against one device pool.

Both entries are rebuilt in the hybrid case. The KV-only pin used an identity
map over range(layers), which is right while KV is the only entry and wrong the
moment a second pool shares the transfer index space -- the two maps collide at
index 0.

SIZING IS PER-SLOT, NOT PER-GB, and it is the one place the mamba half must not
copy the KV half. MambaPoolHost reads host_size in GB only when > 0, else
`device_pool.size * ratio`. The KV pin's GB figure is derived from a token
count; mamba is allocated per request slot. Ratio 1.0 with host_size 0 mirrors
the device pool, which is what a phase-matched staging pin means.

The mamba half is a NAMED HOST-LEDGER POST, priced from what was allocated
rather than from the intention. An unpriced pinned pool is what the ledger
exists to prevent.

A pool set this builder cannot mirror (SWA, indexer, DeepSeek) is logged as an
ERROR at the cause and still refused by the guard -- named, not swallowed.

THE CHECK (sgl-project#871, third scope item): `advance_fence_blind_streak` +
FENCE_BLIND_STREAK=4, aggregating the EXISTING `persisted_nothing` instrument.
One empty fence is legitimate; every empty fence means the canonical store can
never populate, and that shows up only as latency. Gated on `released`,
mirroring sgl-project#719's busy gate at the stale-gate streak: a fence over an empty tree
is correct to persist nothing, so counting quiet cutovers would build a
crying-wolf alarm out of the instrument written to replace one. NO SECOND
COUNTER -- a parallel "recomputed prefix tokens" counter would measure what
this and #cached-token already measure between them.

Extracted as a pure function so it is falsifiable without booting. A guard whose
logic can only be exercised by booting is a guard that ships unexercised, which
is the failure mode this ticket is about.

CLASS (unchanged, carried from the sweep): a store whose only writer is a
lifecycle event another mechanism systematically preempts. Retention is
finish-only; the cutover retracts before finish; the store can never populate,
so every recovery path reading it can never fire.

TEST RESULTS

test_phase_tier_full_pool_set_871.py (new): 15 passed.
FALSIFIED IN THREE DIRECTIONS BEFORE THE GREEN WAS CLAIMED, count gate held on
each:
  guard removed (disarm)          -> 3 failed, and they are exactly the refusal
                                     tests. 3 extracted == 3 in summary.
  guard fires unconditionally     -> 6 failed, exactly the arming tests. 6 == 6.
  streak loses its `released` gate -> 1 failed, the idle test. 1 == 1.
  restored                        -> 15 passed.

Targeted regression set (7 files: sgl-project#847 writer, sgl-project#783 fence x3, sgl-project#856 empty wave,
counters, and the new file): 97 passed, 0 failed, 2 subtests passed. Count gate
0 extracted. Hermetic, CUDA_VISIBLE_DEVICES verified EMPTY at the PROCESS
(/proc/<pid>/environ), not at the command.

AN EXISTING TEST CAUGHT A REAL BUG OF MINE and the first run of that set was
3 failed: the new HOST-LEDGER line read `tp_host.entry_map` directly, but that
writer is driven in tests by stand-ins where HostPoolGroup itself is patched.
Fixed with the getattr discipline this module states at its other probes -- an
instrument may never be the thing that breaks a boot.

Lint: ruff 0 before and after on phase_flip_runtime.py and the new test.
phase_flip_boot.py reads 1 before AND after -- a pre-existing F401 on
`pack_into_arena`, present in HEAD, not mine. codespell clean.
Flag-off path verified byte-identical: without --phase-flip-rebind-hicache the
writer still returns {} and allocates nothing.

NOT CLAIMED, AND DELIBERATELY NOT GUESSED: no metal run. Whether the kv+mamba
pin can actually be ALLOCATED on this box, what its real HOST-LEDGER post
comes to against the 16G floor, and whether #cached-token becomes non-zero are
decidable only on hardware. They belong to a boot window. The sgl-project#857 acceptance
instance was left running and untouched throughout -- it is the standing proof
and the operator declared it taboo.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…ct#868's 15, and one keyword wide

sgl-project#864. The question was whether the 16 standing failures are the same
hermeticity form as the 15 that sgl-project#868 resolved. They are its MIRROR IMAGE,
and the mirror is the direction that hurts:

                       hermetic   cards    the desk gate shows
  sgl-project#868's 15            FAIL       pass     a red nobody can act on
  sgl-project#864's 16            SKIP       FAIL     a GREEN over a standing red

Same class -- the invocation decides the verdict -- but sgl-project#868's announce
themselves on every desk run, while these announce themselves nowhere
anybody looks: the machine that can see them has cards, and the gate people
read does not.

CAUSE, proved without a card. hiradix_cache.py:266-278 calls
all_reduce(tensor, op=op, group=group, async_op=True) unconditionally. The
double at test_unified_radix_cache_unittest.py:2382 is
fake(tensor, op=None, group=None). All 16 die with a TypeError inside
mock's dispatch BEFORE their first assertion, so they are not evidence
about SWA prefetch adoption in either direction. Checked the other half of
the call too rather than assuming: the double returns None, and
hicache_collective.py:190-196 documents None as a completed no-op -- so the
drift is exactly one keyword wide and a one-line fix is the whole fix.
TEST-ONLY. Zero live defects. sgl-project#783 gets nothing from them.

Corrects the prior art it rests on: NOTE_861fg's "16 passed" hermetic is a
numeric COINCIDENCE, not the same 16. Measured: -k tp_swa_prefetch under
CVD="" gives 38 skipped / 0 passed / 0 failed (8+8 for want of a card,
11+11 non-SWA parametrisations; tally 38 == summary 38). They are inside
the 1506 skips, not the 16 passes.

sgl-project#862. mem_cache hermetic: 2 failed / 1774 passed / 1658 skipped, both tally
axes OK (names 2 == summary 2; extracted 1658 == summary 1658). Both
failures DEVICE-caused; REAL defects 0. The 1658 split by reason:

  HARDWARE 996 | DEPENDENCY 12 | CONFIG-not-darkness 645 | BY ACCIDENT 0
  UNCLASSIFIED 5

So the darkness is BY DESIGN -- BY ACCIDENT is zero and the conftest
mechanism explains itself in its own docstring. The ticket is still right
for a finer reason: 1008 of those would run on a card and 645 would never
run anywhere, and the summary line renders those identical facts.

My own classifier was wrong first and the data said so: two buckets put 636
of 1658 in neither. Folding them into "dark" would have inflated the figure
by 62 %. All 36 survivors were read individually and encoded; 5 remain
unclassified and are reported as such rather than tuned away.

Two more corrections to prior art: mem_cache DOES run directory-wide on CPU
here (3396 collected, no abort), so the recorded collection-time abort does
not reproduce on this tree -- reported, not bisected. And the escape behind
it is real but at RUNTIME and two tests wide: rotation_executor.py:344
allocates with pin_memory=True, a device door that never calls get_device(),
which is the one door the conftest guard patches.

THE CHECK, in the CPU lane on purpose. New
test_double_signature_drift_864.py (2 tests, 0.11 s, register_cpu_ci) parses
the double and the production call sites and asserts the first can bind the
second -- no device, no ranks, no import of a module that needs them. It
also asserts production still passes async_op, so the guard reports a change
instead of silently testing a call that no longer exists. Can-fail proven:
re-introducing the drift turns it red with the exact diagnostic.

Regression: mem_cache after the change is 2 failed / 1776 passed / 1658
skipped -- same failures, same skips, +2 from the new guard.

NOT done and recorded: the 16 were never run WITH cards, so the fix is
proved to remove the drift and not proved to make them pass. Every number
here is a CVD="" number, verified on the process; nvidia-smi compute-apps
showed only the other runner's serving ranks throughout.

(cherry picked from commit a428907)
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026
`--verify` named exactly one new module,
test/registered/unit/mem_cache/test_pool_invariant_live_double_912b.py, and
no sha256 among the 184 already in the table moved. `test_seam_extent_
contract_783.py` DID change bytes, but it sits in the managers table's
long-standing unclassified set, so no fast-lane proof went stale with it --
PARALLEL stayed at 270 there, checked rather than assumed.

MEASURED on this tip, hermetic (CUDA_VISIBLE_DEVICES=""):
  * fresh serial reference, 185 modules, one process, one order:
    2 failed / 2761 passed / 927 skipped, 164.8s. 2761 = 2745 + exactly the
    16 the new module contributes; the 2 failures are the same two
    test_acceptance_emitters_758.py::RefillTiming names, compared as a LIST.
  * the 184 solo logs carried forward, plus one new solo probe.
  Demotions: ZERO, counted by diffing the table's own rows before and after.

  before  PARALLEL 171  RANKS 13  UNCLASSIFIED 1
  after   PARALLEL 172  RANKS 13  SERIAL 0  EXCLUDED 0   -> VERIFY OK

gate_partition.tsv (managers) again NOT rebuilt, for the reason recorded in
Train-2c: its 35-module gap has no surviving solo logs, so a rebuild would
demote 270 proven rows to make two new ones look better. sgl-project#920's module and
sgl-project#917's before it therefore run in the serial lane by the default-for-the-
unknown rule, both named in the runner's report, and the lane's arithmetic
shows it: 617 -> 630 passed, which is exactly the new module's 13. The
adapted sgl-project#783 contract contributes 0 to that delta -- 11 passed on this tip
and 11 on 3502a9a, measured on both, so pinning the DECISION instead of
the literal changed what the tests assert without changing how many there
are.

gate_partition_scheduler.tsv untouched: seam_census_double.py is a harness,
not a test_*.py, so it carries no table row.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…omote, and the silently-wrong pair

ONE build closing the publish -> cached_tokens chain, plus the health-probe
confound that made every measurement of it ambiguous.

THE TWO BINDING TERMS (both desk-proven at the tree, both root, not symptom):

1. The promote `pending -> effective` existed at exactly ONE site, inside
   `_event_loop_pp_body`. The warm prefill on this rig runs boot-measured in
   the TP phase (the instance flags it itself: `sgl-project#838 work_in_wrong_layout
   class=prefill phase=tp`, 9 genuine violations), where that body never runs.
   PP0 published into `pending` and nobody ever promoted it: `sgl-project#968 LOAD-BACK
   PROMOTED` = 0/0/0 over three whole boots, and that is a REAL zero -- its
   cadence counts promotions, so the first would have printed.
   Fix: `promote_pp_load_back_pending()`, idempotent, called from both loop
   bodies AND from `_get_new_batch_prefill_raw` (the phase-neutral consumer),
   above every early return.

2. `init_pp_loop_state` nulled `_pp_load_back_pending` UNCONDITIONALLY while
   the line directly below guarded `_pp_load_back_effective` with `hasattr`.
   The unconditional null was deliberate ("only pending is a per-pass
   quantity") and is only fatal BECAUSE the promote was phase-bound. Fix 1 is
   the root; this is the guard that carries the fact across the cutover.

REGRESSION IN e0bc960, FOUND AND FIXED: that commit moved the offer write
to the load-back site but kept only the `_hit > 0` half, deleting the only
deleter -- the old `= _hit if _hit > 0 else None`. The comment it removed
states the lost property verbatim ("goes to None on its own"). The offer then
outlived its own truth: stale-HIGH re-publish -> a peer told an extent its
host tier no longer covers -> `EXTENT UNHONOURABLE` -> PPScheduleRefused,
which voids the pass for the WHOLE GROUP. Deleter restored at the spend.

STAGE A: the stamp population and the entry population were two different
sets -- the stamp sits behind eight `continue` branches (`already_in_batch`
is the sgl-project#946 shape: `add_chunked_req` appends the continuation to
`can_run_list` BEFORE the loop, which then skips it), while the decision is
built from `can_run_list`. Boot-measured in the gap: `loop_skips(
prefetch_pending=1(first=bce57cf8...))`, 3x, the same 4618-token rid. The
key is now stamped over `can_run_list` itself, so the two populations are
identical by construction. This also closes the NEXT link, which was already
visible: the publish filter needs `load_back_len AND load_back_key`, so an
entry with a live offer and no key would still have published 0.

SILENTLY-WRONG PAIR, mandatory in this build because fixes 1+2 are what arm
the path -- once live it returns wrong tokens rather than crashing:
* The adopted GDN state is the node's END state, at `_applied`; the S1 clamp
  cuts KV to `told` and cannot cut a transfer that already ran. The scan
  would resume at `told` from a state that has consumed `told.._applied-1`.
  No assert can fire; the KV geometry is self-consistent at `told`. And it is
  the EXPECTED relation, not a corner (a4c9a7a: "stale-low by
  construction", measured 1215 -> 1216 -> 2114). The whole load-back now
  falls instead; the state is NEVER trimmed, per the hazard the code already
  spells out. The slot is returned before the raise, because the exception
  bypasses the sgl-project#991 revert site.
* Mamba slot starvation used to skip the restore while the KV prefix grew to
  `told` -- warm-booked prefix over a state never loaded. It now fails the
  whole load-back. KV extent and GDN state stand or fall together, one rule.

HEALTH ISOLATION (sgl-project#942, standing user order): a /health_generate probe is a
liveness instrument, not work. Upstream only dispatches one when the box is
`is_fully_idle`, so at idle the probe IS the entire backlog, and under purity
`tp_threshold` is 0 -- its single token decided the layout of the whole
instance. Excluded from all FOUR backlog terms via
`_pending_prefill_tokens(include_health=False)` and
`_admissible_prefill_tokens(include_health=False)`, at the economy readers
only.

THE SPLIT IS AT THE CALL SITE, AND THAT IS THE WHOLE POINT. A blanket
exclusion is the obvious form and is wrong in the silent direction:
`tp_compute_fits_in_one_chunk` grants only while `0 < pending < chunk`, so
subtracting the probe there too collapses the sgl-project#887 grant and the probe is
served by nothing at all. Service readers keep the default; economy readers
exclude. No new tag: upstream already stamps the rid at the
/health_generate source, and its own predicate reads `.rid` via getattr, so
it applies unchanged to Req, TokenizedGenerateReqInput and PPAdmissionEntry.

MEASURABILITY (without these the next boot cannot read its own result):
* `cached_tokens` was structurally blind for exactly the population the warm
  acceptance lands on: the booking sits under `if not req.retracted_stain`
  and the stain is an ever-flag, while the cutover design retracts and
  re-admits everything. Now booked per admission by assignment (never `+=`,
  so the double count the stain guards stays impossible) and the stain is
  kept, because two other consumers read it.
* The publish probe counted PASSES: with fewer than 64 publishes the `%64`
  arm never fired and the whole budget went to the boot's cold head, leaving
  every warm opportunity unobserved. Now event-cadenced, and health is its
  own column rather than mixed into the denominator.
* `sgl-project#783` named a term that was not the one firing (it printed two identical
  numbers); each of the three terms now names itself.
* The `sgl-project#998` sample printed `items()[:3]` of a never-pruned dict, i.e. the
  OLDEST entries forever -- one genuine event echoed 35k+ times, which reads
  as a request that stayed resident all boot. Now `[-3:]`.

Ordering guard at `truncate_prefix_to`: it zeroes `host_hit_length`, which
the offer is captured from. Dormant only because sgl-project#1015 deleted the receiving
end; whoever restores that wire must re-check this order first.

Tests: test/srt/test_health_isolation_942.py, 7 tests against the production
functions, all green; can-fail proven by mutation (exclusion -> no-op gives 3
failures, incl. "a health probe armed the tp-ward flip"). Cross-module
exception identity for the new refusal verified as identical. AST checks on
signature/call-site agreement (9 sites) and on control-flow ordering (35/35,
24 failures against pristine HEAD). ruff 108 = 108 vs HEAD, no new findings.
Not proven on metal at the symptom -- that is the next boot.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant