Repository navigation
Move sampling logits to float32 - #773
Merged
Merged
Conversation
2 of 3 tasks
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cherryblo
added a commit
to cherryblo/sglang-project
that referenced
this pull request
Jul 2, 2026
update perf param and delete minimax testcases
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…e lineage that actually runs The floor in mamba_pool_floor.py is only a guarantee if something bounds what cache retention may pin. That bound existed in exactly one place -- HiMambaRadixCache._mamba_pin_budget -- and registry.py:107 records that "HiMambaRadixCache has no construction site anywhere": every hybrid-SSM boot under --enable-hierarchical-cache is routed to UnifiedRadixCache. So the floor was charged at boot while the runtime ran unbounded. Two independent protections were unreachable on that lineage: 1. THE PIN BUDGET. UnifiedRadixCache.write_backup took inc_lock_ref(node) with no budget check. With write_through the threshold is 1, so every inserted checkpoint was pinned the moment it was created; acquire_component_lock takes a MAMBA ref on any node carrying a value, so each pin froze a state slot, and the eviction walk skips pinned nodes. The pinned set could therefore ratchet until it owned the pool and a REQUIRED allocation had nowhere to go. That is sgl-project#581 verbatim, and it explains why raising --max-mamba-cache-size only ever bought time: the ratchet scales with the pool. sgl-project#743 already COUNTED these pins for the trace line (_mamba_pins_in(ongoing_write_through)) without ever bounding them -- a counter with no actuator. The counter is reused rather than duplicated so the trace and the bound cannot disagree. 2. EVICT-BEFORE-FAILING. bind_tree_cache was called only from MambaRadixCache.__init__. Unbound, HybridReqToTokenPool.tree_cache stays None, so _alloc_mamba_slots_or_evict never evicts and retries -- the pool reported exhaustion while cached, EVICTABLE checkpoints sat in the tree. The #639b rank-parity tombstone branch hangs off the same handle and was equally unreachable. The budget is derived in mamba_pool_floor.mamba_retention_pin_budget so both lineages read one function; the previous split is what let a lineage ship without it. Refusing a backup costs at most a host-tier miss: the state stays cached on the device and stays evictable. Refusals are counted and rate-limited into the log, because a sustained count means the ack drain is not keeping up with insert pressure. Measured, not assumed, on boot_798_0822_0646 (Qwen3.8-27B-INT8, PP3): the floor for that config is 16, not 24 -- mamba_radix_cache_strategy is no_buffer so the ping-pong term is 0, and mamba_slot_reorder is on, so per_req = 1 active + 0 ping-pong + 1 donation/pin. The pool of 24 sits 8 slots above the floor, which is exactly the budget that was unenforced. TESTS: test/registered/unit/mem_cache/test_mamba_pin_budget_live_773.py, 11 tests, hermetic (CPU-only, no DMA controller). Red-first, and the can-fail proof for each gate was EXECUTED, not asserted: * removing bind_tree_cache -> 3 tests fail * disabling the guard in write_backup -> 1 test fails The second mutant initially SURVIVED, because the other tests drove the predicate directly rather than write_backup; TestWriteBackupActuallyConsultsTheBudget exists because of that survival and asserts on control flow. Both directions of the sgl-project#581 argument are covered: unbounded pins are shown to starve a required alloc on this lineage today, and a zeroed floor is shown to make that starvation reachable again through the guard -- so the safety is a property of the floor, not of the guard. REGRESSION: test/registered/unit/mem_cache/ = 1449 passed, 8 failed. All 8 proven pre-existing by an identical run on an untouched HEAD worktree (test_acceptance_emitters_758, test_localslot_family_756, test_mamba_anchor_seams_747), diffed name-for-name before claiming it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…y where the lineage delivers it sgl-project#755 reduces the floor from 1+P+1+1 to 1+P+1: the donated slot BECOMES the next pin, so the two terms share one slot. The mechanism is real, gated per node, and tested -- but its config gate asks three questions about the CONFIG and none about the LINEAGE, and those select for opposite worlds. The gate requires enable_hierarchical_cache, because only a write-through host tier can promise the released anchor still exists. But registry.py routes a hybrid-SSM model WITH hierarchical cache to UnifiedRadixCache (:107-111), and MambaRadixCache -- the only class implementing the reorder -- is reachable only at :133, i.e. only when hierarchical cache is OFF: hierarchical=False -> reduction NOT taken (floor 24) | MambaRadixCache HAS impl hierarchical=True -> reduction TAKEN (floor 16) | UnifiedRadixCache NO impl Inverted in both rows. CacheInitParams.mamba_slot_reorder is filled from that same predicate on every boot (kv_cache_builder.py:237) and read only by mamba_radix_cache.py:525 -- always False where it is read, always ignored where it is True. The unified lineage's real demand was measured against the code, not assumed, and it is the FULL 1+P+1+1: * replacement slot allocated before the donation -- mamba_component.py:565 (_alloc_mamba_slot), called at :784/:757/:793 before donate_mamba_ping_pong_slot (memory_pool.py:1823), so a request transiently holds active + donated; * resume-anchor pin -- unified_radix_cache.py:1103, inc_lock_ref(new_last_node) right after insert at :1081, reaching MambaComponent.acquire_component_lock (mamba_component.py:491), which is what makes a STATE SLOT unevictable; * grep for mamba_slot_reorder across unified_radix_cache.py and unified_cache_components/ returns zero hits; MambaComponent's donate flow performs the un-reordered sequence regardless of the param. This was not cosmetic. schedule_policy.py:795-801 sizes the ADMISSION budget from mamba_slots_per_running_req for this same lineage (the branch keys on HybridReqToTokenPool, which is the token pool for BOTH trees; the "non-unified" in its comment refers to --enable-unified-memory, a different axis). So the standing boot charged 2 slots per request at both the floor and admission while the runtime held 3: a live one-slot-per-request under-floor, which is the sgl-project#581 direction -- the boot validates a pool the runtime over-draws and the shortfall surfaces late. The fix is a lineage predicate, and the capability is a module CONSTANT, not a config flag: it describes what the code can do, and an operator must never be able to assert it. Porting the reorder into the unified lineage flips UNIFIED_LINEAGE_IMPLEMENTS_SLOT_REORDER and the reduction returns with no other edit. Also adds the boot instrument that would have caught this: _validate_max_mamba_cache_size returns SILENTLY for any pool at or above the floor, so the floor only ever reached a log by REFUSING a boot. A pool sitting exactly on the floor looked identical to a comfortable one. The new MAMBA-FLOOR line states pool, floor and retention budget once at construction, and says plainly when the budget is 0. CONSEQUENCE, stated rather than left to be discovered: on the standing boot (--max-mamba-cache-size 24, --max-running-requests 8) the honest floor is 24, so the pool sits exactly ON it and the retention budget is 0 -- every mamba write-through backup is declined. That is the true posture, not a new restriction: the pool was always fully committed to the running set, the accounting just claimed otherwise. Buying cache retention means raising the pool, or implementing the reorder so the floor honestly falls to 16. The three sgl-project#755 tests that asserted the reduction from config alone now assert it under an explicit lineage patch. They test the MECHANISM's arithmetic, which is unchanged; what moved is where it is reachable. TESTS: test_mamba_reorder_lineage_773.py, 6 tests + 3 subtests. Can-fail EXECUTED: removing the lineage check fails 2. The original three gate conditions are each shown to still refuse on their own, so the gate was narrowed and not silently widened. Direction is asserted explicitly: the refused reduction moves the floor UP by exactly one slot per running request. REGRESSION: test/registered/unit/mem_cache/ = 1455 passed, 8 failed, the same 8 as an untouched HEAD worktree, diffed name-for-name.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
… unified lineage The previous commit refused the sgl-project#755 floor reduction because the lineage that gets built does not implement the mechanism. This builds it, and the floor drops back to 1+P+1 -- honestly this time, because the runtime now does what the arithmetic claims. MECHANISM. cache_unfinished_req releases the OLD anchor's mamba pin BEFORE the donation allocation instead of after the insert, so the old and new anchors never coexist and a running request holds active + donated rather than active + donated + old pin. That is the slot, and on the standing boot (max_running_requests 8) it is 8 of them. STRICTLY SAFER THAN THE ORIGINAL, and this is why the port was worth doing rather than re-gating. MambaRadixCache releases the WHOLE node lock early; inc_lock_ref walks ancestors for the FULL component, so its window also leaves the request's own matched KV prefix evictable. The unified tree locks per component, so dec_mamba_lock_only drops ONLY the mamba lock: the KV path stays protected and the single state slot being un-double-counted is the only thing made evictable. dec_swa_lock_only is the same shape for SWA, so this is the file's existing idiom. PER-NODE GATE. The config predicate cannot know whether THIS anchor survives losing its slot, so MambaComponent.anchor_release_admissible asks again: the node must carry a host copy (an evicted-but-backed anchor is a valid match that triggers load_back -- is_resume_candidate's device_only=False rule), and that copy must have LANDED. sgl-project#767: write-through publishes host_value the moment the transfer is handed to the controller and records the node in ongoing_write_through in the same block; between those two facts the anchor is an intention only, and releasing there is the dead anchor the gate exists to prevent. PAIRING, which is where this class of change goes wrong (sgl-project#583). Two exit paths, both handled: * the normal path skips the mamba half at the usual dec_lock_ref, via the skip_lock_node_ids set the components already consult -- a ref that was never taken must not be given back; * the bail path (nothing inserted, so no new anchor takes over the pin and req.last_node is unchanged) RE-ACQUIRES, restoring the exact pre-call state rather than carrying an imbalance into the next call. TESTS: test_mamba_reorder_unified_773.py, 17 tests, CPU-only. The end-to-end pair is the real proof: the pool is sized so the donation allocation can only succeed if the old anchor was released first (2 slots = one active + one anchor, free = 0). A backed anchor caches; an unbacked one, identical in every other respect, does not. That is the saved slot, observed as behaviour rather than as arithmetic. CAN-FAIL, ALL EXECUTED, and three of them were earned the hard way: * unwiring the reorder in cache_unfinished_req -> 1 fails * dropping the skip set -> 1 fails * dropping the bail re-acquire -> 1 fails The last two SURVIVED the first version of this file, because every pairing test simulated the sequence by hand instead of driving the entry point. release_component_lock no-ops at zero, so a double release on a singly-held node hides completely; it only shows when the anchor has a SECOND holder, whose ref the unskipped release would steal. Both tests were rewritten to go through cache_unfinished_req, and the skip-set case needed a spare slot -- with a full pool the step bails and correctly restores the pin, which is a different path and now has its own test. Floor on the standing boot: 24 -> 16. The retention budget introduced in 703b05c becomes 8 slots instead of 0, so mamba host backups are possible again rather than uniformly declined. REGRESSION: test/registered/unit/mem_cache/ = 1473 passed, 8 failed, the same 8 as an untouched HEAD worktree, diffed name-for-name.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…d-pinned number THE ACTUATOR. The earlier sgl-project#773 commits made the floor honest and made the write-through pin bound real, but they moved no bytes: the boot still pins --max-mamba-cache-size 24, so a corrected floor of 16 just sits underneath an unchanged pool. A counter with no actuator, which is exactly the shape this task exists to remove. Dropping the pin alone does not work, and that is the trap that kept it there. With the pin gone the sizing falls through to the --mamba-full-memory-ratio branch, which takes 0.9/1.9 of post-weights VRAM and produces a pool far LARGER than 24. The pin was compensating for a fall-through with nowhere good to land. The demand path already exists, is already floored at mamba_hard_floor (sgl-project#581) and already fitted to the budget (sgl-project#307). It was simply gated to uneven-DCP boots -- and the PP phase runs dcp_size 1, so the standing boot could never reach it. That single unreachable gate is why a VRAM number had to be chosen by hand on a rig whose planner is supposed to be the only VRAM authority. The widening adds a second route rather than replacing the first: size by demand when the operator STATED the concurrency. The predicate reuses the distinction _auto_mamba_target_concurrency already draws and already trusts -- a user-supplied --max-running-requests is a stated demand, an auto-defaulted one is not (the speculative hook resets an unset value to 48, and sizing to that over-provisions several GB and OOMs at pool init). Gate and target therefore cannot disagree about which numbers are real. A boot that never stated a concurrency keeps the fraction path, byte-identical. Derived result for the standing boot (mrr 8 stated, reorder on, ratio 2): floor 16, pool 20, retention budget 4. Cheaper than the hand-pinned 24 AND with real headroom above the floor for cache -- where pinning 16 by hand would have zeroed retention. 4 slots x 37.4 MiB = ~150 MiB on PP0, the binding rank, once the boot stops passing --max-mamba-cache-size. The number is now DERIVED. Retiring the pin is a boot-config change and is deliberately not made here. TESTS: test/registered/unit/model_executor/test_mamba_pool_from_floor_773.py, 9 tests + 4 subtests. Can-fail EXECUTED on both halves of the new gate: * removing the widening -> 1 fails (standing boot unreachable) * dropping the user_set check -> 2 fail (an auto-default would size) The second mutant first reported GREEN because ruff had reformatted the expression and the patch text silently did not match; the mutation step now asserts its target is present before believing a result. Recording that because a false green in a can-fail proof is worse than no proof. Each pre-existing refusal is shown to still refuse on its own (pinned size, explicit fraction, disabled radix), so the gate was narrowed-then-widened deliberately and not loosened by accident. REGRESSION: test/registered/unit/model_executor/ = 745 passed, 15 failed; the same 15 on a control worktree with my commits reverted, diffed name-for-name, and the +9 are these tests. mem_cache unchanged.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…o errors of mine TWO DEFECTS I INTRODUCED IN 5a1113e AND FOUND BEFORE ANY BOOT USED THEM. Both are recorded here rather than quietly repaired, because the second one is a false claim I put in a commit message. DEFECT 1 -- THE ACTUATOR WOULD HAVE INFLATED THE POOL. The demand path multiplies by _calculate_mamba_ratio(), not by the floor's per_req. That returns MAMBA_CACHE_SIZE_MAX_RUNNING_REQUESTS_RATIO (3) plus a buffer term, i.e. 3 on the standing boot, while the floor's per_req is 2. So dropping --max-mamba-cache-size would have derived max(ceil(8 * 3 * 1.25), 3, floor 16) = 30 against the 24 it was meant to replace: SIX SLOTS MORE per rank, and max_total_num_tokens DOWN. 5a1113e's message claims "pool 20"; that was true only of a test that passed ratio=2 by hand and never exercised the call site. Root cause is this task's own family defect once more. _calculate_mamba_ratio is a SECOND statement of "slots per running request" while mamba_pool_floor is the declared single source of truth for it. They agreed for every shape that existed when they were written; sgl-project#755 then taught the FLOOR about the lock reorder and nobody taught the RATIO. Measured across all five shapes: reorder OFF -> identical everywhere except extra_buffer_lazy, where the floor deliberately charges the transient second ping-pong slot (5) and this ratio does not (4); reorder ON -> this ratio overstates by exactly one slot per request in every configuration the reorder applies to. min() is therefore the exact repair: it moves nothing that already agreed, keeps lazy's deliberately smaller sizing ratio, and stops the ratio claiming a slot the floor no longer reserves. Corrected derivation for the standing boot: ratio 2, floor 16, pool 20, retention budget 4. DEFECT 2 -- I CLAIMED A CLEAN SUITE I HAD NOT CHECKED. 5a1113e says "mem_cache unchanged". It was not: the widened gate added a self._mamba_demand_target_is_stated() call that seven stub-driven tests in test_mamba_checkpoint_interval.py could not resolve. I ran that suite, saw 15 failures against a control of 8, and did not compare the two numbers. The repo had already built the guard for this exact mistake. _HANDLE_MAX_MAMBA_CACHE_SELF_CALLS plus its transitive-closure check exist because a0ed7dc added `self._auto_mamba_demand_active()` to the same function without extending the list, breaking four tests that nobody could see for six weeks. I reproduced that incident and its own guard caught me. The repair is the one the harness documents: declare the name. TESTS: test_mamba_pool_from_floor_773.py grows to 13 tests + 9 subtests. The new class asserts the ratio in BOTH directions -- that it no longer overstates under the reorder, and that every pre-reorder shape is byte-identical (no_buffer 3, extra_buffer+overlap 5, extra_buffer 4, extra_buffer_lazy 4, disable_radix 1), lazy included precisely because it is the one shape where the floor is larger and min() must NOT inflate. Can-fail EXECUTED: removing the cap fails 3. test_mamba_checkpoint_interval.py: 39 passed, 19 skipped. METAL, from boot_798_0822_0810.log (r6, commit ef9ccb6), all three ranks, matching the prediction made before the boot: MAMBA-FLOOR pool=24 floor=16 retention_budget=8 (8 running requests x (1 active + 0 ping-pong + 1 donation/pinned checkpoint (sgl-project#755 reorder: the donated slot BECOMES the pin, so the two share)) = 8 x 2 = 16 slots) floor=16 not 24 means the ported reorder ARMS in production; ping-pong 0 confirms the no_buffer reading; the retention budget is bounded and enforced where it was previously unbounded. Zero pin-budget refusals and zero reorder skips over the boot. REGRESSION: model_executor + mem_cache = 2222 passed, 23 failed, and those 23 are name-for-name identical to a control worktree with my commits reverted. Diffed, not eyeballed -- which is the lesson from defect 2.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…the unaccounted rows `_pool_census` derives `unaccounted` as `set(range(1, size+1)) - free - cached` from exactly ONE allocator and ONE tree, both read off the scheduler, with `cached` collecting only BASE-component DEVICE values. So a row owned by a DIFFERENT pool object is unaccounted BY DEFINITION rather than by defect -- and the census line cannot tell the two apart. This names every pool object it can reach, so they stop looking alike. WHY NOW. On the 2026-08-22 flips the ambiguity was load-bearing. r5 read ~94000 rows (21% of a 448698-row pool) as unaccounted, FLAT across four censuses, which reads as an unenumerated owner rather than a leak (a leak accumulates). r6 then showed the flatness was itself an artefact of r5 being wedged in TP -- every sample fell in one phase. With the flip cutting over, unaccounted swings with the phase instead: 08:16:44 post-cutover tp_to_pp free=345101 cached=82589 unaccounted=4096 08:17:09 post-cutover pp_to_tp free= 18704 cached=82589 unaccounted=330493 Same size, 25 seconds apart, cached IDENTICAL, free collapsing by 326397 rows across the cutover into TP and returning across the cutover into PP. That is not a leak and not static: it is whatever the census-read allocator stops listing while the TP stack is live. A second owner is known to be possible there -- `model_runner.py` splits `is_draft_pool_worker` from `is_draft_worker` on `is_phase_flip_tp_stack`, so a runner on that stack owns pools the scheduler's handles do not name. LIKE-FOR-LIKE, which the first draft of this got wrong. The census holds an ALLOCATOR while the flip's own reshard builder names POOLS (`scheduler.tp_worker.model_runner.token_to_kv_pool` and `stacks.tp_worker.model_runner.token_to_kv_pool`, :2041-2042). Comparing an allocator id against a pool id answers nothing, so both sides are reduced to the pool (`_owner_pool_of`) before any id is printed. The PP-stack pair is named alongside the census pair for the same reason: if CENSUS tracks PP_STACK in BOTH phases while the TP stack is live, the census is scoped to the PP stack and the rows are a measurement artefact, full stop. SHAPE OF THE SET. The census prints `sorted(leaked)[:12]`, which cannot resolve contiguity -- a "one contiguous block" reading was made and withdrawn on exactly that evidence, and the first twelve ids are byte-identical between the 4096-row census and the 330493-row one 25 seconds later, because the sample is just the low end of a sorted set. This reports n/min/max/runs/longest_run instead: bounded output, and `runs == 1` versus `runs` in the thousands answers the question directly. Read-only and best effort throughout, preserving the existing rule that a census can never affect the flip it watches; a hostile handle is caught and reported rather than raised. TESTS: test/registered/unit/managers/test_census_owner_probe_773.py, 9 tests. Can-fail EXECUTED: breaking run detection fails 1, and removing the call from `_pool_census` fails 1. That second mutant SURVIVED the first version, because every test drove the helper directly and none drove the call site -- the third time in this task a direct-helper test failed to bind its own wiring, so it now has its own class. REGRESSION: managers flip/census/presence/spill selection = 604 passed, 9 failed, and those 9 are name-for-name identical to a control worktree with my commits reverted. The +9 are these tests.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
… point The sizing tests in model_executor/test_mamba_pool_from_floor_773.py call _auto_mamba_demand_size(ratio) with a ratio passed BY HAND. That is exactly how the ratio defect shipped: the hand-passed 2 proved a number the call site never computes, while handle_max_mamba_cache multiplies by _calculate_mamba_ratio() and would have derived 30 -- larger than the hand-pinned 24 it was meant to replace. A test that chooses its own inputs cannot catch that, so this one drives handle_max_mamba_cache itself, bound from the real class, and reads the size the production path installs. It also pins a COUPLING that is easy to miss and expensive to get wrong. Removing --max-mamba-cache-size is only an improvement while the sgl-project#755 reorder ARMS. Unarmed, the floor is 1+P+1+1, the capped ratio follows it back up to 3, and the derived pool becomes 30 -- a REGRESSION of 225 MiB per rank against the pin it replaced. The two changes are therefore not independent, and a boot that drops the pin must confirm floor=16 on the MAMBA-FLOOR line before trusting the pool it is handed. Can-fail EXECUTED: restoring the uncapped ratio (the bug as shipped in 5a1113e) fails 2 of these 4 -- the failure the hand-passed tests could not see. Numbers are PP0, the binding rank, from boot_798_0822_0646: 0.877 GiB of main mamba state over 24 slots = 37.42 MiB per request, 12.153 GiB of post-weights budget. CONFIRMED ON METAL by boot_798_0822_0829 (r7, this branch with the pin removed from the argv), all three ranks: MAMBA-FLOOR pool=20 floor=16 retention_budget=4 and max_total_num_tokens 1198400 -> 1222016 on PP0's sizing line, +23616 tokens, +1.97%. Not 30 (the ratio cap took) and not 24 (the pin is gone). REGRESSION: test/registered/unit/mem_cache/ failures identical to a control worktree with my commits reverted, diffed name-for-name.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
ROOT, corrected against the handover's framing. The handover read the store
census as "15457 KV pages vs 11 mamba anchors = grid behaviour" and ordered
the anchors published at every write-through. Measured on the same boot
(boot_855_1028fence), that premise is a PAGE-vs-NODE denominator mix and the
ordered fix is already in place:
* `#969H BACKUP` = 33 lines = n=1..11 on EACH of PP0/PP1/PP2 at identical
timestamps. The probe logs every call up to n<=40, so 11 lines means
`write_backup` reached its component loop exactly 11 times per rank in the
whole run -- and all 11 carried `mamba_value=has_value`.
=> mamba coverage OF THE HOST-BACKUP PATH is 11/11 = 100%, not 11/15457.
* All 11 `.mamba` hashes in the store are also full-KV page hashes
(intersection 11, mamba-only 0): one anchor per node at a real page
boundary, shared key namespace.
* The 15457 KV pages are the PAGES of those same 11 nodes:
`write_backup_storage` writes `keys=node.hash_value` (every page of the
node) while the mamba branch writes `keys=[node.hash_value[-1]]` (one
trailing page).
The real hole is one link earlier and is not mamba-specific: `_inc_hit_count`
returns before any backup when `chunked=True` (upstream's "skip the hit count
update for chunked requests"). Under chunked prefill NOTHING is published per
chunk -- so the 11 backups are the 11 FINISHED requests, and a chunked prefill
that never finishes publishes nothing at all.
That is why the 13179-token prompt found its deepest anchor at 3072 and
recomputed the remaining 10107 tokens: the anchors are as dense as finished
requests, not as dense as chunks.
FIX: allow a chunked-prefill node to reach the host tier, gated structurally
on (storage tier present AND a MAMBA component present). The per-chunk node
already carries a donated state on the device
(`MambaComponent.prepare_for_caching_req`, is_finished=False branch); this
early return was the only reason it never reached host or storage.
UPSTREAM-MINIMAL: the chunked skip IS upstream (`hiradix_cache.py`), so this
is a DEVIATION and carries its burden of proof. Named: (1) drain-and-flip --
a chunked prefill interrupted by a flip never reaches `cache_finished_req`
(sgl-project#856 removed the carry, the flip DISCARDS it), so upstream's publish-at-finish
never fires; (2) GDN hybrid -- a recurrent state is valid at exactly one token
position, so the per-chunk anchor has no pure-attention analogue. Gate off =
byte-identical to upstream (verified: branch truth table, 0 mismatches over
all 8 chunked/write_back/force combinations).
LAWS: `mamba-per-knoten-nicht-gitter` in its own words ("states per radix
node/chunk like KV pages"), and it waives write volume explicitly.
`kein-doppel-prefill` (sgl-project#939): loss bound becomes ONE chunk.
sgl-project#968 PP0 DEBT -- WHERE THIS PATH STAYS RANK-LOCAL: the publish decision is
taken at the scheduler's chunk boundary, which every rank runs for the same
request at the same split, so it is unanimous BY CONSTRUCTION, not by
agreement; this path holds no reduce (the `check_prefetch_progress` MIN is
TP-scoped and the boot runs tp_size=1/pp_size=3, so it is structurally
skipped). `raenge-nie-uneins` is met by construction and `#1028P
CHUNK-PUBLISH` is how the claim gets CHECKED: identical n at identical
timestamps across ranks, the evidence shape `#969H` gave for the 11.
The one rank-local input reachable from here is the sgl-project#581/sgl-project#773 write-through
pin budget, which fired ZERO times in that boot (trap-safe: bare 0, genuine 0)
because 11 backups never approached it. Per-chunk publishing makes it
reachable for the first time; `pin_skipped` rides on the same line so a
nonzero count next to a divergent n is the divergence, named in advance.
COST, QUANTIFIED NOT BUILT: one `.mamba` blob is 78446592 B = 74.8 MiB
(measured, all 11 identical); a KV page is 32768 B. A published 4096-token
chunk therefore adds 74.8 MiB of anchor on top of 128 MiB of KV (+58% L3).
Host RAM is UNCHANGED -- the host mamba pool is pre-sized at boot from
`hicache_ratio` (1.5) x device slots, so anchors roll through a fixed tier and
land on disk. The int8-anchor idea (sgl-project#1013) would cut the 74.8 MiB and is
deliberately NOT built here.
ALSO: #1028B FETCH CAP instrument at the `min` in `hicache_storage.py`, the
cap that decides how much of an existing KV prefix a prefetch may claim. It
printed nothing: `final_pages`, `kv_pages`, `boundary=`, `hit_pages` each
occur 0 times in the whole 5.87 MB log, so "anchors too sparse" and "KV prefix
too short" produced the same number and were NOT separable from that boot.
Now both terms print on one line.
Desk checks (error-class matched): ruff F821/F811/F841/E9 clean on both
changed files (new names + new attribute reads); branch truth table executed,
gate-off equivalence exact. The gate is deliberately NOT memoised --
`enable_storage` is False at __init__ and only set in `init_hicache`, and a
memoised early False would leave a wired-but-inert write path, the sgl-project#742/sgl-project#745
class this area has produced before.
NOT YET BOOT-PROVEN. Acceptance is the next boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…lders; the cutover moved one WHAT WAS RED (parent a890b8a, test_1201_phase_stamped_handles.py): 13 failed, 3 passed. The 3 green are the default-path parity cases (an unowned cache still answers with its constructor pool; no counter to carry is not an error; a plain ReqToTokenPool without the hooks is not an error). After the cut: 16 passed; with the two sibling files (test_1040_req_pool_per_phase, test_cutover_participants_859) 82 passed. THE DEFECT. ReqToTokenPool is stamped onto four holders at CONSTRUCTION and rebind_req_pool_for_cutover moves exactly one, the scheduler's: * UnifiedRadixCache.req_to_token_pool (unified_radix_cache.py:429) * the pool's tree_cache back-reference (memory_pool.py:1871), whose only two bind_tree_cache callers are tree-cache constructors (unified_radix_cache.py:514-515, mamba_radix_cache.py:502-503), so the incoming pool keeps tree_cache=None for the whole phase and the evict-then-retry at memory_pool.py:2019-2024 never arms -- the sgl-project#581/sgl-project#773 regression that unified_radix_cache.py:507-513 was written to close * the pool's layer_transfer_counter (memory_pool.py:1876) * FutureMap.pool (overlap_utils.py:339), NOT moved by this cut Rank-uniform, so no ballot, digest or MIN can catch it: every rank is wrong the same way. Both phases' pools hold the same row count, so the divergence lands IN RANGE. Loud half: common.py:1849 -> free_slot's double-return refusal (memory_pool.py:492-497). Silent half, which runs FIRST: common.py:1836 reads the wrong pool's req_to_token row and hands those indices to token_to_kv_pool_allocator.free(). THE CUT. (a) UnifiedRadixCache.req_to_token_pool is a read-at-use property onto a registered owner, following kv_session_offload.py:2921-2933, whose docstring names this exact hazard. Unbound = byte-identical fallback to the constructor pool, so no non-flip boot changes. (b) _restamp_phase_handles re-stamps tree_cache, the pool's back-reference and the layer-transfer counter onto the incoming pool at the seam. (c) assert_req_pool_identity refuses at the end of the cutover (phase_flip_runtime, after the sgl-project#719 HiCache rebind, outside its try/except) when any holder still names a different pool. (d) cutover_participants: two new REGISTRY rows (request_pool_phase_ownership, req_pool_back_references) and req_to_token_pool added to MUTATED_STATE -- a HANDLE the cutover REPLACES, next to ten quantities it recomputes. future_map is on the registry as an explained gap, not moved. BRIEF PREMISE FALSIFIED -- (c) as briefed cannot be built. "Turn the two silent returns at memory_pool.py:1995-1998 / :3992-3994 into refusals" would refuse the DEFAULT path. layer_transfer_counter is None on every boot without a hierarchical cache: it is only ever set by register_layer_transfer_counter, whose callers are all HiCache/cache- controller paths, and swa_memory_pool.py:124-125 registers None on purpose; register_layer_transfer_counter's own docstring calls mamba_transfer_frame =None "the historic no-wait behaviour, unchanged". So None is a legitimate steady state, not a defect. What IS a defect is the handle failing to cross the seam, and that is decidable where both pools are in hand. The refusal therefore moved to the seam and the carry was built; the join sites are untouched. ANCHOR DRIFT (three, all reported rather than worked around): * "phase_req_pool_binding.py:180 is the only runtime rebind" -- :180 is the def line at the tree; the assignment is at :230, moved by CUT C (a890b8a). devindex still prints :180 because its store is pinned one commit back. * memory_pool.py:3992-3994 -- the guard is at :3993-3994; :3992 is a comment line. * memory_pool.py:496 -- the enclosing method free_slot is right, but :496 is a fragment of the double-return message; the line every finish hits is :506 (free -> free_slot), and :492-497 is the guard. All other anchors verified verbatim at the tree: unified_radix_cache.py:429 /507-513/514-515, mamba_radix_cache.py:502-503, memory_pool.py:1871/1876/ 1995-1998/2019-2024, mamba_component.py:480, common.py:1780/1836/1847/1849, kv_session_offload.py:2921-2924, cutover_participants.py:92/314-325. MUTANTS (scratch copies in /tmp/wt1201-orig, all restored): 1 DANGEROUS DIRECTION -- bind_req_pool_owner stores None, i.e. the pre-cut cached reference, which answers wrongly instead of refusing. Killed by 4: test_a_bound_cache_follows_the_owner, test_the_cutover_leaves_cache_and_scheduler_on_one_pool, test_the_kv_read_sees_the_rows_the_running_phase_wrote (the silent one), test_a_clean_cutover_passes. 2 DANGEROUS DIRECTION -- bind_tree_cache re-stamp removed, so the pool's evict-then-retry stays disarmed silently. Killed by 2: test_the_incoming_pool_is_bound_to_the_tree_cache, test_a_clean_cutover_passes. 3 assert_req_pool_identity returns early (#505a shape). Killed by 2: test_a_cache_left_on_the_outgoing_pool_is_refused, test_a_pool_left_bound_to_nothing_is_refused. 4 counter carry dropped. Killed by 1: test_the_layer_transfer_counter_is_carried_to_the_incoming_pool. VERIFIED: red-then-green tallies above; ruff parity on all four touched files (unified_radix_cache.py had 1 pre-existing F401 typing.Sequence before and after, zero new); import smoke of all four modules; no external writer to a tree cache's req_to_token_pool and no __dict__ access to it (grep, none); UnifiedRadixCache has no subclasses (devindex symbol_profile), so the property cannot be shadowed. NOT VERIFIED: no boot. Whether the layer-transfer counter carry ever fires on the standing flip boot is UNMEASURED -- it depends on whether the assembler registers a counter on both stacks' request pools, which only a boot decides. MambaRadixCache.req_to_token_pool (mamba_radix_cache.py:498) is the same cached-reference shape and is NOT converted here. ANSWER TO THE COMPLETENESS CRITIC, recorded as asked: this registry axis covers HANDLES THAT MOVED -- objects the cutover replaces and every holder that cached one. It does NOT cover WHAT WAS STILL OWED at the arm: launched-unreaped passes, outstanding receives, ring slots, lap counters. Nothing here detects a debt; MUTATED_STATE's read-window axis and the new handle axis are both about identity, not about obligations in flight. That second axis remains unbuilt and unclaimed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 24, 2026
…tzten Zustand auffrischen Upstream 48ae829 (sgl-project#31648, v0.5.20), angepasst: hinter SGLANG_MAMBA_LRU_REFRESH_USED_ONLY (Default aus). Portiert: MambaComponent.refresh_lru-Override (unified radix cache, den P und D des 27B bauen). Eingeschaltet: MATCH_END frischt im Mamba-LRU nur best_match_node auf (acquire_component_lock pinnt ohnehin nur dessen Zustand), WALKDOWN/INSERT_END lassen den Mamba-LRU in Ruhe (neue Zustaende kommen weiter ueber commit_insert_component_data). Upstream-Begruendung: Ganzketten-Refresh haelt die Zustaende einer Session im LRU beieinander und wirft kalte Sessions unter Pooldruck komplett raus. Warum opt-in statt Default: das ist eine Eviktions-POLITIK, keine reine Hot-Path-Optimierung (Match passiert je Admission, nicht je Decode-Schritt). Sie ordnet um, was drive_eviction zuerst tombstoned -- und die Fork-Anker-/ Retention-Mechanik (sgl-project#747 Grid-Retention, sgl-project#773/sgl-project#755 Anker-Release, sgl-project#1470 Flush-Join, sgl-project#1481 END-ANKER-Hold) ist nur gegen die Ganzketten-Ordnung gemessen; das aktive Flip x HiCache-Problem liegt genau dort. Aus = Basis- refresh, byte-identisch. Kandidat fuer einen A/B-Boot. Nicht portiert: sgl-project#31643 (dieselbe Idee fuer MambaRadixCache) -- keine weg2-Gruppe baut MambaRadixCache (D- und P-Log xsn411: "UnifiedRadixCache hybrid_ssm=True hierarchical=True"). Test (CPU, incg 3 GiB): test_unified_radix_cache_unittest.py neu TestUnifiedMambaLRUMatchRefresh (upstream-Test unter Env an + Default- Fall byte-identisch): 2 passed; Gegenprobe mit Basis-mamba_component.py: opt-in-Test rot. Teilmenge -k "lru|LRU|mamba|Mamba": 39 failed / 246 passed -- identische Fehlmenge wie auf der Basis (CPU-Altlasten, z.B. hicache Stream.query). test_mamba_reorder_unified_773 17, test_unified_mamba_views gruen; test_unified_mamba_pin_trace_581 1 failed / 10 passed = Altlast (auf der Basis identisch).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…aumaufbau -- fnFL2h91bb2 schrieb vom 97k-Flip nur 192 von 1528 Seiten Befund (Auftrag AW): im ersten 97k-Flip P->D von fnFL2h91bb2 (e17bd54) war die Arena beim D-Wake nicht "noch nicht", sondern NIE fertig geschrieben. P-Log Z. 2577 (14:25:54): 'MAMBA-FLOOR pool=32 floor=32 retention_budget=0 (8 running requests x (1 active + 1 ping-pong + 1 donation + 1 pinned checkpoint) = 8 x 4 = 32 slots) -- every mamba host backup will be declined' -- nur gesagt, nicht verweigert. Ab Chunk 2 (Z. 15270-15275, 14:30:41) 'mamba write-through pin budget reached (0 in flight, budget=0, pool=32)', 'sgl-project#1421 BACKUP-REFUSED why=mamba_pin node=9' (Chunk-1-Endknoten, der erste mit Mamba-Wert) und danach 'parent_unbacked' fuer die ganze Kette; RETAIN-PUBLISH issued=0 refused=217, keine MAMBA-ARENA-Zeile. In der Arena landeten nur die drei ankerlosen Stuecke von Chunk 1 = 192 Seiten -> D 'sgl-project#1439 ARENA-PRESENT leading_complete=192', '#1028B FETCH CAP mamba (0,-1)', '#1035c CAPPED by=mamba', 'sgl-project#1471 SETTLE read still short'. h91v1 (39fd662): pool=24 floor=16 budget=8, kein Refuse, 1528/1528, SETTLE-RELEASE nach 0,3 s. Das Arm-Extra '--max-mamba-cache-size 32' rechnete 2 Slots je Sitz ('haelt das Pin-Budget bei 16'), die Laufzeit rechnet 4: write_back verhindert den sgl-project#755-Reorder-Rabatt (braucht write_through) und damit auch nach 20 s kurz aus, der zweite Prefill wird genauso verweigert (xsn127-134: dieselbe Klasse bei Budget 1). Fix: UnifiedRadixCache._refuse_weg2_p_mamba_retention_zero nach der MAMBA-FLOOR-Zeile (ausserhalb deren breitem except): auf SGLANG_WEG2_GROUP=P mit hierarchischem Cache und _mamba_pin_budget == 0 -- genau die Zahl, gegen die die Pin-Zulassung spaeter vergleicht, keine zweite Herleitung -- benannter Stopp Weg2PMambaRetentionZero mit Rechnung, Mindestwert (Sitze x Slots + 1), Referenzwert (Budget 8 wie h91v1) und maximal tragbarer Sitzzahl. Gruppe D und Nicht-weg2 behalten die sgl-project#773-Haltung (Budget 0 wird nur gesagt). mamba_pool_floor.weg2_p_mamba_retention_refusal dokumentiert die 4 Slots je Sitz und die write_back-Frage. Kein Schalter. Tests test_weg2_p_mamba_retention_h113 (5, echter UnifiedRadixCache auf CPU): bb2-Form (P, 8 Sitze, Pool 32) verweigert benannt; h91v1-Form (4/24) und bb2 mit Pool 40 bauen mit Budget 8; D und Nicht-weg2 bauen mit Budget 0; 4/3/2 Slots je Sitz fuer write_back / write_through+Reorder / +Ack-Release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.