Repository navigation
misc: correct the int data type for token ids and indices - #969
Merged
Merged
Conversation
hnyls2002
approved these changes
Aug 7, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
Both PP void sites released their non-resident requests with `_release_dynamic_chunk_probe`, which hands back KV rows, the mamba slot and the req-pool row by calling the allocator and the pool DIRECTLY. Nothing on that route reaches `cache_finished_req`, and that is where `dec_lock_ref(req.last_node)` lives -- the last two lines of every implementation of it. So the pages went back to the allocator while the radix tree kept its lock on them, and `reset_for_retract` cleared `last_node` one line later, putting the ref out of reach for the life of the process. One leaked lock per voided request. WHY IT EXPLAINS THE BOOTS. `evictable_size_` counts UNLOCKED tokens, so a tree whose nodes are all pinned reports nothing evictable at all. Every funding rung reads its reclaim off that number: "reclaimed 0 MiB from [nothing]" against a full tree is not a pressure reading, it is the statement that nothing in it was ever unlocked (sgl-project#813/sgl-project#694 family). NOT A THIRD MECHANISM. `release_req` (schedule_batch.py) already IS this discipline and both correct retraction paths -- `retract_decode` and `retract_all` -- are its only callers. The void sites join them through `_release_voided_request`, which holds no release logic of its own: it supplies the scheduler's collaborators and this path's never-raise contract, and delegates. Because `release_req` calls `reset_for_retract` as its last act, the call sites no longer reset themselves -- doing both double-counts `retraction_count`, which feeds `retract_decode`'s solo-OOM abort ladder. THE PROBE KEEPS ITS OWN RELEASE, deliberately. The profiler's request is a bare `Req` that is never matched against the tree and holds no ref, and `UnifiedRadixCache.cache_finished_req` decrements `req.last_node` with no None guard. Routing it through the disciplined path would be a decrement with no matching increment -- the same accounting defect mirrored (sgl-project#929). Two premises, not one discipline written twice; both docstrings now say so. SIBLING, SWEPT AND FIXED IN THE SAME PASS. `DecodeKVCacheOffloadManager._release_finished_req` ended with `tree_cache.protected_size_ -= len(req.prefix_indices)`, a hand-rolled imitation of one of the four effects `dec_lock_ref` has per node on the path to root; `lock_ref` was never touched, so the same leak by another route. Reachable with --disaggregation-decode-enable-radix-cache alongside the offload flag. Fixed rather than filed because it is decidable by inspection: under the `ChunkCache` default the old line was wrong in the OTHER direction, since that cache's `inc_lock_ref` never raises the counter it subtracted from. TESTS. test_pp_void_lock_ref_969.py, new, 14 tests + 2 subtests. Red-first, PP void, before the fix: 7 failed / 6 passed -- `lock_ref` stayed 1, zero `dec_lock_ref` calls reached the node. After: all green. Red-first, sibling, with the fix reverted: 2 failed on the lock assertion itself (the fake carries `protected_size_`/`evictable_size_` so the old code RUNS instead of raising -- a red for a thin harness is indistinguishable from a red for a broken product). Gegenrichtung: `test_the_underflow_detector_can_actually_fail` drives a doubled decrement at the detector and requires it to fire, so the two one-decrement-per-request assertions cannot go green through a sgl-project#929 underflow. Source pins run through `ast.unparse`, not raw `getsource`: both void sites NAME the probe helper in prose that stays true after the fix, and a pin that reads the comments is not a pin. test_pp_void_chunked_retracted_798.py: its stub list gains `_release_voided_request`. The stub RESETS the request, because the real one does; a no-op stub would have silently removed the precondition that file's first test asserts. DESK GATE (scripts/gate_tier2_partitioned.py, CUDA_VISIBLE_DEVICES="") BEFORE 4849 passed, 2 genuine (test_collective_family_siblings_610.py x2) 619.34 s AFTER 4863 passed, 2 genuine (the same two) 638.07 s delta +14 = exactly the new module. Count probe: 2 named == 2 in summary, SUBFAILED and ERRORS included, both zero. Lane shift explained, not waved past: wide 3701 -> 3695, serial 946 -> 966. Editing test_798 invalidates its sha256 proof in scripts/gate_partition.tsv, so the runner demoted it to the serial track, where it was measured green -- the table's expiry mechanism working as designed. The new module has no row and was measured in serial for the same reason. Re-proving both rows needs a full serial run plus per-module solo logs; NOT done here, and named as open. ruff: 2 F841 in the touched files, both pre-existing at HEAD (`inc_len`, `carries_flip_arm`), none introduced. codespell clean. NOT CLAIMED: no boot, no metal, no effect proof. The acceptance marker for the window is in TICKET_961_WINDOW.md 8.4 -- evictable_size() sampled right after each void line, under load, printed unconditionally including zero.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…d becomes symmetric
Root (R7, boot 6): every void treated the ranks differently. Followers
obey the family's own doctrine ('A VoID IS A PARK, NOT A RETRACTION')
and keep prefix+pages; rank 0's batch members went through
_release_voided_request -> release_req -> reset_for_retract, which
clears prefix_indices and returns the pages. The two ends of the ring
then disagree about the same request every cycle: 169 told=0 refusals
(a FULL re-prefill of computed prefix per voided rid per cycle -- a
standing double-prefill-law violation on every void), 333 past-fill
refusals (total-token mismatch downstream of the same state), 6
duplicate-rid decisions (dedup-free tail requeue), rotating membership
that killed boots 3-6 after every earlier layer was closed.
Fix at both void release loops: pp_park_voided_batch_member -- the
member is queued UNRESET (prefix_indices, prefix KV pages, req slot,
retraction_count survive); only THIS pass's admission-side increments
are handed back: the prepared-never-run chunk via
_park_chunked_prefill_chunk(pass_allocated=True) and the admission
tree-lock ref via pp_give_back_admission_lock_ref. Dedup behind
pp_void_keeps_request (ordering load-bearing: reachability scans
running_batch, a different slot than running_mbs[mb_id]).
_release_voided_request keeps one caller: already-FINISHED members.
retract_decode/retract_all untouched -- their passes RAN.
Briefing premise falsified by the builder, recorded: keeping the
last_node lock ref would have REOPENED sgl-project#969 by a new route --
init_next_round_input re-matches with no dec_lock_ref anywhere on that
path, then admission takes a second ref. Pages-versus-claim is the
boundary, not keep-both: the prefix pages stay in the tree, the
request's claim is returned, which is the ordinary state of a queued
request. Both the wrong draft and the correction are in the docstring.
Instruments: 'sgl-project#984 VOID-PARK' per member (rid, prefix, kept_pages,
chunk_given_back, lock_ref_returned, slot, route) + per-loop census
INCLUDING the zero case, outside the sgl-project#801-spin suppression; #791b/#797d
now also print parked=/dup-skipped=. Boot 6 measured #791b 514x but
#797d 0x -- only the void-output loop is metal-proven; the census
closes exactly that blind spot next boot.
Execution proof (standing order): the rewired void-output loop IS boot
6's hot path (514 traversals); import smoke green; cold check driving
the REAL _pp_void_own_batch body: 23/23 (prefix survives, requeue,
dedup, one free = never-run chunk only, one lock-ref give-back, no
double free, finished-member path intact). Suites live in the test-agent
lane; predicted red there: test_pp_void_lock_ref_969.py encodes the
retract premise this commit retires -- its invariant (no leaked ref)
is preserved via the give-back.
Named posten, not bundled: sibling unreturned admission refs on the
#968b route and the kept chunked member; _pending_prefill_tokens_for
over-reports a parked member's backlog (honest figure: total -
len(prefix_indices)); req-pool kv_committed_len discrepancy duration now
unbounded for queued members (worth a test); sgl-project#963 prefix-floor wiring
stays its own posten.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…invariant is re-encoded, and a dead can-fail is revived sgl-project#984 makes the void PARK rank 0's members instead of retracting them. Three arms of the 969 suite encoded the old premise. None is deleted; each says in place what changed and why the thing it was protecting still holds. FIRST, THE FAILURE WAS NOT THE PREDICTED ONE. At 8be86f5 all six reds were 'TypeError: lambda got an unexpected keyword argument pass_allocated' -- the harness stub for _park_chunked_prefill_chunk taking two positional args while the fix passes a kwarg. A mechanical signature drift that MASKED the real result: with the stub widened, the truth is 3 failed / 11 passed, and the three are the premise, not the invariant. test_the_request_is_retracted_exactly_once: asserted is_retracted and retraction_count == 1. Now asserts NOT retracted and count 0. What the arm protected is unchanged -- retraction_count feeds the solo-OOM abort ladder in retract_decode, and a request must not be aborted at half the configured retractions. A double count was the hazard when the void retracted; an unasked count is the hazard now. test_the_release_happens_before_the_reset -> renamed test_the_ref_goes_back_while_the_prefix_handle_is_KEPT. Its assertIsNone( req.last_node) encoded an ordering problem of a world where reset_for_retract cleared the handle. sgl-project#984 removes that world and KEEPS last_node on purpose. The invariant is untouched and is now what the arm states: the admission-side inc_lock_ref must not outlive a pass that never ran (exactly one give-back -- not zero, a leak; not two, the sgl-project#929 underflow), while the prefix handle survives so the next offer can report it as executed. test_the_underflow_detector_can_actually_fail: its mutant patched tree_cache.cache_finished_req, reached via release_req. sgl-project#984 does not take that path, so the injected fault could no longer occur at all -- the arm failed 0 != -1 because the mutant was never invoked. A can-fail whose mutant became unreachable reports nothing while looking like a red someone would 'fix' by deleting the assertion. Re-aimed at pp_give_back_admission_lock_ref, which is module-level for exactly this purpose by its own docstring. MEASURED, same file: 646f410 (pre-sgl-project#984): 3 failed, 11 passed 8be86f5 (post) : 14 passed (14 defs, count probe) Two of the three reds are behavioural -- 'None is not <node>' (the reset cleared the handle) and 'True is not false' (the void retracted). The third is an AttributeError on the symbol sgl-project#984 introduces, which is inherent to a can-fail arm aimed at that symbol and is reported as the weaker red it is.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
… and the mamba park fence sgl-project#990's ownership guard is right in intent and Boot 10 confirmed it on metal (no underflow). It discriminates the wrong thing: 'is still in the field right now' instead of 'will keep the field'. MEASURED at b27d7c2 by instrumenting the give-back's caller: member IS current chunked_req -> pp_queue_orphaned_chunked_req, guard fires, lock_ref stays 1, request sits in waiting_queue with its admission ref ordinary member -> pp_park_voided_batch_member, guard does not fire, lock_ref -> 0 So in these two shapes the guard's only observed firing is the harmful one. Cause is a statement order: the re-home runs BEFORE the field is overwritten (:7909/:7913, :9261/:9265) and the comment there requires that order for #968b -- 'Re-homed BEFORE the overwrite -- after it, the reference is already gone'. Re-admission then takes a fresh ref: +1 per void cycle, prefix never evictable, sgl-project#969's leak by a new route. Recorded as unittest.expectedFailure, not inverted and not deleted. Inverting would encode the leak as correct; deleting loses the only executable record. When ownership is asked properly this becomes an UNEXPECTED SUCCESS, which is a loud self-clearing signal to drop the marker. The suite stays green. The rid-discriminator arm is withdrawn rather than shipped half-understood: its shape is entangled with the re-home and needs its own study. MAMBA (boot 10's next form, mamba_pool_idx=None at cache_unfinished_req): measured that the void-PARK does NOT release the slot, and fenced that, plus the park's prefix contract beside it so the two cannot be traded off. What produced the None is NOT established and the file says so in an executable arm rather than guessing between the two candidates -- the release route via reset_for_retract, or a re-admission that never re-acquires. Guessing there is exactly this window's recurring class. One of my own arms cited the wrong function for the clearer docstring and went red; the citation is corrected in place with a note, since an arm citing the wrong function is that same class and was caught only by running it. Test: 969 -> 16 defs, 15 passed 1 xfailed; 990b -> 4 defs, 4 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…request Boot 12 (7b855f6) died on PP0 at 21:46:06 with `AssertionError: reusing request must be chunked or have committed KV` (memory_pool.py:395), via get_new_batch_prefill -> prepare_for_extend -> alloc_for_extend -> alloc_req_slots -> HybridReqToTokenPool.alloc. ROOT, from the ledger and not from inference. One second earlier the same rank logged, for the same rid: sgl-project#969 voided-request release failed for 5708abdd579e4f7097ab97ed796481ea: Committed KV cache already freed (self.kv_committed_len=4096) `_release_voided_request`'s never-raise contract caught that, logged it, and fell through to `reset_for_retract`. The contract is right -- an instrument that raises while cleaning up after a divergence turns one defect into two -- but it only decided that this frame does not PROPAGATE the failure. It never decided what the half-released request IS afterwards. The answer was: a request still holding `req_pool_idx`, whose `inflight_middle_chunks` and `kv_committed_len` the reset then zeroes -- and those two fields are the allocator's only evidence that a retained row is legitimate. It was re-queued, re-admitted one second later, and killed rank 0. The assert is correct and is the only thing that caught this. The defect is upstream of it. CLASS: warn-then-continue. The compensator converts a partial failure into a live, corrupt, re-admissible object. This is the catalogued `warn-then-continue swallow` shape, and the cure is not to remove the swallow but to make the post-failure STATE explicit. FIX, at the junction: on a failed release, hand the row back -- mamba slot first, then the req-pool row, the ordering `_release_dynamic_chunk_probe` already documents. The reset below drops prefix, pages and geometry anyway, so the request re-prefills regardless; the row it keeps buys nothing and justifies nothing. `free_slot` carries its own membership scan and REFUSES an already-free row (sgl-project#616), so a release that failed AFTER returning the row is caught by name and logged instead of corrupting the free list -- both outcomes are reported with the rid. NOT CLOSED, and deliberately not guessed at: why `pop_committed_kv_cache` found `kv_committed_freed` already True at `kv_committed_len=4096`. That producer is a second defect. This change contains it and names it in the log (`sgl-project#993 INCOMPLETE RELEASE DISOWNED`), so the next boot measures how often a release fails at all -- a number no boot of this family has ever had. FUTURE CHECK: any new raise inside `release_req` now ends in a disowned row plus a named line, not in an allocator assert three passes later. Evidence: desk. py_compile + import smoke + wiring assertion on `_release_voided_request`. Belegstufe: DESK-BEWIESEN.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…tay sticky until spent Boot boot_855_968pp0lb_..._0831_060725 ran clean -- 0 exceptions, 0 aborts, 0 proxy mismatches, sgl-project#969 EXTENT rank-uniform on all three ranks, so the replacement gate is safe -- but delivered 0 applied load-backs against 12 opportunities. Two defects in the carry, both mine, both measured: 1. init_pp_loop_state cleared BOTH halves at every cutover, by analogy with the pass-scoped voids beside it. That analogy is wrong for this quantity: the voids describe THIS EPOCH'S PIPELINE, while a load-back extent describes the HOST TIER, which is exactly what survives a cutover. And the post-cutover re-admission IS the read-through path, so the clear destroyed the fact with the very event that creates the only chance to use it. Measured: 27 cutovers, and every one of the 12 deferrals fell 1-2 s after a CUTOVER line (06:14:07, 06:16:57, 06:18:24, 06:19:59). The same rid's host hit persisted and GREW across those cutovers (1215 -> 1216 -> 2114), which is the direct evidence that the tier the number describes is durable. 2. The promote replaced `effective` with `pending`, giving every extent a life of exactly ONE pass, while the gap between PP0's offer and the group's next re-admission spans a cutover by construction. A one-pass fact and a next-cutover consumer cannot meet. Fix: `effective` is sticky -- merged, not replaced, and not cleared by a cutover -- and is spent where it is acted on (the load-back site flags the request, the admission loop pops the rid), which bounds the map without it needing to know when a request ends. Same shape as pp_clear_parked_continuation. Staleness is caught where it is cheap: a rank told an extent its host tier no longer covers raises sgl-project#968 LOAD-BACK EXTENT UNHONOURABLE, an epoch-independent check against the tier the number is about. Only `pending` remains per-pass; that is what still keeps PP0 from applying its own offer on the pass it makes it. Desk checks: imports; cutover reset now guarded; promote merges and clears pending; spend set at application and popped by caller; extent helper regression (honest-miss / told / pp_size<=1) unchanged.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…ot end a pass PP0 launched `_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's pass ran nowhere. Its own docstring named the return trip that made that safe -- "the void output carries the observed local match home, and PP0's guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a). The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed set while the last rank's did not, so no output was ever sent and PP0 blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s own comment relies on ("sender and receiver ask one question of one batch"). Measured, twice, and the second is on the stall second itself: 1068cap 07:34:02-09 sgl-project#797 void on rank 1 ONLY (no void/retract line on PP0 or PP2); PP0 parked, PP2 spinning. 1069cohort 08:00:54/55 sgl-project#791 unhonourable on PP1, told=12493 local=8397 then told=13399 local=12493 -- `local` exactly one pass behind `told`. After 08:00:55 only ranks 1 and 2 emit at all (slot_occupant / output_fill / width_agreement run to the takedown at 08:13:32, 4177 and 7973 lines; rank 0 emits nothing). That last measurement also refutes the occupant-sleep node as the halting member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F made permanently None. The only halting member is PP0's unbounded output receive. Repairing the return trip would repair a compensation layer for a rank-local verdict, which is the arc the sgl-project#968 order forbids continuing; under upstream-minimal the repair carries the burden of proof and the deletion does not. So the verdict is deleted and the disagreement is DETECTED instead (RAENGE-NIE-UNEINS: a detected divergence stops the group, never a compensating wait): * `_pp_assert_told_honourable` replaces it -- an unhonourable told names rank, slot, rid, told and local and raises. No clamp: clamping to this rank's own local match is rank-local geometry, i.e. sgl-project#631. * The chain-receive throttle arm gets a horizon (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named stop. Taking the arm is legitimate and frequent; outliving it never is. * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to` already was (runaway guard + reported counter), and the launcher now sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed since 2026-08-24 and shipped disabled by default, which is why it never fired in either stall. (A-i) is WITHDRAWN rather than built: its counter proof is blind at that site (the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is itself compensation for the rank-local verdict this commit deletes, so its trigger is removed at the source. Its two red-first tests are replaced by zombie tests for the deletion. Desk evidence: hermetic import + AST (deleted verdict absent, watchman and horizon present, recv bounded); test_968_deletion_falsifiers 24 passed, 1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built, which stays red until the relay SYMBOLS are swept out too. The relay is already unreachable at runtime (no rank originates a void any more, and `_pp_absorb_void_output` has no caller in production), so that sweep is a dead-code deletion scheduled beside this commit, not a runtime dependency of it. Belegstufe: DESK-BEWIESEN. Boot pending.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 1, 2026
…sends, nobody relays, one rank absorbs sgl-project#969 CUT V deleted the void-output EMITTER on the premise that the batch is the verdict ("sender and receiver ask one question of one batch"). What it left behind was the rest of the arc, and the arc has been dead ever since: _PP_VOID_OUTPUT_KEY: True constructed at exactly ONE site -- inside _pp_absorb_void_output, i.e. only when re-forwarding a void already received. Zero originating senders. _pp_void_forward_payload 1 writer, 0 readers. Computed and never sent. _pp_absorb_void_output 0 call sites (ast over python/sglang). pp_void_forward_payload called only from the absorber. pp_void_relay_stop_rank called only from the forward payload. pp_void_relay_launched_verdict same. pp_first_retracting_rank called only from the forward payload. A closed orphan subtree, whose own docstring states the sgl-project#801 relay invariant it can no longer keep: "a non-last rank that took exactly one message off this wire for a ring generation must put exactly one back on it, void included". WHY DELETED AND NOT WIRED. Wiring it back means rebuilding the second bookkeeping sgl-project#969 CUT V removed, and under the upstream-minimal law (user order 2026-08-29) repair carries the burden of proof that upstream semantics cannot carry the goal. It can: PP0 owns the verdict and the travelling batch is the verdict. Half a relay is the worst of the three states -- PP0 parks on an output the mid-rank void swallowed and its successor throttles on an occupant nobody will speak for, which is the shape both measured stalls took (1068cap 07:34:09 and 1069cohort 08:00:55, roles swapped, same halting node). Tests of the deleted mechanism go with it rather than being patched green: the five files whose SUBJECT is the void relay (sgl-project#797 retracted-pass void, sgl-project#798 chunked-retracted void slot, both sgl-project#801 relay files, #791b output-ring retraction), plus test_pp_continuation_cross_slot_rehome_968b (28 of its 30 tests drive the deleted absorber transitively) and test_pp_void_lock_ref_969 (its harness is the deleted method). Files that merely mentioned it keep their pins; only the individual test that exercised the deleted site is removed. Comments naming a deleted symbol are rewritten rather than left to send the next reader after a ghost. Desk-proven: py_compile + import smoke (scheduler and mixin), the sgl-project#968 deletion falsifiers 25/25 with the wired-or-deleted-but-never-half-built contract now green by the deletion arm, ruff unchanged at 104 pre-existing findings on both touched files (baseline taken from HEAD under the same config).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…oved, not the set the release re-enumerates
Boot 9 (boot_855_weg1b9_1116175f6d_0904_164023.log) died at the first
pp_to_tp cutover. The measured shape is TEMPORAL, not container-shaped:
log:1883/:1886/:1888 at-arm cur_slot_reqs=1 on ALL THREE ranks
(cur_slot_reqs IS len(_live_reqs(scheduler)))
log:2203/:2210/:2213 one second later, RESIDENTS RELEASED 1 / 0 / 0
log:2189/:2194 PP1+PP2 sgl-project#938 PROTECTED RESIDUE AT DROP: 67 row(s)
still locked after a drop that evicted 0
log:2305/:2351 ReqPoolRebindRefused: 1 of 8 rows are still held in
the OUTGOING request pool (free=7, rids=[], rows=[])
The same authority, the same rank, two answers one second apart, and
nothing reconciled them. Four cuts:
1. LOAD-BEARING. `_enter_armed_state` now opens an armed-window resident
ledger (`_armed_residents`) and `on_round` unions each armed round into
it; `_release_residents_for_cutover` retracts the release-instant
enumeration RECONCILED against that ledger (`cutover_resident_set`).
Snapshot rather than a re-run of the quiescence term at :11564, because
that point is past the no-return -- PP0's PROCEED has already ridden the
request stream -- and a quiescence term there could only be a rank-local
verdict where no rank may hold one (sgl-project#969 SS-W3). A snapshot changes WHAT
is retracted, never WHETHER the group cuts over.
The reconciliation is a FILTER, not a union: a snapshot member is carried
only if it names a row, the row is not in the pool's free list, and no
live request names the same row. Under-retraction stops the boot loudly
at the rebind; over-retraction frees a row its owner still holds and
corrupts silently, so the two are not traded symmetrically.
2. `mbs` is now a route of the ONE authority `_live_reqs`, beside
`running_mbs`/`last_mbs` (scheduler_pp_mixin.py:7556-7561 builds three
distinct arrays; `mbs` is written on all three planning paths). Dedup is
by id(), so the scheduler.py:8354-8362 alias costs nothing.
3. `phase_req_pool_binding` drops its own two-container walk and delegates
naming to that authority (keeping `waiting_queue`, which the flip has
not got), and `census_outgoing_req_pool` now derives the HELD ROW IDS
from the pool's own free list. `escapees=1 rids=[] rows=[]` becomes
`rows=[N] rids=[...] unnamed=K`. The raise is NOT relaxed -- the module
docstring :16-30 is right that a request pool has no disarmed state.
4. phase_flip_runtime.py:1202 cited `orphan_resident_reqs` in
`phase_flip_resident_carry.py` in the present tense; sgl-project#969 deleted that
module whole. Rewritten in the past tense, naming the deletion.
RED FIRST, at parent 1116175, test/registered/unit/managers/
test_1202_arm_release_residency_reconcile.py: 8 failed, 2 passed.
The follower test failed with boot 9's verbatim refusal
("1 of 8 rows are still held ... free=7, rids=[], rows=[]").
After the cut: 10 passed.
MUTANTS (scratch, all killed):
M1 drop the reallocation guard (DANGEROUS: frees a row its new owner
holds) -> test_a_row_reallocated_to_a_live_request_is_not_taken_from_it
M2 drop the already-free guard (DANGEROUS: returns the same row twice)
-> test_a_row_already_returned_is_not_retracted_a_second_time
M3 release ignores the armed ledger (boot 9 restored) -> 4 tests
M4 remove the mbs route -> test_mbs_is_a_route_of_the_authority
M5 census rows from the request list again
-> test_held_rows_come_from_the_pools_own_arithmetic
VERIFIED: siblings at parity, per file, parent vs cut --
test_phase_flip_runtime.py 30 failed / 38 passed at BOTH, identical failure
set (pre-existing); test_1040_req_pool_per_phase + test_seam_order_856 +
test_seam_readmission_w31 + test_tree_drop_returns_rows_856: 78 passed at
both. ruff check clean on both sources at parent and at cut; ruff-format
diff 4 lines at parent and 4 at cut (pre-existing, unchanged). Matched
structural check: all three `self._parked_extent = None` statements sweep
the ledger too (the sgl-project#746 M5 shape on the request axis). Import smoke on
both modules.
NOT VERIFIED: nothing was run on metal. No boot, no GPU, no server. Whether
the request that held the row on PP1/PP2 in boot 9 was reachable through
`mbs` is still PLAUSIBLE only -- an mbs-only resident would have read
cur_slot_reqs=0 at arm and the log reads 1. Cut 1 does not depend on that
attribution. test_quiescence_no_carry_858.py could not be run: it fails to
import at the parent commit too (`cannot import name
prefill_runnable_in_current_layout`), a pre-existing breakage.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…eft the nodes naming them ROOT, file:line: python/sglang/srt/mem_cache/unified_radix_cache.py:6069-6250 (UnifiedRadixCache.reclaim_rows_for_drop). The pass walks every node and returns the rows and mamba slots the tree still holds -- the sgl-project#1050 contract for locked nodes `evict` refuses -- but it never took the reference away from the node. Every other release of a node-held anchor goes through MambaComponent.evict_component (mamba_component.py:653-663), which frees AND nulls cd.value AND corrects the size book; this pass had only the first third. THE SIBLING FORM, and why sgl-project#924's own guard was silent: nothing is freed twice. The slot is returned ONCE and the node goes on naming it, so MambaSlotAllocator._refuse_double_free (allocator/mamba.py) has nothing to see (free_list_duplicates=0 in the killer line). The free list and the tree then both own the slot: alloc() hands it to the next request while the tree offers it as a resume anchor -- one GDN state read by two requests. MEASURED (boot 10, /spinning/evidence-665-f1/boot_855_weg1b10_2126a4a1d2_0904_211702.log 21:29:28Z, all three ranks, line 364223 for the census): [mamba] total=20, available=20, evictable=4, withheld=0, double_owned_src=live, free_list_duplicates=0, duplicate_slot_ids=None, free_and_cached=4 TREE CENSUS nodes=5 | MAMBA: tracked_evictable=4 recomputed_evictable=4 DESK REPRODUCTION of that exact line against the real function (two requests through cache_unfinished_req + cache_finished_req, then reclaim_rows_for_drop): before: avail=20 evictable=4 free_and_cached=[1,2,3,4] duplicates=0 after: avail=20 evictable=0 free_and_cached=[] CLASS: "a component's rows are released by a walker over tree nodes instead of through the component's own owner-transfer primitive." SIBLING SWEEP: the FULL half of the same function has the identical omission (_free_full leaves node.component_data[FULL].value standing); fixed and pinned in the same test. FUTURE CHECK: after any pass that returns rows the tree held, free_set & tree_set must be empty for every component -- asserted directly. WHAT CHANGED * unified_radix_cache.py: nodes are carried beside their values; _disown_reclaimed_value(node, ct) is the factored-out other two thirds of evict_component (null the value, debit protected OR evictable per lock_ref, leave the device LRU, join the host LRU when a host copy survives). The FREE stays where it was, differenced against the allocator ledger (sgl-project#1055), because this pass legitimately meets already-free slots; only the DISOWN is new. An already-free slot is now disowned too -- that is the aliasing, not a reason to leave the reference. * invariant_checker.py (_check_mamba_pool): NAMED STOP "sgl-project#924 MAMBA SLOT ALIASING". A negative mamba occupancy (pool.size - available - evictable < 0) or a free list longer than the pool is now a leak verdict with both numbers. Boot 10 printed `mamba usage: -0.10 ... -0.30` five times over eight minutes (first 21:21:20Z, log:3595) and nothing acted on it. Derived at on_idle, on every rank with nothing in flight -- never from a batch line inside the no-return region between collectives (sgl-project#969 SS W3). * allocator/mamba.py: note_924d(), the "#924D" per-(rid, station) trail that decides sgl-project#1190 on boot 11. Stations: alloc (memory_pool.alloc), backup (BACKUP_HOST build), load_back (H->D COW target, with the node's own anchor printed beside it), first_state (the deferred COW execution, with cow_src), free / relinquish (memory_pool). Bounded by construction: one line per (rid, station), global cap 8192, and the cap prints its own suppressed count so a missing rid is never read as "that station was not reached". Line format: #924D station=<s> rid=<rid[:12]> mamba_slot=[<ids>] node=<node id> <extra> If a B probe's first_state slot equals a slot some node still names, sgl-project#1190 IS this defect; if not, sgl-project#1190 is a separate carrier and stays open. EVIDENCE * RED on the parent (2126a4a, scratch tree /tmp/parent924 from git archive): 9 failed / 1 passed. The 1 pass is the can-fail control (test_a_balanced_pool_is_not_a_verdict), which must stay green on both sides. * GREEN after: 10 passed. * MUTANTS, all killed (each against a copy of the source tree): M1 drop the mamba disown (aliased slot survives) -> 4 failed M2 STOP threshold unreachable (STOP cannot fire) -> 2 failed M3 disown without the size-book debit (silent drift) -> 2 failed M4 drop the FULL-half disown (sibling sweep) -> 3 failed M5 discriminator per-call instead of per-rid -> 1 failed * SUITES, the 63 registered unit files touching the changed symbols, run once on each side, hermetic (CUDA_VISIBLE_DEVICES=""): parent 78 failed, 1612 passed, 860 skipped, 195 subtests passed worktree 78 failed, 1622 passed, 860 skipped, 195 subtests passed The failure NAME SETS are byte-identical (diff clean); the +10 is this commit's new file. Zaehlprobe: 78 extracted FAILED/SUBFAIL/ERROR names == 78 in the summary, on both sides. * ruff: new test file clean; the six modified modules carry exactly their parent error counts (43 / 2 / 0 / 0 / 0 / 1), i.e. nothing added. NOT PROVEN: this is DESK-PROVEN, not fixed. The desk shows the mechanism and the exact ledger line; only boot 11 can say whether it was the whole of the boot-10 trajectory, and the #924D trail is what will answer that and sgl-project#1190.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 12, 2026
…dules, decided per file WHY: four modules could not be COLLECTED on this tree, so EVERY gate baseline here ended rc=3 / VERDICT: INCONCLUSIVE -- an automated NEW=0/GONE=0 verdict was structurally impossible, which cost half a verdict cycle on 2026-09-11. The exit code is not the tree's redness: gate_tier2_partitioned.py:614-616 is `if inconclusive and rc == 0: rc = 3`. NOT A SKIP (sgl-project#910): a skip can darken a working test; here it darkens nothing -- there is nothing behind these imports to run -- so it would only make rc=3 permanent under a friendlier name. sgl-project#905's form instead: delete a retired promise with its reason, and leave ONE tripwire that fires the day the mechanism returns and NAMES the arms then owed. DECIDED PER FILE, not in bulk: DELETED test_phase_flip_decode_relay_631.py (7 tests). Its only subject was `phase_flip_resident_carry.reseed_decode_input_relay`, deleted by 069f98c "[sgl-project#969 CUT K] Delete the resident carry: the cutover is a re-entry, not object surgery" -- also standing user design (a flip nulls everything and re-admits through HiCache; object surgery at the cutover is the root class of ~15 boot killers). A retired promise. DELETED test_pp_proxy_stamp_631.py. It bound `SchedulerPPMixin.pp_flip_drain_tensor_dicts` at CLASS-BODY level (:334), which is why it failed at import while test_pp_presence_stash_on_real_wire_800.py, which names the same symbol inside a function, still collects. THE RENAME HYPOTHESIS IS REFUTED: `pp_flip_drain_leftover_dicts` is not the same mechanism under a new name. Its own docstring calls it "CORPSE S DONE CORRECTLY" and records that the old drain was kind-blind and ATE AN OWED OUTPUT (PP1, 07:33:30Z), while the successor demultiplexes first; HANDOFF_656.md:1348 says "Do not re-enable pp_flip_drain_tensor_dicts as written". The promise it carried is held by the successor, which six live test modules already cover, so retiring this file loses no coverage. REWROTE test_phase_flip_spec_seam_631.py: dropped the dead `harvest_resident_batches` import and the THREE tests that drove it, two of them sgl-project#905-shape falsifiers whose counterfactual can no longer be constructed. The seven tests over `_reachable_batches` and `clear_spec_info_for_unspeculated_phase` -- both live -- survive unchanged. REWROTE test_phase_flip_draft_bootstrap_631.py, three separate repairs: (a) dropped the dead `ResidentCarryError` import; (b) retired the cap-refusal test and the two-ceilings-agree pin. The product states the reason itself (phase_flip_draft_bootstrap.py:456-461): the defect-M ceiling is gone with the carry, "a count above the cap can no longer be the signature of a corrupted carried set -- it would just be the load", and BOTH copies of IN_FLIGHT_CHUNKED_ALLOWANCE are gone, so there is no second ceiling to agree with; (c) FIXED FIVE STALE-DOUBLE FAILURES that only appeared once the module could be collected again: `retune_carried_batches_for_phase` enumerates through `_harvest(scheduler)` -> `scheduler._resident_batches()`, which sgl-project#969 moved ONTO the Scheduler (scheduler.py:7040, "eight lines replacing harvest_resident_batches ... 911 LOC"), and this file's SimpleNamespace double predates that move. The double now BINDS THE PRODUCT'S OWN METHOD rather than copying its eight lines: the identity dedupe is load-bearing (running_batch is normally an alias of one slot) and a second copy is free to drift. NO production file was touched -- `_resident_batches` exists and has live callers (phase_flip_runtime.py:941/3765), so this was test drift, not a product defect, and I verified that before editing anything. TRIPWIRE test_1347_retired_631_mechanisms_stay_retired.py, three assertions over the retired mechanisms, each naming the arms owed on red and where to recover them (04cd920). CAN-FAIL PROVEN, four mutants, all fire: re-importable resident carry -> red; the kind-blind drain back on the mixin -> red; the SUCCESSOR also gone -> red (so the file cannot pass by the whole family vanishing, the green-by-vacancy direction); the arming allowance back -> red. MEASURED: 47 tests collect and 47 pass across the two rewritten modules plus the tripwire (before: 0 collected, 4 modules uncollectable, 11 pytest ERROR entries). Running them was not optional -- collection alone would have traded rc=3 for five new GENUINE names.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 24, 2026
… x149 (142 MiB statt Bild 70); W132 verweigert x149, Kante bei 16384 0.304/0.544/0.808 (fnFL2 H41b) fnFL2x149 (74c06de, FR_P 0.351,0.642,0.39 -> H25 0.733887) passierte die erste P-Karte (PP0 Kopfraum 408 gerechnet, PASST) und starb im ZWEITEN 16k-Chunk (sgl-project#969 EXTENT 16384..32768) im GDN-Extend (chunk_delta_h k.new_empty): "Tried to allocate 384.00 MiB ... 292.81 MiB is free ... 26.28 GiB is allocated by PyTorch ... 290.04 MiB is reserved by PyTorch but unallocated". Bilanz gegen das Modell, beide Richtungen: - Chunk 0 lag auf dem Zeilenbild: Kopfraum 1300 gemessen gegen 4594 (x146) - 46 Zeilen x 29 x 2.417 - 73 privat_frei = 1297; Transiente 3.58 GiB wie bei 166 Zeilen (keine groessere Chunk-0-Transiente, (b) nein fuer Chunk 0). - Chunk 1: gegen x146 (Kopfraum 4273 @chunk 1) fehlten 6541 MiB statt 3224: torch-Bedarf 27295 MiB (belegt + Anforderung) gegen erwartete Spitze 25505 (+1790, (a) ja: nicht 321 MiB je Chunk), und cap (frei + reserviert) fiel 28951 -> 27494 (-1457, Speicher ausserhalb des Allokators; x146/x147/ x148 halten cap ueber alle Chunks auf +-12 MiB). Zusammen ~ein zweites Zeilenbild; der Mechanismus ist nicht identifiziert. - (c) NEAR_OOM 400 ist nicht der Hebel: der Tod liegt 2268 MiB unter null. - (d) privat_frei 2467 statt 2394 (+73) geht direkt in die Todesschranke. Fix (p_card_chunk): der Tod ist ein Messpunkt. death_from_log liest den OOM-Text (Stufe aus 'Scheduler hit an exception', Chunk-Index aus sgl-project#969 EXTENT, privat_frei aus WEG2-GRAPH-POOL) als obere Kopfraum-Schranke; row_card_cost_from_deaths misst daraus je Stufe den Kartenpreis einer Zeile ueber der Referenz (untere Schranke): PP0 142.2 MiB = (4273 + 2268) / 46 (Bild 70.1); PP1/PP2 ohne eigenen Tod erben das Verhaeltnis 2.029 (53.9 / 39.2 MiB, als Hochrechnung benannt). Launcher: --p-card-death-logs. Verdikte: x145/x146-Form PASST (3632/3006/1914 MiB); x149 W132 (PP0 -2909, PP1 -2276); x121/x122 W132. KARTEN-DECKE bei Chunk 16384: 0.304 (188 Zeilen) / 0.544 (311) / 0.808 (446). Tests: x149-Tod aus der eigenen OOM-Zeile, Chunk 0 auf dem Zeilenbild, Zeilenpreis ~2 Bilder, x149 verweigert (Modul und Launcher-Naht), Mutant mit Zeilenbild laesst x149 durch, Kante 188/189 Zeilen; Fixtures um sgl-project#969 EXTENT, die Exception-Zeile und fnFL2x149 ergaenzt.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…ller Rows-Formen, LMEM-Zensus am Triton-Loader -- rc9p D-TP0 starb am Local-Memory-Wachstum im ersten Launch der Form 16/1/2 Befund rc9p (dkrnfbar1rc9p09261011, Baum d1c7094, D-Log Z. 64368-64527): 10:34:24 'RuntimeError: Triton Error [CUDA]: out of memory' im Launch von _sparse_attn_rows_fwd (sparse_attn.py:1000 <- _attend_rows :630 <- _forward_extend_impl :1929). Z. 64404: 'QSA-ROWS-LAUNCH arch=sm120 kv=fp8 decode=exp2 cfg=16/1/2 first_total_q=3585' -- das X-Direkt-Extend weg2-22-25 (uncached=3585 auf 23744 Praefix, Z. 64384 sgl-project#969 EXTENT (...,23744,27329,23744,3585)) war der erste Request des Boots mit Praefix UND >512 neuen Zeilen. Alle frueheren X-Admits mit Praefix <= 219 Zeilen; der 1423er war praefixfrei (sgl-project#969 (...,0,1423,0,1423), Z. 13423 -> _sparse_gqa_prefill, STACK 32 B). Wurzel: der L20-Tisch gibt >512 Zeilen (16, 1, 2). cuobjdump -res-usage im Container-Triton-Cache (/spinning/docker-acceptance/nf/triton): die D-Variante (fp8 exp2, USE_COUNTS=False) ist unter dem CUDA-13-Image (rc2.1d 8f1db7a, PTX 9.0, SEVCQXIANJ...) REG 128 / STACK 2320 B je Thread; dieselbe Quelle unter PTX 8.8 (63XNNJEYJA..., 25.09. 08:44) REG 255 / STACK 992 B. Der Kontext hielt 1024 B (H15-Wake 'saved 1024', ganzer Boot) = 255 MiB; der Launch musste auf 2320 B x 170 SM x 1536 = 578 MiB wachsen (gemessen agent09252021, dieselbe Form bei 1153 Zeilen ueberlebt: 'WEG2-SLEEP-LMEM lmem 578->0 MiB ... NVML -580 MiB'), +323 MiB bzw. 578 MiB Neuanlage in cuLaunchKernel, von keinem Posten gebucht; driver_free vor dem Extend 1706.8 MiB (Z. 64349), den Rest nahm das Extend selbst. Gruppe P fuhr die Form nie: NF_ENV_P_FORM setzt SGLANG_FORCE_QSA_ROWS_CONFIG=inf=64/8/2 + FP8_DECODE=ptx, Gruppe D hatte die Zeile nicht. Kein Modul-Unload: der sgl-project#1056-Wrapper feuert je neuer Spezialisierung (rows: 4 Formen = 4 Loads, 12 Zeilen; l2norm_fwd_kernel 17x, T ist tl.constexpr). Fix: (a) sparse_attn._get_rows_config: sm120-Default = L20-Tisch mit NUR dem Eintrag >512 -> (64, 8, 2) (REG 152 / STACK 0 unter PTX 9.0, UZIGNZUEW4... = genau die D-Variante, im Container-Cache). Baender <=512 unveraendert, sm86 unveraendert, Env-Tisch gewinnt weiter. QSA-ROWS-LAUNCH traegt table=env|h101-sm120|l20|h20. (b) qsa/rows_prewarm.py + Scheduler.warm_qsa_rows_forms (nach der Graph- Capture, vor der #603b-Barriere und dem ersten Sleep): jede Rows-Form x USE_COUNTS einmal gestartet, mit der Spezialisierung, die die Capture- Launches aufgezeichnet haben (Heads/HEAD_DIM/dtype/K/Pool-Strides, weak). Zeile 'H101 QSA-ROWS-PREWARM'. Form-A-Worker ueberspringen. (c) utils/lmem_census.py am sgl-project#1056-Chokepoint: nach jedem Cold-Load wird LOCAL_SIZE (n_spills*4) gezaehlt; braucht der Kern mehr Stack als der Kontext haelt, wird er VOR dem ersten Launch per cuCtxSetLimit gebucht ('H101 LMEM-GROW ... action=grown'), bei Verweigerung empty_cache + ein Retry, danach 'H101 LMEM-GROW REFUSED' mit MiB. Der H15-Wake restauriert das Zensus-Maximum (booked=<B>(<kern>), auch ueber der H47-Kappe), die Sleep-Zeile traegt census=<B>(<kern>) (P-Planer-Regex unveraendert). Tests: test_h101_rows_lmem (neu); H58/H65-Erwartung sm120 >512 auf (64,8,2) nachgezogen; H15, 603b, prefill_config_h65 unveraendert.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…alten dieselben Host- und Anker-Eintraege -- gruppeneinheitliche Freigabe-Politik statt fuenf Verdikt-Riegel Wurzel der Klasse rc9i/rc9k/rc9l/rc9m/rc9o: auf NF-D (Form A, --rank-tp-ratio 1,0,0) entschied jeder Rang die Host-Lebensdauer eines Knotens selbst. Store-Ack (_drain_storage_control_queues_impl/_drain_backup): TP0 bindet die Zeilen an die Arena (_weg2_rebind_host_to_arena) und behaelt sie, TP1/TP2 (0-Byte-Plain-Pools) geben sie als Transit frei (_weg2_release_chain_piece_host: HOST-Schicht ALLER Komponenten, Mamba-Anker eingeschlossen) -- rc9m-D-Log: 'WEG2 PUBLISH-CHAIN host released' nur auf TP1/TP2, rc9o ebenso (0/8/8, ratenbegrenzt). Dasselbe nach dem Load-Back (loading_check, nur ohne arena_read). Bei der naechsten Device-Eviction ueberlebt der Knoten auf TP0 als Host-Knoten mit Anker (sgl-project#1469 EVICT ... backuped=True host=True), auf den Workern wird er geloescht (host=False) -> Anker auf verschiedenen Tiefen (rc9o MAMBA-HOST-RESUME 17/13/13). H93/H96/H97/H98/H99 schliessen je ein VERDIKT auf diesen Baeumen; R12 nimmt den Unterschied aus den BAEUMEN. Politik (Variante c + a, Kapazitaet b), neues Modul mem_cache/form_a_host_shadow.py: - (c) Ein Form-A-Worker behaelt seine bytelosen Host-Zeilen, solange TP0 sie haelt: kein Transit-Release am Store-Ack und nach dem Load-Back ('R12 SHADOW-KEEP'), keine eigene Host-Eviction (evict_host -> 0, Aufrufer verweigert benannt, 'R12 SHADOW-OWN-EVICT-REFUSED'). - (a) TP0s EIGENE Host-Entscheidungen reiten auf dem vorhandenen tp<-reqs-Broadcast (request_receiver.recv_requests, Muster sgl-project#969 W3; keine neue Kollektive, kein neuer Bogen): TP0 haengt ein FormAHostVerdict VORN an die Liste, direkt nach dem Broadcast nimmt jeder Rang es heraus und wendet es an, bevor ein Request des Passes (Intake, Zulassung, Votes) den Baum liest. Schluessel = letzter Seiten-Hash + End-Tiefe (inhaltlich, nicht die rangeigene node.id). TRANSIT: gescheiterter Arena-Rebind auf TP0 -> TP0 gibt nicht mehr sofort frei, JEDER Rang (TP0 eingeschlossen) ruft beim naechsten Broadcast dasselbe _weg2_release_chain_piece_host. STATE: Abgaben, die TP0 sofort nehmen muss (H19-Anker-Verdraengung, eigenes evict_host, verweigertes Backup sgl-project#1421) -> Zustand (exists, KV-Host, Anker-Host); der Worker gleicht an (unter eigenem Host-Lock/laufendem Write-Through wartet der Eintrag und wird je Broadcast erneut versucht; ein spaeteres erfolgreiches TP0-Backup ueberschreibt ihn). Baum-Reset (Flush/Flip, auf allen Raengen an derselben Stelle) verwirft Ledger und Wartende. - (b) Kapazitaet aus replizierten Eingaben: der 0-Byte-Anker-Pool eines Workers bekommt 2 x SGLANG_HICACHE_ARENA_MAMBA_SLOTS Zeilen (TP0 adressiert Staging 6 + Arena 32 auf rc9m, der Worker hatte nur die synchronisierten 6; 0 B/Zeile). Der KV-Pool des Workers deckt TP0s ID-Raum schon (fnFL2x62, 353600 = 4096 + 5461 x 64). Schalter SGLANG_WEG2_ENABLE_FORM_A_HOST_SHADOW (EnvBool(True)); wirkt nur mit installiertem Form-A-Rollenplan, dessen Host TP-Rang 0 ist, auf reiner TP-Gruppe (keine DP-Attention, pp 1). Aus bzw. klassischer Boot = d1c7094. H93-H99 bleiben als Riegel; mit identischen Baeumen feuern nicht mehr: H96 RU FLOOR CAP / CAP-MISS, H97 RU FLOOR SKEW-ZERO, H98 FormAFollowMiss und die FOLLOW-Abweichungszeile, H99 W65 PrefetchSpanSplit und PREFETCH-FOLLOW, der H93- Usable-Arm nullt nur noch einheitlich. Tests test_form_a_host_shadow_r12 (14, echte UnifiedRadixCache-Methoden -- Storage-Drain, Transit-Release, loading_check, Evict-Trichter, _evict_device_leaf/_evict_to_host/_evict_host_leaf, evict_host, sgl-project#1421, H19 -- auf echten UnifiedTreeNodes, je Rang mit echtem installiertem Form-A-Rollenplan, Broadcast als Pickle-Rundreise): auf d1c7094 3 rot (rc9m-Form: TP0 {128,256,384: KV+Anker}, TP1/TP2 leer; Store-Ack; Load-Back), 2 Waechter gruen (Schalter aus, klassischer Boot), 9 ohne Modul uebersprungen; mit R12 14/14 gruen. Betroffene Suiten (H74, 1424, x20, 1068-mamba-mib, x25, 810, dround-27b, 841, 904, H19, 1481, Task3-Loadback, H81, 1469, 639, 645, 1158, H98, H99, H97, H96, RU): 33 Rote identisch zur Basis (Altlast), 0 neu. H99b (0002) wendet sich unveraendert darauf an (andere Dateien).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…ithheld-Posten des Idle-Ledgers, kein Leck -- fnFL2h91bb2 starb am ersten Idle nach dem ersten n=1-Wake Befund (Boot fnFL2h91bb2, Baum e17bd54 = erster Metallboot mit H95c, D-Log /spinning/evidence-665-f1/boot_weg2_fnFL2h91bb2_e17bd548b5_0926_142251.D.log): der erste 97k-Flip P->D weckt D um 14:31:00 mit 'WEG2 D-PHASE-SEATS (H95) ... handoff_n=1 -> n=1 of cap 6' (Z. 14022) und 'WEG2 D-SEAT-VRAM (H95c) n=1 ... mamba_slots=1..7' (Z. 14057); das erste on_idle danach wirft auf ALLEN DREI Raengen (Z. 14100-14163): [mamba] total=38, available=7, evictable=0, withheld=0, ... leaked_mamba_pages={8, ..., 38} (alle slot_used=False, releaser none) D tot, Boot tot. Wurzel: MambaSlotAllocator.set_phase_limit (H95c, allocator/mamba.py) haelt die Slots ueber L(n) in _phase_withheld -- frei, ohne Besitzer, ohne Seiten -- und SchedulerInvariantChecker._check_mamba_pool gab dem Ledger withheld=0. 31 freie Slots standen weder in available noch in withheld und zaehlten als verloren. Der KV-Pool hat fuer genau diese Klasse seit sgl-project#656/sgl-project#790 den Posten withheld (residency_withheld_slots); der Mamba-Zwilling fehlte. Fix (kein Schalter, keine Kollektive): - allocator/mamba.py: phase_withheld_slots (Anzahl) und phase_withheld_ids() veroeffentlichen den zurueckgehaltenen Satz. - invariant_checker.py: _mamba_phase_withheld liefert den Posten mit Namen (', withheld_by=d-seat-phase(H95c) limit=1..L'); _check_mamba_pool und die aktive Haelfte des int8-Ledgers fuehren ihn als withheld-Term; die sgl-project#924- Belegung rechnet ihn auf der Kreditseite (ein zurueckgehaltener Slot, den der Baum noch nennt, ist Aliasing); die Doppel-Frei-Zensur (_mamba_double_claimed) nimmt die zurueckgehaltenen Ids zur Freiliste (ein Duplikat dort bleibt benannt toedlich); die Leck-Diagnose zaehlt sie als frei, nennt also nur echte Verluste. - Echtes Leck innerhalb der Sitze bleibt Alarm: zurueckgehalten wird nur ueber dem Limit, und set_phase_limit verweigert, solange darueber ein Slot lebt -- ein in 1..L verlorener Slot fehlt weiter in available + evictable + withheld. - Rang-einheitlich: Limit und Satz sind Funktionen der Wake-Anfrage und des eigenen Ledgers, jeder Rang liest denselben Term. Tests: neu test/registered/unit/weg2/test_weg2_d_seat_withheld_h95d.py (8, echter Allocator + UnifiedRadixCache + Checker auf CPU): bb2-Form (38 Slots, n=1, 7 frei) ohne Alarm mit benanntem withheld=31; echtes Leck in 1..7 bleibt Alarm und die Diagnose nennt genau diesen Slot; Duplikat im zurueckgehaltenen Satz benannt toedlich; zurueckgehaltener Slot im Baum = sgl-project#924-Aliasing; n=6 ohne Posten; n=6 -> n=1 -> n=3 ueber zwei Wakes mit bedienten Requests und gecachten Ankern; drei Raenge gleicher Befund. Auf e17bd54 7 rot / 1 gruen (n=6), mit Fix 8 gruen. Betroffene Suiten (18 Dateien: 924/912/927/929/827/841/gdn_slot_executor, idle_census_1262, kv_page_invariants, kv_backing_relief_631, tree_drop_856, q0, pingpong, H95c, slot_trail_1467, train2_fix2_1264): Basis 12 rot (7 davon der neue Test), Fix 5 rot -- die 5 Altlasten test_mamba_double_free_924 (erwarten den per sgl-project#969 CUT C geloeschten double_owned=-Term) identisch. Nebenbefund, nicht Teil des Fixes: '#1028B FETCH CAP ... mamba (0,-1)' und '#1035c ... cause=CAPPED by=mamba' (Z. 14055/14056, schon 14:30:58 vor dem Wake und damit vor dem Slot-Limit) sind keine H95-Folge: die Arena hielt beim Wake erst 192 von 1528 Seiten (sgl-project#1439 ARENA-PRESENT leading_complete=192, complete=264), die Mamba-Anker liegen am Ende (h91v1: deepest_idx 1527 bei 1528/1528), also kein Anker im Bereich; sgl-project#1471 SETTLE parkt den Request wie in h91v1 (dort SETTLE-RELEASE 0,3 s spaeter). bb2 starb am Ledger, bevor der Settle loslassen konnte.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 27, 2026
… Gruppe -- Worker uebernehmen ihn, Uneinigkeit ist ein benannter Stopp dpr-Tod (rc12j 3e97ef0, dkrnfh91dprbar1dauer09270832, D-Log 203833-203988, 08:51:31): weg2-14-70, host-gedeckter Treffer 75264 (KV-Key 81856, uncached 6817). Alle Raenge matchten 75264 (sgl-project#1042), X-GATE admit und sgl-project#794 3712 ueberall gleich; TP1/TP2 bauten den Extend (sgl-project#969 n=68 prefix 75264, 3712 Zeilen) und gingen in den Forward, TP0 schrieb fuer den rid nichts mehr: kein sgl-project#969, kein sgl-project#988 LOADBACK (unbedingt geloggt), kein WEG2-ARENA-LOAD, kein WEG2-LOADBACK-WAIT (im ganzen Boot 0x, druckt seine ersten drei). TP0 verliess PrefillAdder.add_one_req also VOR init_load_back -- dort sind die einzigen rang-lokalen Ausgaenge die Gates `total_tokens >= rem_total_tokens` (NO_TOKEN, stumm; batch_is_full haelt den rid in der Queue). Watchdog-Batch TP0 08:56:59 = Decode 12-55/14-73/14-62/14-65/14-71 ohne 14-70, Pool available=99008 evictable=704. Arena-Pin (2436) war ab 08:50:59 flach und wird vor init_load_back nicht gefragt -- es ist das KV-Budget-Gate, nicht das Arena-Pin-Budget. rc12g-Tod 09:21:17 (8f0bf40, #791b DIGEST MISMATCH, weg2-13-55) ist dieselbe Klasse: TP0 extent 52288 (0 Device-Zeilen), Worker sgl-project#988 extent=128. Wurzel: das Gate bepreist `fill - len(prefix_indices)`. Auf TP0 (einziger Rang mit KV- und Arena-Bytes) ist der Treffer host-gedeckt, 0 Device-Zeilen bis zum Load-back HINTER dem Gate (82081 bzw. 52288 Zeilen + Reservierung); die Worker gehen mit Device-Praefix hinein und bepreisen 6817 bzw. 44+128. Gleicher rid, gleiche Arithmetik, zwei Verdikte. Fix (H98-Form, TP0 entscheidet, Worker folgen): - add_one_req sammelt die rang-lokalen Budget-Gates (vor und im Lock) zu EINEM Verdikt; auf Form A nimmt jeder Rang an genau einer Stelle -- im Lock, nach allen Gates, vor dem Load-back und der Tail-Adopt-Abstimmung -- den Verdikt des Hosts (Scheduler._form_a_admission_follow_fn, ein gestempelter broadcast_pyobj von TP0 je Gate-Aufruf, Watchdog-Arm chain-recv/form-a-admission/tp<-verdict). Host-NO_TOKEN laesst den Request auf allen Raengen in der Queue. - Riegel: rid im Verdikt -> "H105 RU FORM-A ADMISSION SPLIT"; nach der Schleife vergleicht jeder Rang die gebaute Extend-Menge (rid, start, end) mit der des Hosts -> "H105 RU FORM-A EXTEND-SET SPLIT"; verschiedene Zahl von Gate-Aufrufen -> "... MALFORMED" (benannt statt Hang). - Warte-Zeile "H105 RU FORM-A ADMISSION WAIT rid host_price host_budget local_price local_budget refusals waited_s" und "AFTER-WAIT" beim Zulassen. - Ausserhalb Form A (tp 1, pp > 1, SGLANG_WEG2_ENABLE_FORM_A_TP0_FOLLOW aus, 27B) wird nichts installiert: Gates rang-lokal, byte-gleich, kein Broadcast. Tests test_nf_form_a_admission_follow_h105.py: dpr-Form ueber das echte add_one_req und die echten Scheduler-Methoden (Basis 9417507 rot: Host NO_TOKEN, Worker admit), Aufrufzahl-Riegel, kein Aushungern (Pass n abgelehnt, n+k zugelassen, Raenge einig, Warte-Zeile mit beiden Preisen), Aus-Pfad.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 27, 2026
… so the same pass's extend evicts what it needs; a rank-local extend relief provider is the net Metal rc12q (dkr27breleasedraftbar1w109271525, 21d7e3b, with ED/EF), P log, PP2 15:38:49Z, weg2-4-37: SF LOADBACK-ROOM SAME-PASS kv_tokens=5932 floor=5336 avail=5336 evictable=245003 evicted=810 avail_after=6146 sgl-project#988 LOADBACK prefix moved to 20479; WEG2-ARENA-LOAD rows=5932 sgl-project#969 EXTENT (20479, 21254) 775 -> "extend allocation failed and NO relief provider" -> Out of memory Root: alloc_token_slots' eviction trigger reads uniform_avail_for_evict = published floor - admitted ledger. The floor (5336, published at the top of the iteration, this rank's own value on P) was not charged for the load-back's 5932 rows (cache_controller.load allocates outside the ledger), so the trigger read 5336 >= 775 and SKIPPED, while the live pool held 6146 - 5932 = 214 -> alloc failed with 245003 evictable. (1) weg2/pp_slot_fidelity.note_loaded: UnifiedRadixCache.load_back, after a successful device load, charges the loaded rows to the floor ledger (note_uniform_admitted) -- only while the floor is marked local-PP (tp group of one, pp>1) and SF is on. The same pass's extend trigger then reads 5336 - 5932 < 775 and evicts. On a TP group the ledger is untouched (group MIN, replicated). (2) ensure_relief_provider: once per process (registered from the single-rank floor publish, pp>1), a rank-local provider for common.register_extend_relief_provider: acts only while the tree's floor is marked local-PP (returns 0 and touches nothing on a TP group, so D's replicas cannot diverge), evicts exactly num_tokens - available, applies staged frees, logs "EXTEND-RELIEF evicted asked avail a->b evictable_left (n)". Not built: SF pre-evicting kv_tokens + this pass's extend inside local_pp_room -- the extend width is not known at load-back time; with (1) the extend's own trigger evicts the exact amount it needs. Switch: SGLANG_WEG2_PP_SLOT_FIDELITY (SF's own; 0 = no charge, no provider). Tests: test_weg2_sf_extend_relief_rc12q_0927.py 6 with the metal numbers (floor/avail 5336, evictable 245003, load 5932, extend 775): the charge makes the extend trigger evict and the alloc succeed; without charge or net -> Out of memory (the metal); the provider alone funds it and logs EXTEND-RELIEF asked=775, registered once; on a group floor the provider returns 0 and touches nothing; charge only on the local-PP form and with the switch; wiring (load_back, scheduler).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…ern -- Chunk-0-Punkt nur Ein-Sequenz-Chunk, Wachstum = Grundstand (Rate je Request-Instanz, Kappe ueber alle Lastzustaende) Befund (28.09., Boots der H91-Dauerform rc12z14..rc12z19, 13 P-Logs mit GRAPH-POOL): p_card_reference_from_logs nahm das Wachstum aus der Hochwassermarke last = max(samples, key=(chunk_index, peak)) minus Chunk-0-Punkt. Eine GRAPH-POOL-Zeile ist ein Hochwasser (Schritt 0,25 GiB), ihr Chunk-Index ist der Start des ERSTEN Requests, und unter H91 traegt ein 16k-Chunk 2-5 Sequenzen. Boot dkrnfh91dprsavisbar1dauer09280831 08:45:23: bs4-Chunk am Chunk-Index 1, Spitze +1942 MiB ueber dem Chunk-0-Punkt -> PP0-Wachstum 1942 MiB je 16k, P-KARTE PP0 Kopfraum -10646 MiB bei 262144 Token. Das ist die Mehr-Sequenz-TRANSIENTE (traegt P_ACTIVATION_MIB, sgl-project#242), kein Wachstum mit der Tiefe. Dazu wiederholen sich rids (weg2-4-2 startet bei 0 neu, waehrend die erste Instanz bei 53185 parkt). Messung (H55 WEG2-VRAM-PEAK phase=chunk je Chunk, allocated NACH dem Chunk = Grundstand): der Grundstand waechst mit der Tiefe und saettigt -- ueber alle Lastzustaende der heutigen Boots PP0 +1211, PP1 +1056, PP2 +398 MiB gegen den Ein-Sequenz-Chunk-0 (bs1 bis Tiefe 98304, bs2-4 bis 96880); innerhalb EINER Request-Instanz in Ein-Sequenz-Chunks steilster Anstieg 377/347/340 MiB je 16k-Chunk. Die Spitze darueber ist Transiente (peak - allocated danach): heute max 4585/2945/2943, Record 5984/2952/2944. Fix: p_card_reference_from_logs(windows=True) -- der Launcher waehlt es, wenn ALLE Referenz-Logs H55-Fenster und #969N ADMIT tragen (logs_carry_seq_windows; fnFL2x163-165 bleiben beim alten Weg, P_CARD_REFERENCE_FNFL2 byte-gleich): - seq_windows: je H55-Chunkfenster Sequenzzahl (letzte #969N ADMIT), Request-Instanz (rid#n, neuer Start bei 0 = neue Instanz, aus allen reqs der sgl-project#969 EXTENT), Tiefe, Grundstand gegen den ersten Ein-Sequenz-Chunk-0 der Stufe; - Chunk-0-Punkt nur ein Ein-Sequenz-Chunk voller Breite, normiert mit SEINER gemessenen Transiente statt dem Stuetzpunkt; - window_growth: Rate = steilster Anstieg je Chunk in einer Instanz (nie unter einem Messpunkt), Kappe = hoechster Grundstand ueber alle Fenster jeder Last, Saettigung = ceil(Kappe / Rate); Fenster aller Boots gepoolt (Instanzen je Boot benannt), auch Boots ohne eigenen Chunk-0-Punkt; - PCardReference.growth_cap_mib (neu, leer = ungekappt): solve_p_card kappt Rate x Chunks am gemessenen Maximum. Planer-Dry-Run (offline, CPU, Live-Form rc12z17: Schnitt 29,11,8, Chunk 16384, 6 Sitze, Mamba 32, Budgets 28240/17680/17168, NF-Records sgl-project#242, Mitbewohner-Spanne 2310/1012/1012, Stuetzpunkt Karte = max(builtin, Record)), P-KARTE bei 262144 Token, FR_P heute 0.324/0.637/0.734 (198/359/408 Zeilen): - Referenz ALT fnFL2x163-165: PP0 Kopfraum 338 (STIRBT) Decke f 0.322 (197 Z.), PP1 1100 Decke 0.689 (385), PP2 3407 Decke 0.996 (563) - Referenz naiv neu (Hochwasser): PP0 -10646 (Wachstum 1942/16k) -- das Instrument, nicht die Karte - Referenz NEU (Fenster): PP0 Kopfraum -462 Decke f 0.298 (185 Z.), PP1 519 Decke 0.646 (363), PP2 3316 Decke 0.996 (558; W120-Kappe 2 Scratch-Zeilen) - Pool (PP-Cut-Modell) an der Decke 185/363/558: Tokens je Rang 706365/1193240/809739, Pool 706365; nur PP2 an die Decke: 573058/1229543/809739; Boeden 262144 und 524288 halten. Luft am Metall (H55-Fenster, NVML card_free am Fensterende, 390 Chunks je Stufe): PP0 min 167 (bs1) / 319 (bs2) / 171 (bs>=3), PP1 86/88/88, PP2 2098/2284/2144 MiB -- NVML ist Untergrenze (Allokator-Polster reserved - peak kommt dazu). Abdeckung: nur P-Phase Chunk-Forward; Last bs1..4 gemessen (bs5 nur im Record 5984), Tiefe bis 121868 Token (Wachstum saettigt ab Chunk 4, dahinter Saettigung angenommen), Mitbewohner D-TP schlafend (Flips bis 6); NICHT abgedeckt: Flip-Phase (Wake-Kredit H54/W126) und D-Phase -- dort ist PP2s Mehr-Residenz eigens zu rechnen. Tests test_weg2_p_card_windows_242b (8, Fixture = gefilterte echte P-Zeilen dkrnfh91dprsavisbar1dauer09280831, 645 Zeilen): alle 8 rot auf d604c34, gruen mit Fix (darunter: alter Weg liest die bs4-Transiente als 1942/16k; Instanz-Neustart bei 0; ein Boot mit nur Chunk 1 kuerzt die Saettigung nicht; Kappe im Solve; ausgelieferte Referenz ungekappt). Gate betroffene Dateien (p_card_chunk/launcher-Leser: h41, h92c, h55, h57, 1a, h101, 242): NEU 203 gruen, Basis 195 gruen + 8 neue rot, 0 neue Rote.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
… sibling harness moved in The module did not COLLECT on any tree since 069f98c (sgl-project#969 CUT K, 29.08.): it imported _FakeBatch and _req from test/registered/scheduler/test_phase_flip_resident_carry.py, which that commit deleted with the resident carry ("No module named 'test_phase_flip_resident_carry'"). Not a sys.path case -- the file is gone; its other siblings (seam_census_double, test_phase_flip_runtime) resolve, scheduler/ is no package. The two pure stubs are moved in verbatim from 069f98c^ as _carry_req / _CarryFakeBatch. Test-only. Consequence, stated: the module now RUNS -- 7 passed, 23 failed. All 23 pin APIs the sgl-project#969/sgl-project#1233 unweave removed (_resident_batches 14, Scheduler.handle_phase_flip 4, run_phase_flip_event_loops 2, arm_phase_flip 2, kv_capacity_runtime 1): dead tests that the collection error hid for a month. Deleting or rewriting them is a separate decision; gates that included this file no longer abort their whole block on the collection error.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…CUT K / sgl-project#1233 unweave: pinned removed APIs Operator decision 28.09.: delete, do not rewrite. After dea9529 made the module collect again, 23 of its 30 tests failed on APIs the sgl-project#969 CUT K (069f98c) / sgl-project#1233 unweave removed: _resident_batches (14), Scheduler.handle_phase_flip (4), run_phase_flip_event_loops (2), arm_phase_flip (2), kv_capacity_runtime (1). Removed whole: TestProductionCutover (3), TestResidentSetSurvivesTheCutover (4), TestEventLoopRedispatch (2), TestPhaseFlipRpc (4), TestSleepOnIdleSkip (1); removed methods: TestSpeculationIsTpDecodePhaseOnly (7 of 8), TestPin4SchedulerLevelReplay (2 of 3). With them the helpers only they used (_ResidentStubScheduler and the two stubs dea9529 moved in), the sibling imports (seam_census_double, test_phase_flip_runtime) and the now-unused names (dataclasses, threading, TP_TO_PP, PHASE_TP, PhaseFlipLoopExit, PhaseFlipRuntime, verify_flip_cutover). The 7 green tests stay: module 7 passed, 0 failed. Test-only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…cute its row (W102b) NF rc12z30c -st (f33274e) died 28.09. 20:50:53Z. P-log boot_weg2_dkrnfh91dprsavisnoadoptstbar1dauer09282039_f33274eac0_0928_203953: - PP1/PP2 20:41:59 'sgl-project#631 ROW AUTHORITY DISABLED' (followers plan rank-locally) - PP0 20:48:53 'W102 Weg2VisionStage run=1 ASYNC started rids=[weg2-6-33]' -> held - PP1/PP2 20:48:53 'sgl-project#969 EXTENT n=9 fwd=1 reqs=[(weg2-6-33, 0, 12579 ...)]', PP0 none - PP1 'PP-RECV-OBJ ... awaiting_size' 30/60/90/120 s (proxy frame PP0 never owed) - PP0 20:50:53 'sgl-project#973 RING COMMIT TIMEOUT' in _pp_join_comm_work, PP1 gloo closed. Not GIL, not a stream sync, not the worker: the ranks split on the admission. The same shape sgl-project#1066/#969Z closed for the prefetch withhold, and the same law (pp_row_carrier_present: PP0 may withhold only when the followers execute its row). The async stage broke it; the synchronous stage stages AND admits in one pass, so every rank admits the same request (z29b, where vision worked). vision_async_admissible(): single stage -> yes; else pp_row_carrier_present( term="vision_async") -- new Fix B term SGLANG_WEG2_P_ROW_VISION_ASYNC, default off. Not admissible -> the synchronous stage, one named line 'W102b Weg2VisionAsyncNotAdmissible'. NF (no carrier) and 27B row form (term off) both run the synchronous stage by default; the async form is opt-in per term on the row form. Tests: test_weg2_vision_rank_runner (+2: carrierless PP3 -> sync + one line; row form needs its own term), p_row_authority_0927, vision_rank_wiring, vision_rank_stage_core: 64 passed. Mutant (gate forced true): both new tests red.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 29, 2026
…a-Claim + Mamba-Evict ohne Endlosrunde -- M1s rc12z30j tot M1s D-only auf rc12z30j (bb82fbc, Schnitt [0,32,32]/64, D-Log boot_weg2_dkrnfdh91dprsavisnoadoptstcutvsyncdonlybar109290056_bb82fbcb68_0929_005633.D.log): alle drei Raenge treten 01:04:50 in denselben Extend ('sgl-project#969 EXTENT n=62'), TP0 raeumt per Write-Back frei und geht ins Forward, TP1/TP2 schreiben ab dort nur noch '#969H BACKUP' / 'sgl-project#1421 BACKUP-REFUSED why=mamba_pool_unbound' / 'R12 SHADOW-SHORT' (16189 Zeilen, bis 5700/min, immer dieselben node-ids 22/25/32/35/37/46/84) -- 01:06:34 TP0 'Bar1CollectiveAborted ... exceeded its cycle deadline' im MoE-All-Reduce des Extends (Z. 51493). Wurzel 1: unter dem Schnitt ist der KV-Hostpool eines Form-A-Workers ein ArenaMHAHostPool (seine Zeilen sind die Seitenbytes), sein Mamba-Hostpool bleibt der schlichte byteless R12-Schattenpool ('mambahost=MambaPoolHost', 'R12 SHADOW-CAPACITY mamba anchor rows +64 ... 0 B/row'). Der Direct-Pfad (#1427b) verlangte fuer den Anker einen Mamba-ARENA-Slot, fand keinen und verweigerte den ganzen Knoten -- jede Worker-Sicherung, auch die der Eviction. Fix: auf einem KV-only-Rang (rank_role.kv_only_rank) mit schlichtem Mamba-Hostpool nimmt der Anker eine Schattenzeile aus diesem Pool (controller.write, Staging-Weg); ist der Schattenpool voll, geht die Eviction KV-only (P-FUND-Form), R12 gleicht den Anker ab. Gewichtsrang und klassische Raenge unveraendert (dort bleibt die Verweigerung benannt). Wurzel 2: MambaComponent.drive_eviction startet am LRU-Ende neu, sobald der Nachfolger nicht in der Liste ist -- auch am Listenkopf (get_prev -> None). Ohne Fortschritt ist das eine Endlosrunde im Scheduler-Schritt. Fix: eine neue Runde nur, wenn die vorige etwas frei gemacht hat; sonst endet die Eviction kurz, benannt ('MAMBA-EVICT NO-PROGRESS'). Kein Schalter, keine Reserve; Decode/Flip unberuehrt. Tests test_cut_worker_shadow_anchor_m1s (7): 6 rot auf bb82fbc (Livelock mit der Metall-Signatur, Worker-Backup 0, Evict frei 0), gruen mit Fix; 1 Waechter (Gewichtsrang verweigert weiter) gruen auf beiden. Nachbarn (P-FUND, R12, R12-Schnitt, sgl-project#248, sgl-project#910, sgl-project#1481, kv_only_rank, EB, rc12o): 94 gruen; 639b 16 Altlasten identisch auf Basis und Fix, 0 neue Rote.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 29, 2026
…roject#1400/sgl-project#1416-Told -- z30j (bb82fbc) starb an der rang-lokalen Erstfassung 879222e Beleg z30j, P-Log ...dauer09290109_bb82fbcb68_0929_011011, 01:16:17-20: PP0 'WEG2 P-FORK-CUT CUT rid=weg2-4-6 site=add_one_req prefix=0 extend=16384 prompt=16919 fork=4608 src=store cut=4608' -> 'sgl-project#969 EXTENT ... (weg2-4-6, 0, 4608, 0, 4608)' PP1/PP2 'sgl-project#1400 STORE-TOLD ABSORBED ... told=0 verdict=declined:weg2_told_zero' -> 'sgl-project#969 EXTENT ... (weg2-4-6, 0, 16384, 0, 16384)' -> W27 PPWidthDivergenceRefused Die Erstfassung las den Fork auf jedem Rang aus dessen eigener Store-Probe (_STORE_UNCAPPED, vom Prefetch-Thread notiert) und dem eigenen Baum. Ihre Praemisse "Folger fuehren PP0s weitergereichten Umfang aus (sgl-project#791)" gilt auf der NF-P-Form nicht ('sgl-project#1245 ARMED ... no sgl-project#631 row carrier', 0x sgl-project#631 ROW-DELIVER): jede Stufe leitet den Extend selbst ab. Bei told=0 registriert ein Folger keinen Read, probt nie, schneidet nie. Neubau: - weg2_store_told: PP0 entscheidet den Fork EINMAL beim Told-Publish (_pp0_fork -> p_fork_cut.pp0_fork_verdict aus PP0s Probe/Baum) und haengt ihn als Draht-Attribut WIRE_FORK an den Told (Einphasen-, Zwillings- und gepacter #1416e-Told); nur bei Schnitt gesetzt, sonst ist der Draht byte-identisch (PF-Golden unveraendert). Folger uebernehmen ihn im Absorb, admission() setzt req._weg2_fork_told auf JEDEM Rang gleich. - p_fork_cut.apply schneidet nur an req._weg2_fork_told, nie an Probe oder Baum des eigenen Rangs. Kein Told-Fork -> kein Schnitt auf keinem Rang. Die wiegende Regel bleibt (Schnitt nur, wenn er keinen Forward kostet; Intervall-Regel), kein Schalter, keine Reserve. - schedule_policy (add_chunked_req, add_one_req) und hybrid_cache_controller (note_store_uncapped) wie in 879222e. Tests (CUDA_VISIBLE_DEVICES=): - NEU test_weg2_p_fork_cut_uniform_0929: echte PP3-Folgerzulassung (drei Stufen-Doubles durch intake -> pp0_publish -> follower_absorb -> admission, nicht gemockt; Store-Probe je Rang getrennt wie im Prozess), dann der Adder-Hook. Auf 879222e: 5 rot, u. a. 'W27 width divergence: PP0/PP1/PP2 extend [4608, 16384, 16384]' (einphasig und gepact). Mit Neubau 8/8 gruen. - test_weg2_p_fork_cut_0928 auf Told-Quelle umgestellt, 10/10 gruen. - Nachbarn: 23 Told-/Fork-Suiten 367 passed, 5 rot = identisch zur Basis 3f171d0; 62 Adder-/Controller-Suiten 783 passed, 46 rot = identisch zur Basis (Mengen per diff verglichen), 0 neue Rote. Metall-Marker: 'WEG2 P-FORK-CUT TOLD rid=... fork=... src=store|tree' auf PP0, 'WEG2 P-FORK-CUT CUT ... src=told cut=X' auf PP0, PP1 UND PP2 mit gleichem X, 'sgl-project#969 EXTENT' je rid auf allen drei Stufen gleich breit, 0x W27 PPWidthDivergenceRefused.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…rid decode view (F1 prefill ext, F2 decode reqs, F4 P leg-1 times) User decision 02.10. (list from the dashboard seat): only additive fields in existing IPC / metric events, no behaviour change, no new sync point, no .item() / D2H in the decode path, host values only. * F1 rankstats prefill: 'last.ext' (the timed flush's folded chunks) and 'last_ext' (the newest chunk, timed or not) = [[rid, start, end], ...], start = req.extend_range.start (the sgl-project#969 EXTENT start, else len(req.prefix_indices) -- a tensor's len is its shape), end = start + req.extend_input_len; cap 8. Built in report_prefill_stats after the forward (metrics_reporter.prefill_ext), carried through RankPrefillLog.record (_pending tuple + 5th element, read back tolerantly) and _cum_untimed; the timed 'last' reading is untouched by an untimed chunk. * F2 rankstats decode: 'reqs' = [[rid, prompt, out], ...] (len(origin_input_ids), len(output_ids)), read in the rankstats timer thread beside the 'running' count (rankstats._decode_reqs), cap 16. * F4 request_done: 'p_leg1_dispatch_ts', 'p_leg1_end_ts' (the row's leg-1 clocks). * F3 decode.tokens_by_bs NOT built: the decode round log keeps no per-round host token count. Tests: test_weg2_dash_fields_1002.py 7 (7 red on the base 6dd8d7e, 7 green); rankstats_s3 key-set pins + last_ext / reqs. Suites over RankPrefillLog / rankstats / RequestBook (25 files): 336 passed, 5 red = test_tp_decode_formation_861j x4 (red on the base too) + the key-set pin (fixed).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 3, 2026
…31032, weg2-0-18, P-PAUSE abort in the told's own pass) The log's 'weg2-0-1' is rid[:8]; the sgl-project#1223 dump names weg2-0-18. Same class as 10030958: PP0 published instance 1's told=0 in the pass the DUAL P-PAUSE abort landed, the followers' admission visit consumed theirs, PP0's stood; instance 2 (P-PAUSED requeue) was admitted on it at 10:36:47 ('sgl-project#1400 STORE-TOLD WAITED told=0', sgl-project#969 EXTENT (0,1024)) and PP1 died #791T. Red on edeb022, green on ee24fb3.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 14, 2026
…'s progress, not a fixed bound NF h16 a2cfc77 (boot 1014_151524), P 15:34:27-32, weg2-10-86 (92784 tokens): PP0 and PP1 each waited the fixed 5-s JOIN-WAIT on 190 stems held by PP1's own l3fill claims ('holders=[1904/1ms/pid778/l3fill...]', pid778 = PP1, 'still_claimed=190 waited_ms=5001/5005'). At 5 s PP1's stale reap broke them and read them itself ('sgl-project#1433 190 of 190 ... ms=5111'); PP0 had given up 4 ms earlier and found them freshly re-claimed ('ARENA-GET MISS find=(slot=1904,state=1)'), its read ended at 65536, sgl-project#257 cut the claim to 65280 -- uniform on all three stages (PP0 told 65280, PP2 W27-UNIFORM re-read to 65280, 'sgl-project#969 EXTENT 65280..' on every rank) -- and 26624 tokens were re-prefilled. _await_fill_joins: any of its stems completing, or a pending claim taken over / re-claimed (slot or generation moved, claim_info), is progress and restarts the idle clock; the wait ends when nothing moved for max(bound, SGLANG_WEG2_L3FILL_STALE_CLAIM_S) -- the caller's existing stale reap then breaks a stalled or dead holder's claim and reads the page itself -- or at the emergency cap SGLANG_WEG2_L3FILL_JOIN_CAP_S (60 s) however much moves. _await_fill_room: same rule (slots still coming = progress, idle bound = the room bound, the same cap), and each round's fresh claims are read AT ONCE instead of after the whole wait -- a claim held unread through the rest of a room wait is what a peer's JOIN-WAIT sits on. SGLANG_WEG2_L3FILL_JOIN_LIVENESS (default on); 0 = the fixed bounds. Tests: test_l3fill_join_wait_0930.py +5 (a holder that keeps completing is waited for past the bound; liveness off ends early; a holder that never moves ends at the idle bound, not the cap; a stalled holder is reaped after the stale age and read; room wait keeps going while slots come); two bounded-miss tests pin STALE_CLAIM_S=0: 16 passed; 1433, stale_claim_reap, joined_owner, l3_fast_read, anchor_bypass: 27 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 15, 2026
…n before the release and is not the released set; chunk start from its own EXTENT / rank batch, never t0_unix_ms; pin the 9591442 lane class in the front Review NO of 5e320a8: the round-1 monitor was blind for the real class (route=D on an empty D): it counted the released set's own sgl-project#969 EXTENT (+1 s after the release) and read WEG2-VRAM-PEAK t0_unix_ms (start of the instrument window, end of the previous chunk) as the chunk start. Real cases 14.10. 10:29:45Z, 10:30:14Z (boot bc35462240_1014_094512) and 10:54:41Z (95914428c2_1014_105023) were swallowed (54 -> 3). Monitor: D-log stamps are floored to the second -> a point witness counts only if s+1 s <= release; exact epochs (Decode rank batch t:, chunk end t_unix_ms) in [release-waited, release]; chunk interval from its own EXTENT stamp (+1 s, rows pairing) and from t_unix_ms - gpu-ms of its Prefill rank batch (rows pairing; the sgl-project#969 probe is throttled after n>400); rids D-ADMITted in [release, release+2 s] never count; replay prunes by the D-log clock (the wall-clock prune erased every witness of an old boot). Tests: monitor 25 passed (real cases A/B/C verbatim + the 13:13:02Z false alarm in one monitor), front busy-D 9 passed (new: lane-parked and below-floor granted SHORTs, control, mixed book; weg2-54-149/150). Mutants: 18 monitor mutants all killed; round-1 branch fails 13, reviewer candidate 7, old live copy 13 of the same 25 tests; front mutant M4 (idle filter ignored) 3 red, M1-M3 5/2/2 red. Regression with the six older collect suites: 91 passed. Replay of 66 boots 13.-15.10. (264 replays): 15.10. 1437 releases / 1396 checked -> 0 idle (old live 2, round 1 0); 14.10. bc35462 boot 54 (old 54, round 1 3). front.py unchanged (git diff 226aad7 -- python empty). Report: deskq/done/nf-core-collect-1015.md, Runde 2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The data type for token ids and KV cache indices are supposed to be int32 consistently instead of int64.
Modification
Simply corrected the misuse of torch.int64.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.