Repository navigation
misc: update e2e test benchmark config - #825
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
SGL-Router adds and modifies the health status interface. A new service parameter --router-url is added. When a service process exits, the health status interface is called to change the health status to False. Co-authored-by: wangguoyan <wangguoyan6@h-partners.com>
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
… on resident lock refs FALSIFIED ON METAL, boot_826_review_0823_0912, 08:55:45, all three ranks ~1 minute after ready and 36 s into load: 08:55:24 sgl-project#825 TREE DIVERGENCE ONSET (all 3 ranks) 08:55:42 sgl-project#825 TREE RECONCILE (all 3 ranks, pp_to_tp cutover) 08:55:45 AttributeError: 'NoneType' object has no attribute 'id' full_component.py:239 release_component_lock 08:55:50 kill_process_tree Mine, not the ambient root-D family: the previous boot on the same load profile, same duration, without sgl-project#825, has zero release_component_lock occurrences. My premise was "requests are parked between the movers and the cutover, so dropping the tree is safe". PARKED IS NOT UNREFERENCED. The cutover carries RESIDENT requests across and each holds a last_node with a lock ref; reset() rebuilds the root, orphaning those nodes, so dec_lock_ref's parent walk no longer terminates at the live root. So the ACTION is withdrawn behind SGLANG_TREE_RECONCILE, default off. The DETECTION half stays on and is what earned its place: it found real divergence on all three ranks within 35 s of load, which is the first direct measurement of the B.10 defect rather than an inference from #cached-token. A reconcile that must not run while any node is locked has to be built against the lock refs -- evicting only the unlocked portion, or reconciling where no resident reqs exist -- and that is a design, not a flag flip. sgl-project#827, independent of the above: the walk's only exit was reaching root, so ANY broken ancestor chain -- detach, evict, rebuilt root -- ran off the top and died on None.id, naming an attribute instead of the actual fault. Now a named RuntimeError carrying the node, the component and the hop count. NOT a break: swallowing it would leave component_protected_size_ overcounted for the life of the process, trading a loud fault for a slow leak. Mutant (raise -> break) killed by 3 arms. Tests: 48 green across the four suites. sgl-project#827 red first, with the two crash-pattern arms failing on the real AttributeError before the fix.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
Wave 3, stage 1 -- the first stage of the batched-window tree. Carries WINDOW-QUEUE tickets W1 (sgl-project#825/sgl-project#827, release-walk crash guard) and W2 (sgl-project#826, solved arming floor), which the queue requires to ship in the SAME tree: W1 lets the boot survive the flip walk and W2 lets the flip actually arm, so validating either alone repeats boot_827's wasted window. Four commits, base 21ff075 (the feat/822 stage): bb09574 [sgl-project#825] Prove the prefix trees congruent at the one aligned point 80ca356 [sgl-project#826] Adopt the solved arming floor, or refuse the boot by name e0895cf [sgl-project#825/sgl-project#827] Withdraw the tree reset: it crashed on resident lock refs 500be7e [W2] Pre-flight: arming reachability must include the measured seam draw ANCESTRY COMPUTED, NOT INFERRED FROM THE BRANCH NAME. merge-base with the tip is 21ff075, i.e. this branch does not carry fix/815-rest-stubdrift. That matters only if the two touch the same files, and they do not: sgl-project#815 is test-only across ten helper files, none of which are the five test files here. CONFLICT LAGE: none, and checked rather than hoped. `git merge-tree --write-tree` against the tip returned a single tree with no conflict block before the merge was attempted, and the merge itself reported "Automatic merge went well". The file surface is DISJOINT from wave 3's other stage (fix/801-void-send-contract, which touches only scheduler_pp_mixin.py), so the two stages cannot interact textually. W2's ticket is preflight_pass=N and that is NOT a reason to keep it off the line. The queue's N is CONDITIONAL: the solved arming floor is reachable only while the seam's measured draw stays at or below the solved 218 MiB reserve, and the measured draws on this rig are 162/387/436/907 MiB. That gates the WINDOW, not the merge -- the actuator sits behind SGLANG_ARMING_FLOOR_SOLVED, so merging it changes no default. Delivering one bootable tree is this strand's job; whether the window runs is the operator's. The branch's own pre-flight suite is worth naming because it refuses a false green rather than manufacturing one: test_arming_reachability_preflight_w2.py asserts `band_floor + max(reserve, measured_draw) + margin <= band_ceiling` and records that the obvious W2 assertion -- 819 + 218 + 192 = 1229 <= 1229 -- is true and useless, because it silently assumes a measured draw of zero, which is the case that does not occur. SECTION 18 DUTY, DISCHARGED IN THIS MERGE. This stage adds a new reusable module, managers/tree_congruence.py, so rule 5 applies to this merge the same way it applied to the four modules catalogued in 9f1af20. The entry is in this commit (docs/dev/FEATURE_CATALOG.md, section 18.5). The reusable part is not the prefix-tree logic but the equality-by-MIN-reduce trick: a digest sent as the pair (d, -d) makes the group agree iff group_min == -group_neg_min, which turns "are all ranks equal?" into two MIN-reduces over a collective that offers only min. The entry states the gate honestly: DETECTION is ungated and always runs, while the RECOVERY is behind SGLANG_TREE_RECONCILE (phase_flip_runtime.py:3777, unset = OFF) because the reset it performs is what crashed on resident lock references and was withdrawn rather than repaired in place. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", PYTHONPATH set to this worktree, one battery at a time. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s this stage 7 failed, 8638 passed, 1852 skipped, 868 s NEW failure ids NONE (comm against the baseline list is empty) fixed vs baseline 0 The +53 passes are this stage's own new cases; the failure set is unchanged and is still exactly the seven known ones. The baseline was re-measured in this worktree rather than inherited from strand 17c, and came out byte-identical to its list: the same seven cases, all in test_pp_flip_slot_hold_631.py, left red on purpose (see the stub-drift family in section 12 of the catalog). The gate is not vacuum-green: all five suites this stage adds land INSIDE the battery directories and are therefore actually collected -- test_arming_floor_actuator_826.py, test_arming_reachability_preflight_w2.py, test_tree_congruence_825.py and test_tree_congruence_wiring_825.py under managers, test_lock_release_walk_detached_827.py under mem_cache. Section-18 pointer checker (test_building_blocks_catalog_538.py): 104 passed, up from 101, the three new cases being the three new tree_congruence citations. COVERAGE CHECK, because a battery that silently fails to collect is a gate that measures nothing. The W4/W5 strand reported that its worktree cannot collect the managers suite at all -- 92 modules import `datasets`, which is not installed there. That does NOT apply to this gate, and it was verified in the positive direction rather than inferred from the absence of an error message: the venv here has datasets 5.0.0, the battery log contains zero collection errors and zero ModuleNotFoundError, and an explicit --collect-only over test/registered/unit/managers reports 3436 tests collected. The numbers above therefore cover the managers suite in full. No boot. Serving stays down; this tree is the input to ONE batched window, not a reason to open one.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…le, not by new machinery Follow-up to 3b0c031. The open item was the correctness core: after a cutover that moves no KV, the new phase's device pool holds no valid KV while the radix tree still maps prefixes to row ids, so a lookup must MISS and fall through to the host tier. Which function makes it miss was unanswered. THE OBVIOUS ACTION IS ALREADY KNOWN TO CRASH, and finding that out from the tree rather than from a boot is the point of asking first. phase_flip_runtime.py:4590-4620 records sgl-project#825 trying exactly this on 2026-08-23, on all three ranks at once: cache_finished_req -> dec_lock_ref -> full_component.py:239 `if cur.id in skip_lock_node_ids` AttributeError: 'NoneType' object has no attribute 'id' with the cause in its own words: "PARKED IS NOT UNREFERENCED. The cutover carries RESIDENT requests across, and each holds a `last_node` with a lock ref. `reset()` rebuilds the root, orphaning those nodes." sgl-project#825 withdrew the ACTION and kept detection only, noting the real fix "needs to be built against the lock refs ... and that is a design, not a flag flip." RESOLVED, AND BY THE USER'S OWN RULE. The stale `req_pool_idx` (the resident-carry blocker) and the orphaned `last_node` (sgl-project#825's crash) are the same fact wearing two hats: A RESIDENT REQUEST CARRIED ACROSS THE CUTOVER. The no-carry rule removes it, and both blockers go with it. The mechanism exists and needs no new code: `retract_all` (schedule_batch.py:1812) walks `release_req` (:1783) over every request, and `release_req` does exactly the two releases required -- `release_kv_cache(req, tree_cache, is_insert=False)` and `req.reset_for_retract()` -- taking precisely the objects the flip already holds. SEAM ORDER: fence (maybe_flip_writeback + demotion on) -> retract all -> tree reset (now safe, sgl-project#825's precondition gone) -> cutover + weights refill -> re-admit, served by HiCache read-through. That is the user's sequence verbatim. THE CARRY ONLY EVER EXISTED TO AVOID A RE-PREFILL. With read-through that re-prefill is a cache hit, which is why "no carry" is correct rather than a simplification -- and it is exactly the "honest warm-up cost as served-request latency" named as the validation metric. That cost is REAL and must be measured, not assumed small: it is what this design pays to delete the mover, the staging reserve and a crash class. TWO CAVEATS RECORDED AS UNVERIFIED rather than glossed: that `release_kv_cache` releases the lock ref along the whole parent chain (read from release_req's body, not from release_kv_cache's own), and that retraction at the seam is compatible with the flip's quiescence predicate. Both must be confirmed before this is built. No code changes. Documentation only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
… and retraction feeds the policy a lie Follow-up to 4b6ee1e, which recorded two unverified caveats rather than glossing them. Both are now answered, and only one of them was clean. CAVEAT 1 -- DISCHARGED, AND IT PROMOTES THE ORDER TO A LAW. `release_kv_cache` (mem_cache/common.py:1749) routes to `tree_cache.cache_finished_req(req, is_insert=False)`, and the live `dec_lock_ref` (mem_cache/hi_mamba_radix_cache.py:1610) is while node != self.root_node: ... node = node.parent so the lock ref is released along the whole parent chain -- PROVIDED the walk still terminates at the live root. That is exactly the loop sgl-project#825 crashed in once `reset()` had rebuilt the root and orphaned the nodes. RETRACT STRICTLY BEFORE RESET. Reversing the two reproduces the 2026-08-23 three-rank crash. A red-first test must pin the ORDER, not merely the outcome. Two bypass paths are recorded for the build to handle, both already named in that function: `req_pool_idx is None` under MambaRadixCache, and `kv_spill_state == "host"` routing to the kv-session-offload release. CAVEAT 2 -- NOT A CLEAN PASS. A NEW DEFECT THIS DESIGN WOULD CREATE. Retracted requests return to the waiting queue, so their full context reappears as `pending_prefill_tokens` -- the exact quantity the flip policy compares against N. But N is priced from X and P, the UNCACHED prefill throughputs. Retracted tokens are CACHED by construction: the fence just persisted them and read-through serves them. Their real cost is a small fraction of what the bar prices them at. So every cutover would hand the policy a large pending figure whose true cost is far smaller, in BOTH directions. That is a thrash pathway CREATED by this design, not inherited -- today's carry keeps those tokens out of the pending count entirely. Recorded as unresolved, with three candidate directions and none of them yet evidenced. The honest one is to price the pending figure by cache residency, which is the same class as sgl-project#856(b): a decision is only as good as the quantity it compares. Explicitly NOT claimed as mitigation: sgl-project#856(b) raises N (8.50 -> 18.06 s, 18614 -> ~39500), which makes this harder to trigger. That is a mitigating accident and must not be cited as a fix. No code changes. Documentation only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
… cannot argue for a flip PREREQUISITE for the seam rebuild, and its own defect. The phase-flip seam will RETRACT resident requests rather than carry them -- the carry is what made sgl-project#825's tree reset crash, and read-through makes it unnecessary. But retraction puts each full prompt back in the waiting queue, and `_pending_prefill_tokens` sums `len(req.origin_input_ids)`. That figure is compared against N = C / (1/X - 1/P), where X and P are UNCACHED prefill throughputs. Retracted tokens are not uncached: their KV was computed and the fence persisted it, so re-prefilling them is a CACHE READ -- and a cache read costs the same in TP as in PP. Equal cost on both sides of an inequality cancels, so those tokens cannot make PP cheaper than TP and have no business in the comparison. Pricing them as cold prefill would hand the policy a huge backlog after every cutover, in BOTH directions. sgl-project#731 measured that outcome from a different cause: "51,369 -> 102,307 tokens across one cutover ... six cutovers, nothing served." Its fix (the carry consumes the queue entry) cannot catch this route, because nothing is double-counted here -- the tokens are counted ONCE, at the wrong price. On W25's numbers the shipped sum prices eight retracted 20k prompts at 160,000 against a live bar of 18,614: 8.6x over, from a cutover alone. THE OPTION THAT WAS CHOSEN, AND THE TWO THAT WERE NOT. * CHOSEN -- stamp the residency at retraction. `reset_for_retract` records the fill boundary (`extend_range.end`, the same notion of "computed" the pending counter already uses for a chunked remainder; a request with output has finished its prefill by construction) BEFORE clearing it. Exact, and free. * REJECTED -- residency lookup at the counter. `prefix_indices`, `num_matched_prefix_tokens` and `extend_range` are ALL cleared by `reset_for_retract`, so after the fact the information does not exist anywhere and only a fresh `match_prefix` walk could recover it -- per queued request, per policy round, under the tree lock, duplicating the matching admission already does. * REJECTED -- a dwell-window exclusion. Coarser in the wrong direction: it also excludes tokens that genuinely need computing, so a request retracted mid-chunked-prefill would have its real remainder deleted from the backlog. The stamp distinguishes them (6000 of 20000 computed -> 14000 still count). DEFAULT PATH BYTE-IDENTICAL. A request that was never retracted carries no stamp and is counted in full, so every pre-sgl-project#856 caller gets the same number. A request retracted for a reason OTHER than the flip seam -- priority preemption, sgl-project#731's own path -- has no fence behind it and also carries no stamp, which the can-fail test pins: zeroing anything merely flagged retracted would delete real backlog and pass every other assertion here. ERROR DIRECTION IS THE SAFE ONE. If the cache is evicted under the credit, the stamp overstates residency and this UNDER-reports pending, making the policy LESS eager to flip -- not more. It cannot cause the thrash it exists to prevent. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_uncached_pending_856.py 12 passed, 3 subtests incl. the shipped-sum reproduction of the sgl-project#731 shape, and a source-order assertion that the stamp precedes the clear (afterwards it is unrecoverable) pending-counter consumers (713/677/689/701/819/854/phase_policy) 156 passed retraction consumers (retract_decode_fcfs, prefill_adder, 798, 791b) 28 passed ruff check + format clean
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
The correctness core of the no-KV flip, encoded before the retirement that needs it. Once the flip carries no KV, the new phase's device pool holds no valid rows while the radix tree still maps prefixes to row ids, so the tree must be dropped for a lookup to MISS and fall through to the host tier. That action is already known to be fatal in the wrong order. sgl-project#825 tried it and took the instance down on all three ranks (2026-08-23): cache_finished_req -> dec_lock_ref -> full_component.py:239 `if cur.id in skip_lock_node_ids` AttributeError: 'NoneType' object has no attribute 'id' "PARKED IS NOT UNREFERENCED ... each holds a `last_node` with a lock ref. `reset()` rebuilds the root, orphaning those nodes, so the parent walk in `dec_lock_ref` no longer terminates at the live root and runs off the top into None." THE NO-CARRY RULE REMOVES THE PRECONDITION rather than working around it. Retraction releases each resident request's rows AND its tree lock ref (release_req -> release_kv_cache -> cache_finished_req -> dec_lock_ref, whose loop is `while node != self.root_node: node = node.parent`). Run it FIRST and every walk terminates at a live root; run it after reset() and it walks off exactly as sgl-project#825 did. So the ordering is not a preference, and it is now a named function instead of a comment two callers must remember. REFUSES RATHER THAN REPAIRS when either step is missing, and the asymmetry is deliberate: skipping the retraction leaves locked nodes for the reset to orphan (the crash, loud); skipping the reset leaves the tree naming rows that hold no KV (a WRONG ANSWER, silent) -- which is worse, so neither is allowed to be optional. THE TEST PINS THE ORDER, NOT THE OUTCOME, and that is the whole point. A test asserting only "the tree ended up empty" would pass against the fatal ordering. So the FATAL ordering is reproduced hermetically against a faithful model of the real walk -- nodes with a parent, a reset() that installs a NEW root object, and a release that walks parents until it reaches the CURRENT root, which is the entire mechanism of the crash -- and `test_reset_before_retract_reproduces_the_825_crash` asserts it still raises AttributeError('NoneType'). If that ever stops raising, the model has drifted from the crash it represents and every other assertion in the file is worthless. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_seam_order_856.py 8 passed ruff format clean Correction to the previous commit's message (3ff19e7), recorded rather than force-pushed: it said "ruff check + format clean" for scheduler.py and schedule_batch.py. Format is clean and that change added ZERO ruff errors -- verified, 144 at HEAD~1 and 144 after, ~90 of them the E402s scheduler.py's own comment already documents -- but `ruff check` on those two files is not zero and never was. The claim was imprecise. Coordination: the refill call sites (weights_arena.py, phase_flip_boot.py) are untouched here and stay that way -- W26's refill-root diagnostic owns them on fix/856-refill-root, building on the sgl-project#856(a) instrument from 7500c85.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…ure what warm-up costs TWO PIECES THE DESIGN OWES, both consequences of carrying no KV. 1. VALIDATE-EARLY REFUSAL (sgl-project#806 pattern, ServerArgs.__post_init__). The flip retracts every resident request and DROPS the prefix tree, so the next phase starts with an empty device tier and restores prefixes by read-through. With --enable-hierarchical-cache off there is nothing to read through: the sgl-project#703 flip-time writeback has nowhere to persist, the retracted prefixes are gone, and every conversation re-prefills from scratch on every flip. That is a correctness-shaped cost, not a tuning one, and it is INVISIBLE at runtime -- the flip completes, the requests complete, and only the token bill says anything happened. A silent, expensive, correct-looking failure is what a launch gate is for. Placed beside sgl-project#806's own check and after `materialize_declarations`, for the reason sgl-project#806 documents: the hierarchical cache is switched by handlers above, so an earlier check would pass a launch this one refuses. sgl-project#806 refuses the flip that cannot ENUMERATE what it must move; this refuses the flip that cannot RESTORE what it deliberately drops. NO FALLBACK IS OFFERED AND THE MESSAGE SAYS SO -- a test asserts the phrase. Reviving the mover here would reintroduce the seam this ticket retired, the staging reserve behind W25's 33 refused arms, and the resident carry that crashed three ranks in sgl-project#825. Both exits are named instead, because which one is right depends on what the operator wanted. 2. THE WARM-UP LEDGER (managers/warmup_latency.py), the metric the user named: not "rows carried" but served-request latency by rounds since cutover. PRIOR ART CHECKED AND IT IS A NEAR MISS, not a hit: `regime_classifier.PhaseDwellGate.rounds_since_flip` has the rounds-since-flip concept but is a GATE deciding whether a flip may happen, and carries no latency. Also searched: phase_flip_runtime for warm/latency/post-cutover, the seam census (times the seam, not what follows it), the sgl-project#605 flight recorder (the seam's own peaks). Nothing measures this. Geometric bands (<=1, <=4, <=16, <=64, steady) because the CLAIM is that the cost concentrates in the first rounds and decays; a mean cannot be right or wrong in a way that shows that. Every band reports as a ratio against THIS instance's own steady state, so the figure needs no remembered number from another boot to be readable. The can-fail directions are the file's point. `None` survives wherever nothing was compared -- a ratio against an absent control reads as "no warm-up cost" while meaning "nothing was compared", the sgl-project#606 shape in the one number this ticket is judged on. "Has not flipped" never folds into "has flipped and settled". An empty ledger still says it has nothing to say. HALF-WIRED, AND SAID PLAINLY. `note_cutover` is wired at the cutover. The request feed is NOT: request latency is assembled in `tokenizer_manager`, a DIFFERENT PROCESS, so feeding it is a cross-process change rather than a line. Recorded as an open integration point instead of guessed at -- a wrongly-wired instrument reports a number, which is worse than reporting none, and this is the number the proof window judges the design on. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_phase_flip_needs_hicache_856.py in server_args/ -- 776 passed overall test_warmup_latency_856.py 13 passed, 5 subtests managers -k "phase_flip or seam or warmup or uncached" 564 passed, 28 subtests, 0 failed ruff format clean; ruff check adds ZERO errors -- verified per file from the repo root against HEAD: server_args 358 -> 358, phase_flip_runtime 0, warmup_latency 0, corridor_admission 0. RUFF MEASUREMENT CORRECTION, since a previous commit reasoned from it: ruff must be run FROM THE REPO ROOT or it does not find pyproject.toml and reports a different rule set entirely (phase_flip_runtime.py read as "110 errors" from a stray cwd and "All checks passed" from the root). The earlier 144-vs-144 comparison was taken with the same cwd on both sides, so its conclusion -- zero added -- stands; the absolute number was just scheduler.py's 102 plus schedule_batch.py's 42.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…NCOMPLETE The proof boot did what a proof boot is for: it falsified a claim this note was making. Pin 3111539, boot_w27_0824_1510.log, operator-granted window, died on all three ranks 15:14:45Z. Full result: /spinning/gpu-arb/W27-RESULT.md WHAT PASSED, and it is the one that mattered most. At 15:14:44 all three ranks logged `RESIDENTS RELEASED ... 1 request(s) retracted and the prefix tree dropped, in that order`. Zero `NoneType' object has no attribute 'id'`, zero `dec_lock_ref` -- with the tree dropped while a request was LIVE. That is exactly the sgl-project#825 three-rank crash the no-carry rule was designed to remove, and it did not occur. The order law is now vindicated on metal and not only in its hermetic reproduction. WHAT FAILED, one second later: resident_mamba_slots (gdn_flip_mover.py:620) KvReshardError: PHASE-FLIP-GDN live request da65cfe4... has no mamba slot -- refusing to flip past unmoved linear state I retired the KV mover by rebuilding the transfer plan EMPTY after retraction. I did NOT retire `GdnFlipMover.move()` -- which this note's own retirement inventory lists as a RETIRE entry (3b0c031). So it still runs, still enumerates live requests, and finds one whose mamba slot the retraction just freed. The guard is CORRECT: refusing to flip past unmoved linear state is right, and a loud abort beats the alternative. A SECOND FINDING RIDES ON IT, and it is the more general one: `retract_all` releases rows, mamba slots and the tree lock ref, but the scheduler's batch structures still reference the `Req`. Every seam-side consumer of "live requests" therefore sees a live request with freed resources. The GDN mover is the first to hit it; the sweep for others is a build item. NO RETRY WAS ATTEMPTED, on evidence rather than caution. The obvious in-window fix -- drop the GDN mover from the flip path -- trades a LOUD CRASH for SILENT LINEAR-STATE LOSS, because the GDN anchor resume (sgl-project#745/sgl-project#747) is listed NOT BUILT in this very note. Removing a guard's caller without building what replaces it is the class of change this strand exists to refuse. THE NOTE IS CORRECTED RATHER THAN APPENDED TO. Its "BUILT AND HERMETICALLY PINNED" list claimed "the retirement"; that entry now reads "the KV retirement" and carries the falsification inline. A build-state list that survives its own disproof is worse than no list. FIVE CRITERIA HAVE NO DATA and are recorded as no-data, not as passes: no `PHASE-FLIP DONE` line was reached, so zero abandons and zero staging refusals are absence of evidence, NOT the funding claim C2 makes. Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release; holder -> holder.released-w27; cards 0/0/0 before and after; router PID 142 untouched (it is itself an sglang process, which is why a bare `pgrep -f sglang` matches it). `choom -n 1000` verified on the launcher AND all three rank PIDs before the weights load -- no OOM, so W26's failure mode did not recur.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…rse, and the GDN mover then retires by construction W27 killed the boot in `resident_mamba_slots` and the guard was RIGHT. This is the root behind it, and it is not the GDN mover. THE ROOT. `retract_all` frees a request's KV rows, its mamba slot and its tree lock ref -- and the scheduler's batch structures keep REFERENCING the `Req`. `_live_reqs` is the one authority for "who is resident" and reads exactly four places: every `running_mbs` slot, `running_batch`, `last_batch`, and the out-of-batch `chunked_req`. Retraction touched none of them, so every seam consumer after it was handed a live request whose resources were gone. The GDN mover was simply the first to look. FREEING A RESOURCE AND RETIRING THE REFERENCE TO IT ARE DIFFERENT JOBS. Doing only the first leaves a live object that every reader has to special-case, and the next reader added reintroduces the bug. Same shape as sgl-project#731's fix, where the carry had to CONSUME the queue entry rather than leave one request counted in two places. `consume_retracted_from_live_universe` retires the reference out of all four, using `filter_batch(keep_indices=...)` and NOT a raw `.reqs` edit -- a batch carries per-request tensors beside the list and a list edit desynchronises them. Pinned by a test that asserts filter_batch was the mechanism, because the raw edit is the tempting shortcut and it fails silently, later. FIXED AT THE AUTHORITY, AND THE SWEEP SAYS THAT IS ENOUGH. Every seam reader of the live set goes through `_live_reqs`: resident_mamba_slots (gdn_flip_mover.py:617), the KV enumeration (:835), the sgl-project#822 census (:1431), :5457, the output trace (phase_flip_output_trace.py:266), and the release itself (:8129). Grepping the seam modules for direct running_mbs / running_batch / last_batch / chunked_req reads returns comments and docstrings ONLY -- no live code bypasses the authority. THE GDN MOVER NOW RETIRES BY CONSTRUCTION, WHICH IS THE POINT. W27's no-retry refused dropping `GdnFlipMover.move()` because doing that with live linear state trades a loud crash for SILENT linear-state loss. Downstream of this fix the trade is gone: with the live universe consumed and the tree dropped, both halves of `flip_mamba_slots` -- resident slots UNION tree checkpoints -- are empty, so the mover moves nothing. No deletion; the same way the KV mover was retired by emptying its input. THE ORDER IS THE SAFETY PROPERTY, and it is pinned against the REAL guard, not a stand-in: `resident_mamba_slots` no longer refuses AFTER the consume, STILL refuses without it, and a genuinely resident request still yields its slot. So the guard is SATISFIED, never weakened -- if that middle test ever stops raising, the silent-loss trade has been made after all. W27's CONFIRMED RESULTS, kept out of the FAIL headline's shadow: * C6 PASS ON METAL: the sgl-project#825 three-rank crash did NOT occur with the prefix tree dropped under a live request -- 0 `NoneType ... has no attribute 'id'`, 0 `dec_lock_ref`. The retract-before-reset order is vindicated on hardware, not only in its hermetic reproduction. * `choom -n 1000` held on the launcher AND all three rank PIDs before the weights load; no OOM, so W26's failure mode did not recur. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_retracted_leaves_live_universe_856.py 15 passed incl. the W27 specimen reproduced (a freed request still enumerated), all four live-universe routes, the filter_batch mechanism, seam-safety (no targets / no match / a refusing filter_batch / a bare scheduler), and the three real-guard derivation tests. ruff check + format clean on phase_flip_runtime.py Full window record: /spinning/gpu-arb/W27-RESULT.md
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
… tree drop leaks 152 rows
The root fix HOLDS. 9 flips, both directions, and the two things that killed
the first attempt did not happen once:
has no mamba slot 0 (W27: killed all three ranks)
NoneType ... attribute id 0
dec_lock_ref 0
`resident_mamba_slots` never fired. The live-universe consume removed it and
the GDN mover retired BY CONSTRUCTION, exactly as derived -- the derivation is
now confirmed on hardware and not only against the real guard in a unit test.
C1 IS PROVEN ON METAL, which is the headline this ticket exists for. Every
`PHASE-FLIP DONE` line, all ranks, both directions:
0 live slots, sent 0 cells / 0.00 MiB, received 0 cells / 0.00 MiB
against W25's 116502 live slots and 995.31 MiB sent on PP0.
TWO NEW COSTS, MEASURED RATHER THAN ESTIMATED, and both are mine:
`hicache_quiesce->resident_release` is 753.6 ms (the retract+consume+drop), and
the wave loop still walks 16 EMPTY waves for ~314 ms -- the price of retiring
the mover by emptying its input instead of deleting it. That number is what
turns "should we delete the dead loop" into a decidable question.
THE NEW DEFECT, NAMED:
ValueError: pool memory leak detected! [full] total=472864,
available=126802, evictable=22, protected=0, ... withheld=345888
126802 + 22 + 345888 = 472712 against 472864 -> 152 rows belong to nobody, and
it accumulates per cycle (fired on the third retract+drop). The retraction
frees the RESIDENT REQUESTS' rows; the prefix tree additionally holds CACHED
PREFIX rows, and `tree_cache.reset()` rebuilds the root and ORPHANS them
instead of returning them to the allocator.
Before sgl-project#856 the tree was never reset at the seam -- sgl-project#825 withdrew exactly that
action -- so this is NEW and it is mine. Attribution checked rather than
assumed: W25 ran 33 cutovers on this rig with no such error, and the broken
load driver cannot be the cause because its requests failed to connect and
allocated no rows.
NO RETRY. The fix means freeing the tree's own rows at the drop, which means
choosing the right API (`flush_cache` is a wipe, `reset` is not) and PROVING
it returns rows to the allocator. Guessing a tree API at a live seam is how a
loud leak becomes a silent double-free. Red-first at the desk, against this
leak arithmetic; it gates the next window.
OPERATOR HONESTY: my `load_w27r.sh` was broken -- curl exit 7, `code:"000"`,
spinning to round 640 in 45 s. So the "responses" it logged are FAILURES, C2
and C7 were never exercised under real load, and the zero abandons above are
NOT the funding claim. The staging-limiter question I left open in the funding
inventory therefore stays open. The 9 flips came from the boot's own warm-up
traffic.
Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release;
holder -> holder.released-w27r; cards 0/0/0; router PID 142 untouched;
`choom -n 1000` held on the launcher before the weights load, no OOM.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
… was in the tree ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict: prefetch_ballot.prefetch_done_under_ballot returns the local value when the ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on tp_cpu_group -- a group the PP loop never passes, and which has world 1 under pp_size>1 in any case. The ranks therefore decide independently. scheduler.py :9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream -- add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 -> cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix PP1's and PP2's never received. From there it is self-sustaining. PP0 matches its own tree and offers told=1250; PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the prefix by running the batch it is refused for lacking the prefix. WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each new retraction for THAT rid lowers THAT rid's floor. The argument is sound and silent about the POPULATION. The shortfall is a property of the TREE, so every fresh rid over the same prefix starts unclamped and buys its own voided pass -- six distinct rids in one second, all told=1250 local=0, which is why _learned_floor was measured RUNNING and LOWERING on PP0 and never bound. FIX: the same actuator, scoped to the prefix the offer was made over. No new mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache ack-count reduction) rules out a collective on this path, and a group-uniform INSERT is structurally impossible: the radix value is a tensor of that rank's own KV slot ids, dereferenced as real memory by four consumers (allocator free at unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at :2284, write-back into req_to_token at :1325), and the GDN mamba component donates a live state slot a non-computing rank does not have. offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens -- NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length is mixed in so a prefix and its extension cannot share a floor. prefix_key=None leaves the pre-sgl-project#963 rid-scoped path byte-identical. PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces available_size(), a different quantity, and is switched OFF in the PP phase (tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here. sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477 with require_armed_and_parked=True, so it samples only at an armed flip -- which the livelock prevents. That is this instance's compensator-reachability gap. RED-FIRST, and the first red was rejected as worthless: all six cases failed on the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against the finished fix and their errors READ: consultation withdrawn -> assert 6 == 1 (boot 2's six voided passes, exactly) learn withdrawn -> assert 6 == 1 clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix) key-blind clamp -> assert 0 == 1250 (loss on a prefix nobody reported) The last two are the danger direction: this fix must never discard a prefix every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC prefix, and clears the moment the group serves that prefix, so the cost is one voided pass once rather than one per rid for ever. Double-prefill law: the requeue loses 0 today and still loses 0; what changes is that it terminates. An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the stakes are strictly higher than for the rid scope, since a floor invented from a number nobody measured would cap every request over that prefix. The sgl-project#944 consumer ratchet CAUGHT this file on its first gate run and is now registered. NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), plus 14 new tests green count probe: 2 named == 2 summary, SUBFAILED and ERRORS included ruff clean on both touched files. NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring lap to carry a prefetch-pending fact home, which is its own posten. This makes the system self-healing against a divergence however caused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.