Repository navigation
Update README.md - #792
Merged
Merged
Update README.md#792
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
amote-i
pushed a commit
to amote-i/sglang
that referenced
this pull request
Dec 8, 2025
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.ai to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. --> ## Benchmarking and Profiling <!-- If this pull request impacts inference speed, provide benchmarking and profiling results. --> ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…instead of the group minimum Boots instr10, instr12 and instr13 all died in round 6 or 7 of a flip drive with an allocation that eviction could not pay. sgl-project#790 made the failure name itself: EVICTION UNDER-DELIVERED: asked for 512 tokens, the pool received 94. The tree still reports 67674 evictable tokens. A RESIDENCY CAP IS ENGAGED and is holding 63641 slot ids out of the allocator's free list. WHICH CAP, settled by arithmetic rather than by inspection. Above the shrink's cap of 137233 the id space holds only 161792 - 137233 = 24559 ids in total, so a confiscator holding 63641 cannot be that one. Above the LEVELLED cap of 40960 it holds 120832, and 63641 fits. The cap that kills was installed 95760 rows BELOW the highest live row (136720), four seconds after the cutover: 05:28:17 KV-BACKING released 160 MiB by backing 137233 rows instead of 161792 (highest live row 136720, 24145 ids withheld) 05:28:21 KV-BACKING cap agreement: exposed rows 137233 -> 40960 (group level 40960) 05:29:28 RuntimeError: Out of memory ... the pool received 94 ... holding 63641 slot ids ROOT. phase_flip_spill.recover_kv_backing (:1167-1173) reduced only [backed, -backed] and levelled every rank to the group's minimum BACKED ROWS via level_recovery_to -> reconcile_to (kv_backing_relief.py:2224+), with no reference to the live set at all. The cap therefore lands under the rows the radix tree still holds, KvRowCap._apply (:490-512) confiscates every id the peel frees above it, and the pool can never be paid again. The law was already written one function away: collective_cap_target (:227-230) returns None when the group's MAX floor exceeds the MIN capable. The recovery levelling was the one path in the chain that never made that trade. FIX. New public live_floor_rows() (:2113) exposes the reading the levelling needed and had no way to ask for; reconcile_to declines and logs when level < floor instead of engaging (:2260-2325); recover_kv_backing's payload widens to [backed, -backed, -floor] so the group can make collective_cap_target's trade, and declines loudly when the group's MAX floor is above the MIN backing, a truncated payload included. WHY NOT THE TWO OBVIOUS ALTERNATIVES. Releasing the cap at the cutover re-admits ids over pages the shrink genuinely unmapped -- the cudaErrorIllegalAddress that reverted c4e5579, quoted in _shrink_to. That trades an OOM which raises for a fault that kills every rank without raising. Refusing the carry is mis-ordered: the carried slots are not seeded, they are already live, so refusing them drops a live request's KV -- and on this specimen the cap is engaged AFTER the cutover by the recovery hook, so there is no carry left to refuse. WHAT THIS DOES NOT FIX, stated plainly. The levelling now DECLINES where it used to kill, which means divergent id spaces and flips refused by the frame ballot -- lost flips and reduced admission, never a dead rank. Getting those flips back needs PP2's capacity deficit funded: measured on instr13, PP2 alone runs a shortfall (need = floor 3034 + delta 256 + want 686 = 3976 MiB against free 3210 MiB -> deficit +766 MiB) while PP0 and PP1 sit on surpluses of 4644 and 3408 MiB. That is a planner item, tracked in evidence-665-f1/BUG_planner_corridor_capacity.md, and no hand pin is applied here. TESTS. test/registered/unit/mem_cache/test_residency_cap_flip_levelling_792.py, new, 12 tests on a real KvRowCap, a real TokenToKVPoolAllocator, a real KvBackingRelief and the real recover_kv_backing; only the VMM arena is a stub. fix reverted -> 9 failed / 3 passed, reproducing the specimen text verbatim behavioural neuter of the reconcile_to guard alone -> 3 failed / 9 passed, 0 AttributeError with the fix -> 12 passed; with neighbours 656 and 790 -> 46 passed TheLevellingStillHappensWhenItIsHonest passes in BOTH directions on purpose: it goes red if the fix disables sgl-project#656 C22-e instead of bounding it. ruff, codespell and black clean on all four files; the 2 test_black_ratchet_656.py failures are pre-existing at HEAD.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 21, 2026
…eriving it (slice 1) Ten fixes on this branch (sgl-project#757 sgl-project#789 sgl-project#790 #791b #791c sgl-project#792 sgl-project#795 sgl-project#797 #797b #797c) all have ONE form: each rank RE-DERIVES the pass schedule -- rid set, chunk length, prefix length -- locally from its own state, and a phase flip invalidates that state non-atomically. Every "new root" was the next consumer of the same re-derivation, so the list grew instead of converging. This stops patching consumers. THE DATUM WAS ALREADY ON THE WIRE AND NOBODY READ IT. PPAdmissionEntry.extend_len (pp_admission_congruence.py:177) has crossed the wire since sgl-project#791's first commit with exactly three consumers: to_wire (:223), from_wire (:244), one log string (:824). Nothing ever built a batch from it; reconcile_pp_admission_decision returns Dict[rid, prefix_len] and drops the second number on the floor. THE MECHANISM, CORRECTED AGAINST THE LOG rather than assumed. boot_instr20.log:5171,5181-5183: PP0 ADMIT rid=6cbe2733 prefix_lens=0 chunked=1 -> 512-row chunk PP1 ADMIT rid=6cbe2733 prefix_lens=512 chunked=0 -> 333-token remainder PP1 DID receive prefix_len=0 and DID clamp prefix_indices to it -- scheduler.py:7026-7027 worked. Then add_one_req's HOST LOAD-BACK put the 512 back: needs_host_load_back() went true when the HiCache prefetch landed and schedule_policy.py:1539-1549 concatenates the recovered indices. "MAMBA-HOST-RESUME ... triggers load_back" appears on PP1 and PP2 and is ABSENT on PP0 -- that asymmetry is the bug. 845-512=333 then fitted rem_chunk_tokens whole, so the NON-chunked branch fired. The re-derivation lives INSIDE THE ADDER, after the schedule was already applied, and it re-derived BOTH numbers. DESIGN. forwarded_schedule() (pp_admission_congruence.py:500) is the pass geometry as a value: rid -> (prefix_len, extend_len) for exactly the rids `effective` names; None or a voided decision yields {}. _add_scheduled_req (schedule_policy.py:1237) EXECUTES both numbers: no rem_chunk_tokens, no page/align rounding, no host load-back, no budget veto -- the budget is still charged. The gate sits above every local veto (:1600), and add_chunked_req (:1328) gains the gate it never had at all (it is entered from scheduler.py:7004, BEFORE the admission loop). The three membership vetoes that silently narrowed -- batch_is_full (:7035), the HiCache prefetch_done skip (:7047, the instr20 race itself) and the LoRA gate (:7024) -- become refusals (scheduler.py:7190). REFUSAL IS CONTROL FLOW, NOT A RESULT CODE. PPScheduleRefused (:163) is an exception because every AddReqResult means "build a batch without this request", which is precisely the corruption. A refusal reuses sgl-project#797 end to end (_pp_refuse_forwarded_schedule, scheduler.py:6333 /:6369): sets _pp_admission_pass_voided, voids the forwarded decision, re-notes the slot expectation. No new mechanism. Inside the loop a refusal is CARRIED, not thrown, so alloc_group_end() still runs (:7025, :7118, :7169). WHY THE GUARDS CAN NO LONGER FIRE, each owed a reason: sgl-project#631's _want is extend_num_tokens, which now comes only from the forwarded extend_len with load-back suppressed, so it IS the upstream's row count by construction rather than by agreement (green arm: rows=512, batch_tokens=512). sgl-project#757/sgl-project#787 stamp and sgl-project#795 epoch were already structural; what changes is that they can no longer be correct-but-insufficient, as instr17 and instr20 both were -- every identity right, only the width wrong. Width is now an identity too. sgl-project#789 needs a membership divergence, which is now identical-or-refused. #791c's tripwire detects a self-narrowed batch, and no path creates one. HONEST CORRECTION TO THE SUBSUMPTION CLAIM: retraction does NOT become unnecessary. Physical impossibility is real -- a rank genuinely lacking KV for [local, told) cannot execute. What becomes structurally impossible is the NARROW-THEN-DETECT shape: no code path is left that builds a batch of a geometry the upstream did not name. NOT COVERED BY THIS SLICE, stated so nobody assumes otherwise: - BATCH ORDER. can_run_list follows the local waiting_queue order, the decision follows PP0's. Same rid set in a different order gives EQUAL WIDTHS and permuted rows -- silent. Covered by the full design (execute in decision order), not by this slice. - Decode batches: retract_decode (scheduler.py:7489/:7517) mutates long-lived state on a tp_cpu_group reduce that is world=1 under TP=1/PP=3. Different root, filed by sgl-project#797. - Radix eviction divergence, KV pool sizing, spec-decode draft schedules. NAMED RESIDUAL, filed at the site (scheduler.py:7118) in sgl-project#797's practice: a request admitted earlier in a loop that later refuses has taken a persistent inc_lock_ref, released on batch completion. Undoing it needs the exact IncLockRefResult (SWA/Mamba tombstone params) the adder does not keep, and a blind release makes the one thing a mismatched release worsens. Bounded: reaching that line takes a genuinely unexecutable geometry and kills the pass. DEFAULT PATH UNTOUCHED: _pp_scheduled_extents() returns None on PP0 and on every pp_size<=1 boot, so scheduled_extent_for returns None and both adders take the pre-existing arithmetic unentered; PPScheduleRefused is unraisable there. Pinned by test_no_mapping_is_the_untouched_default_path and corroborated by 56/56 on the schedule_policy neighbours. TESTS. test_pp_forwarded_schedule_791.py: 17 passed (3 live gloo arms + 14 pure), 92 s. test_red_without_the_forwarded_geometry_instr20_reappears -- can-fail, rebinding ONLY the fix's return value in the child: batch=(512,333) rows=512 mismatch=True, byte-identical to instr20 PP1 09:40:30 test_green_the_forwarded_geometry_survives_the_mid_pass_prefetch -- batch=(0,512) rows=512 mismatch=False, load_back_calls=0 test_an_impossible_geometry_raises_rather_than_narrowing -- the architectural property Neighbour set (791/791b/791c/797/631 x3) re-measured AT HEAD: 11 failed / 75 passed; after: 11 failed / 92 passed, failure names byte-identical, zero regressions, +17. schedule_policy neighbours 56 passed / 0 failed. ruff 119 = 119 at HEAD (parity), ruff format clean on everything authored, codespell identical.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…se that could not fail is gone BOOT 17 AND BOOT 18 DIED OF THIS, at the first genuine idle, rank-uniform: `ValueError: pool memory leak detected! [full]` from `on_idle`. The admission wedge that came with it was a symptom -- 99 ADMITs still happened in the same window -- and the timeout that ended the load driver did not cause the death, it EXPOSED it by producing the first idle the boot ever had. THE CHAIN, MEASURED, five boots and no counter-instance (loadback-genuine / orphan-events / max-unaccounted): 1043b 0/0/0, 1043c 0/0/0, 1046cut 3/18/54626, 1048fix 69/21/43803, 1049n9 30/6/21608. Zero load-backs -> zero orphans -> zero ratchet. The orphan sizes are LITERALLY the load-back prefix lengths (5834, 4618, 350), rank-uniform on PP0/PP1/PP2, and the last census `unaccounted=` before death is identical to the size of the killer's `leaked_full_pages` set (21608 = 21608, contiguous 1..21608). An identity, not a plausibility. So this is not a regression of the load-back. It is sgl-project#938 -- registered open since 2026-08-27, Stage A (instrument) shipped as 226d34e, Stage B never built -- made FATAL because the load-back finally works: whole 4618-token prefixes are locked at the drop where the register's pre-registered fork expected ~283 rows. That fork is hereby answered from metal, 16x larger. WHAT CHANGES * `UnifiedRadixCache.reclaim_rows_for_drop` returns every row the tree still holds, locked ones included, and `drop_prefix_tree_returning_rows` calls it before `reset()`. `evict` refuses locked nodes and `_reset_full` zeroes the protected book without freeing a row, so those rows were leaving with nobody owning them, one drop's worth per flip cycle. * THE sgl-project#938 PREMISE IS NOW CHECKED, NOT ASSUMED. sgl-project#938 refused to free because a lock here is supposedly an in-flight write-through, and freeing under a live reader is a use-after-free in the sgl-project#913 family. Correct -- so the reclaim runs ONLY when `ongoing_write_through` is empty. On boot_855_1049n9 all 13 `sgl-project#792 post-retract writeback fence` lines report `outstanding=0`, including the two drops that orphaned 5834 and 4618 rows: the in-flight explanation held on no drop of that boot. When a copy IS outstanding the reclaim refuses and prints the count -- today's behaviour, kept. * NEVER FREES BLIND. Candidates are differenced against the allocator's own enumerated free set (`read_free_rows`, the union the idle invariant and the sgl-project#822 census already share). A non-enumerable reading refuses outright: a watermark allocator can say HOW MANY rows are free and not WHICH, and freeing against a count would be inventing membership. Double-freeing a row is silent corruption -- the one outcome worse than the leak. * The free is addressed through `FullComponent._free_full`, which resolves the allocator per access (sgl-project#941): a captured binding lands the rows on the other phase's free list as duplicates, the same symptom with a different root. * THE DEFERRAL IS INSTRUMENTED UNCONDITIONALLY. `sgl-project#1050 CUTOVER ROW RECLAIM` prints reclaimed/full_rows/mamba_slots/full_held/mamba_held/already_free and the refusal reason on every drop, including the all-zero reading. A clearer that can silently never run is how the same ratchet comes back one level up in a new pocket. * sgl-project#919's EXCUSE COULD NOT FAIL, AND IS GONE. The audit reported "pp_stack_allocator owns ids [1, 578995) and covers every sampled row" as the explanation for 43803 unowned rows, against a census id space of 578994 -- a containment test true for every sample that could ever be drawn. It excused a real, monotone, ultimately fatal loss. `OwnerCandidate` now carries `discriminating`, set false when the range is not narrower than the census's own, and such a candidate can no longer close the question: the new `SECOND-POOL-SPANS-EVERYTHING` verdict says the block is NOT excused. CHECKS (matched to this edit's failure classes, not a battery) * devtools/check_1050_cutover_reclaim.py, 17 cases, all pass: the double-free guard (an already-free row is not re-freed, and the count is reported), the outstanding-write-through refusal with its number, the non-enumerable refusal, root never harvested, and the no-mamba lane. py_compile is structurally blind to all of these -- `component_data` is a LIST indexed by the ComponentType enum, not a dict, and a `.get` there would have raised on every drop, on metal, mid-flip. * devtools/unaccounted_watchman.py --selftest, 4 cases, all pass, INCLUDING a replay of the real boot-18 log. That case earned its place: the first version keyed on the rank alone, passed its synthetic staircase, and read SILENT on the very boot it was written for, because the census emits four readings per cycle and the PP-layout zeros reset the streak. Keyed on the census family it fires. A watchman validated only against its own fixture is not validated. * ruff clean, py_compile clean on all three modules. NOT YET BOOTED -- desk-proven only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…t waited for a chunk the gate refuses to build BOOT-PROVEN EDGE (boot_855_1050dev_0840f82601_0831_115825, the boot 20 livelock). The open question of the handover was WHY the broken chunk never completes before the 30 s abandon. It is not a race: inside the armed window 12:22:19-12:22:49 PP0 logged ZERO `Prefill batch` lines and 15 `Decode batch` lines. The chunk was never attempted, and the re-arm is in the SAME SECOND as the abandon (12:22:49), so no unarmed window existed in which it could be. N9's remaining candidate (abandon deadline vs chunk completion time under growing backlog) is therefore FALSIFIED. THE BINDING TERM, with file:line. `prefill_blocked_here` (phase_purity.py:1087-1095) grants the sgl-project#887 one-chunk exception only when BOTH `prefill_allowed_in_tp_now(spent)` AND `tp_compute_fits_in_one_chunk` hold -- deliberately, so the gate and the sgl-project#838 detector permit the same batches (REGISTER_OPEN_876 on sgl-project#887). `tp_compute_fits_in_one_chunk` is `0 < pending < chunked_prefill_size`. The last grant was 12:22:11 with new_tokens=3392 against a 4096 cap; the backlog then crossed the cap and never came back (12347 -> 48866 -> 101075 tok), so the gate refused on every later round, SILENTLY, at its final `return not _relaxed(...)`. The grant is spent AT the grant (`_spend_tp_compute_chunk`), so a refusal on the `fits` term leaves the budget untouched. The hold (`prefill_runnable_in_current_layout`, phase_flip_runtime.py:156) read ONLY that budget, kept answering "TP can still make progress", and `chunk_blocks_quiescence` kept the rank un-quiescent for ever: PP0 armed 29 / committed 4 / abandoned 48, `this rank ready=0` in all 72 abandon lines, backlog 4618 -> 101075 tok. CLASS: a hold predicate that predicts another gate's decision in a different currency. SIBLING SWEEP over the three consumers of the sgl-project#887 grant: the gate itself reads both terms; `maybe_arm_phase_policy` (sgl-project#942, scheduler.py:13611- 13632) reads both terms and says so in its own comment; the flip hold read one. The fix makes the third consistent with the two that were already right, by passing the gate's OWN side-effect-free probe rather than a second copy of it. FUTURE CHECK: a new reader of `tp_compute_budget_remaining` that makes a runnability claim must also read `tp_compute_fits_in_one_chunk`. AND THE OPERATOR'S CONDITION, which turns this from a deletion into a replacement. Once the hold correctly reports "this layout cannot finish the chunk", the cutover commits over the broken chunk. That is licensed by sgl-project#988 riding the prefix across (loss <= 1 chunk = the sgl-project#939 grant) ONLY while the COMPLETED pages are settled in the store. The old strict arm blocked on "incomplete"; it now blocks on the real hazard -- an unacked prefix -- read from the EXISTING sgl-project#792/sgl-project#703 fence (`ongoing_write_through`, the same counter sgl-project#1050's `reclaim_rows_for_drop` and `on_idle` gate on), never a second ledger. Unknown is refusal, never permission. The hold is bounded by the ack (milliseconds), with the 30 s park deadline still the outer backstop; no second clock. UNTOUCHED AND PINNED BY TEST: mid-admission (no pool row -> still blocks), non-strict (residual, still filed not solved), and pp_to_tp drain-and-flip (runnable HERE -> still blocks, which is what keeps all prefill in PP). The PP-microbatch-in-flight half has its own predicate and is not read here. DETECTION: `_why_not` returned one string for three different holds. It now names which term is holding, including the new unacked-prefix case. DESK CHECK (matched to the error class of this edit -- a control-flow inversion in a restructured predicate, which py_compile and import smoke are structurally blind to): devtools/check_1033_quiescence_edge.py, 24 cases, hermetic CUDA_VISIBLE_DEVICES="", all pass. Includes a RED-FIRST arm that evaluates the PRE-FIX expressions on the measured boot-20 state (budget 2, pending 48866, cap 4096) and asserts they produce the wedge, and the POST-FIX expressions on the same state and asserts they commit only against an acked prefix. Plus compile + import + an order proof that both probes are assigned before either read inside get_next_batch_to_run. NOT YET BOOT-PROVEN. This is DESK-PROVEN at the predicate; the metal proof is boot 21.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove before doc is ready.