Repository navigation
Move sampler into CUDA graph - #1201
Merged
Merged
Conversation
hnyls2002
marked this pull request as draft
August 25, 2024 01:07
hnyls2002
marked this pull request as ready for review
August 26, 2024 08:00
Collaborator
Author
|
For some unknown torch compile bugs, it is fixed in a strange method sglang/python/sglang/srt/layers/sampler.py Lines 58 to 60 in 7378d09 Similar issues: |
merrymercy
approved these changes
Aug 26, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Co-authored-by: Yineng Zhang <me@zhyncs.com>
AniZpZ
pushed a commit
to AniZpZ/sglang
that referenced
this pull request
Apr 14, 2025
AniZpZ
pushed a commit
to AniZpZ/sglang
that referenced
this pull request
Apr 14, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…lders; the cutover moved one WHAT WAS RED (parent a890b8a, test_1201_phase_stamped_handles.py): 13 failed, 3 passed. The 3 green are the default-path parity cases (an unowned cache still answers with its constructor pool; no counter to carry is not an error; a plain ReqToTokenPool without the hooks is not an error). After the cut: 16 passed; with the two sibling files (test_1040_req_pool_per_phase, test_cutover_participants_859) 82 passed. THE DEFECT. ReqToTokenPool is stamped onto four holders at CONSTRUCTION and rebind_req_pool_for_cutover moves exactly one, the scheduler's: * UnifiedRadixCache.req_to_token_pool (unified_radix_cache.py:429) * the pool's tree_cache back-reference (memory_pool.py:1871), whose only two bind_tree_cache callers are tree-cache constructors (unified_radix_cache.py:514-515, mamba_radix_cache.py:502-503), so the incoming pool keeps tree_cache=None for the whole phase and the evict-then-retry at memory_pool.py:2019-2024 never arms -- the sgl-project#581/sgl-project#773 regression that unified_radix_cache.py:507-513 was written to close * the pool's layer_transfer_counter (memory_pool.py:1876) * FutureMap.pool (overlap_utils.py:339), NOT moved by this cut Rank-uniform, so no ballot, digest or MIN can catch it: every rank is wrong the same way. Both phases' pools hold the same row count, so the divergence lands IN RANGE. Loud half: common.py:1849 -> free_slot's double-return refusal (memory_pool.py:492-497). Silent half, which runs FIRST: common.py:1836 reads the wrong pool's req_to_token row and hands those indices to token_to_kv_pool_allocator.free(). THE CUT. (a) UnifiedRadixCache.req_to_token_pool is a read-at-use property onto a registered owner, following kv_session_offload.py:2921-2933, whose docstring names this exact hazard. Unbound = byte-identical fallback to the constructor pool, so no non-flip boot changes. (b) _restamp_phase_handles re-stamps tree_cache, the pool's back-reference and the layer-transfer counter onto the incoming pool at the seam. (c) assert_req_pool_identity refuses at the end of the cutover (phase_flip_runtime, after the sgl-project#719 HiCache rebind, outside its try/except) when any holder still names a different pool. (d) cutover_participants: two new REGISTRY rows (request_pool_phase_ownership, req_pool_back_references) and req_to_token_pool added to MUTATED_STATE -- a HANDLE the cutover REPLACES, next to ten quantities it recomputes. future_map is on the registry as an explained gap, not moved. BRIEF PREMISE FALSIFIED -- (c) as briefed cannot be built. "Turn the two silent returns at memory_pool.py:1995-1998 / :3992-3994 into refusals" would refuse the DEFAULT path. layer_transfer_counter is None on every boot without a hierarchical cache: it is only ever set by register_layer_transfer_counter, whose callers are all HiCache/cache- controller paths, and swa_memory_pool.py:124-125 registers None on purpose; register_layer_transfer_counter's own docstring calls mamba_transfer_frame =None "the historic no-wait behaviour, unchanged". So None is a legitimate steady state, not a defect. What IS a defect is the handle failing to cross the seam, and that is decidable where both pools are in hand. The refusal therefore moved to the seam and the carry was built; the join sites are untouched. ANCHOR DRIFT (three, all reported rather than worked around): * "phase_req_pool_binding.py:180 is the only runtime rebind" -- :180 is the def line at the tree; the assignment is at :230, moved by CUT C (a890b8a). devindex still prints :180 because its store is pinned one commit back. * memory_pool.py:3992-3994 -- the guard is at :3993-3994; :3992 is a comment line. * memory_pool.py:496 -- the enclosing method free_slot is right, but :496 is a fragment of the double-return message; the line every finish hits is :506 (free -> free_slot), and :492-497 is the guard. All other anchors verified verbatim at the tree: unified_radix_cache.py:429 /507-513/514-515, mamba_radix_cache.py:502-503, memory_pool.py:1871/1876/ 1995-1998/2019-2024, mamba_component.py:480, common.py:1780/1836/1847/1849, kv_session_offload.py:2921-2924, cutover_participants.py:92/314-325. MUTANTS (scratch copies in /tmp/wt1201-orig, all restored): 1 DANGEROUS DIRECTION -- bind_req_pool_owner stores None, i.e. the pre-cut cached reference, which answers wrongly instead of refusing. Killed by 4: test_a_bound_cache_follows_the_owner, test_the_cutover_leaves_cache_and_scheduler_on_one_pool, test_the_kv_read_sees_the_rows_the_running_phase_wrote (the silent one), test_a_clean_cutover_passes. 2 DANGEROUS DIRECTION -- bind_tree_cache re-stamp removed, so the pool's evict-then-retry stays disarmed silently. Killed by 2: test_the_incoming_pool_is_bound_to_the_tree_cache, test_a_clean_cutover_passes. 3 assert_req_pool_identity returns early (#505a shape). Killed by 2: test_a_cache_left_on_the_outgoing_pool_is_refused, test_a_pool_left_bound_to_nothing_is_refused. 4 counter carry dropped. Killed by 1: test_the_layer_transfer_counter_is_carried_to_the_incoming_pool. VERIFIED: red-then-green tallies above; ruff parity on all four touched files (unified_radix_cache.py had 1 pre-existing F401 typing.Sequence before and after, zero new); import smoke of all four modules; no external writer to a tree cache's req_to_token_pool and no __dict__ access to it (grep, none); UnifiedRadixCache has no subclasses (devindex symbol_profile), so the property cannot be shadowed. NOT VERIFIED: no boot. Whether the layer-transfer counter carry ever fires on the standing flip boot is UNMEASURED -- it depends on whether the assembler registers a counter on both stacks' request pools, which only a boot decides. MambaRadixCache.req_to_token_pool (mamba_radix_cache.py:498) is the same cached-reference shape and is NOT converted here. ANSWER TO THE COMPLETENESS CRITIC, recorded as asked: this registry axis covers HANDLES THAT MOVED -- objects the cutover replaces and every holder that cached one. It does NOT cover WHAT WAS STILL OWED at the arm: launched-unreaped passes, outstanding receives, ring slots, lap counters. Nothing here detects a debt; MUTATED_STATE's read-window axis and the new handle axis are both about identity, not about obligations in flight. That second axis remains unbuilt and unclaimed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…, in the phase that is about to end WHAT WAS RED (parent 643237b, test_1201_future_map_phase.py): 14 failed, 0 passed. Reasons verified individually, not inferred from the tally: ImportError on build_future_map; the registry entry's hook/probe still None; MUTATED_STATE has no "future_map"; the source-order test cannot find the call. After the cut: 14 passed. With the three immediate siblings (test_1201_phase_stamped_handles, test_1040_req_pool_per_phase, test_cutover_participants_859): 98 passed. THE DEFECT, and it is on the LIVE path. FutureMap carries TWO boot stamps. scheduler.py:820-823 parks a phase-flip instance's speculation at NONE because the boot phase is PP prefill and has no draft worker (sgl-project#631); init_overlap then builds the map out of that NONE (scheduler.py:2257) and nothing rebuilt it. overlap_utils.py:107 branches on future_map.spec_algo.is_none(), and the NON-overlap spec path reaches it -- scheduler.py:13174-13177, outside the "if self.enable_overlap:" above it -- so on the standing boot form (disable_overlap_schedule=True, speculative_algorithm='EAGLE') the frozen NONE is read on every TP decode round and gathers last iteration's sampled token into batch.input_ids on a batch whose input ids the spec worker owns. Nothing raises: _assert_nonneg_and_invalidate is gated on SGLANG_IS_IN_CI (overlap_utils.py:72), off on this rig. The second stamp is the request pool (req_pool_size, max_context_len, ConfidenceRelay.pool) -- the fourth holder of the sgl-project#1201 class, the one 643237b filed as an open gap. THE CUT. build_future_map(scheduler) extracts the construction out of init_overlap, duck-typed on five attributes so the cutover can call it without importing the scheduler module. phase_flip_runtime calls it at the seam and assert_future_map_identity(scheduler) beside assert_req_pool_identity. The registry entry future_map_req_pool_holder now names both obligations and "future_map" joins MUTATED_STATE on the HANDLE axis. ORDERING IS THE LOAD-BEARING PART. The rebuild must run after BOTH swaps: the algorithm swap (scheduler.spec_algorithm = want_spec_algo) and the request-pool rebind (rebind_req_pool_for_cutover, ~330 lines later). Built between them it carries the incoming algorithm and the OUTGOING pool. One test pins that source order, and mutant 3 is exactly that mistake. MUTANTS (scratch copies in /tmp/wt1201b3-orig, restored, md5 verified; grep MUTANT over python/sglang/srt/managers clean): M1 DANGEROUS -- build_future_map hard-codes SpeculativeAlgorithm NONE, i.e. the defect kept: answers wrongly, never refuses. Killed by 5, including test_the_frozen_none_takes_the_branch_it_must_not, which asserts the silent branch is NOT taken rather than that a guard fires. M2 DANGEROUS -- the probe's pool check compares row counts instead of object identity, the in-range mistake this whole class is made of. Killed by 1, test_a_stale_pool_stamp_is_refused. M3 ORDERING -- the rebuild moved to just after the algorithm swap, before the pool rebind. Killed by 1, test_the_rebuild_runs_after_both_swaps. M4 DANGEROUS -- the probe swallows a stale algo stamp. Killed by 1, test_a_stale_algorithm_stamp_is_refused. SIBLING SWEEP (analysis, not test code): every other reader of future_map reads it at use off the scheduler (kv_session_offload.py:3899/:4411/:6040, scheduler.py:13094, and the parameter-passing sites in disaggregation/ and speculative/). No fifth cached holder on this axis. WHAT I VERIFIED: hermetic CPU construction of FutureMap and of the whole suite (CUDA_VISIBLE_DEVICES="", serving venv); import smoke on both edited runtime modules; ruff parity -- 103 pre-existing findings in scheduler.py before and after, zero new, the new test file clean. WHAT I DID NOT VERIFY: nothing on metal. No boot, no GPU, no server. The cut is DESK-PROVEN only. The rebuild allocates fresh device buffers per cutover and that cost is unmeasured. The _confidence_budget_prepare handle next to the future map in init_overlap is phase-stamped by the same argument (it caches the draft worker's callable) and is NOT fixed here; it is gated on enable_overlap, which is off on the standing form, so it is latent rather than live. RETRACTED: test_1201_phase_stamped_handles's test_the_future_map_holder_is_registered_as_an_open_gap asserted the gap is OPEN. That was true one commit ago. Rewritten to assert the entry is complete, retracted rather than deleted so the sequence stays readable.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 4, 2026
…ee adversarial lenses found in the six cuts WHAT WAS RED, and every finding was re-verified at the tree before it was acted on. Two of the eleven reported were REFUTED and are named at the end. 1. sgl-project#1202 added `mbs` to the residency AUTHORITY and not to the retirer. `_live_reqs` walks `mbs`; `consume_retracted_from_live_universe` did not. That is the W30 asymmetry re-opened, and it is the W27 death: the release frees a request the authority then re-enumerates, and `resident_mamba_slots` (gdn_flip_mover.py:620) raises KvReshardError on a live request with no mamba slot -- inside the no-return region, where the only outcome is a dead rank. The mbs-only shape is ordinary, not exotic: under event_loop_pp a freshly planned prefill batch sits in `mbs[slot]` while `running_mbs[slot]` holds the decode set. RED: the property test "authority and retirer walk the same containers" reported authority-only = ['mbs']; the concrete test read "0 != 1 : the mbs route retired no reference". Pinned as a PROPERTY over the two walks, so the next route added cannot reintroduce it. The stale present-tense collateral at scheduler_pp_mixin.py:5121 ("_live_reqs does NOT enumerate mbs", dated 2026-09-04) is rewritten as the dated fact it now is. 2. sgl-project#1203 A5 spent three ESCALATION budgets in the group's abandon currency. A5's premise is right and is kept: a rank-local counter that zeroes itself when this rank's own objection clears can never reach its limit while ranks take turns objecting. But `flip_host_headroom_verdict` and `flip_seam_budget_verdict` do not merely stop deferring at the limit -- they return (allow=True, escalated=True), "PROCEEDING WITH EYES OPEN" under a failed sgl-project#721 host floor -- and the writeback arm proceeds with an incomplete sgl-project#703 fence. `_seam_abandons_in_a_row` is incremented on EVERY reduced-fit abandon or frame divergence, so three unrelated abandons disarmed all three guards and the next GENUINE shortfall escalated on its first firing, having deferred zero times. A bound may be widened to the group's currency; an escalation that permits the hazard may not. REPAIR: each guard is bounded by its own count again, and that count is refunded ONLY by a completed flip (beside the abandon book, whose own comment already states the rule) or by the guard's own escalation. The whole reset policy is `flip_defer_budget_after`, one pure function, so the three copies cannot drift again. RED: 8 of 10 A5 tests failed at the parent. Three instruments that named the wrong counter are corrected with it: "consecutive" becomes "in this abandon run", and the host-RAM defer is booked BEFORE its own log line instead of after. 3. sgl-project#1204's abstain discarded a term the rank had successfully read. Both reads shared one `try`, so a rank whose `backed_rows()` succeeded and whose `live_floor_rows()` then raised contributed `_UNBOUNDED_ROWS` in slot 0 -- the MIN identity, i.e. "I am not the poorest". Slot 0's MIN becomes the level the peers cap to and the ceiling `book_deferred_grow` clamps to, so erasing a poor rank makes the group agree an id level it cannot map, which `backed_rows`' docstring calls a cudaErrorIllegalAddress that kills every rank. The hang the commit removed would have been traded for a wrong group answer -- the corrupting direction bought to avoid the refusing one. REPAIR: abstain PER SLOT. Each slot carries this rank's own measurement or that slot's neutral, unchanged for a wholly unreadable relief. RED: "30 != 1099511627776". 4. sgl-project#1205's `seam_refused` measured a population its label does not name -- the sgl-project#1205 defect class, committed inside the sgl-project#1205 fix. The fence read `seam_refusals()` with NO direction, summing "load" refusals raised on the controller's own background threads while the fence runs, and printed the delta as "of refused_silently, the share the FLIP SEAM refused". REPAIR: `seam_refusals("write")`, and the log line and the field comment now say what it is: process-wide write-direction refusals during the fence, not a subset of `refused_silently` and not per-node (consume_gate refuses a whole queue and counts one). RED: "1 != 3" when the direction-less read is put back. 5. `assert_future_map_identity` was REACHED and CANNOT-FAIL. Measured: four combinations of (algo NONE|EAGLE) x (pool pp|tp), ZERO fires. `create_future_map` passes `self` as `spec_algo` and the scheduler's pool straight into `ConfidenceRelay`, so both stamp arms are construction invariants at a call site 57 lines after the rebuild; the two suite tests that are green hand-mutate a map no code path produces. REPAIR: the registry's obligation is "a probe proving the rebuild ran", so the probe now takes `previous` -- the map held BEFORE the rebuild -- and refuses when the rebuild did not replace it. A memoised build, a deleted assignment or a rebuild moved out of the cutover all land there and none of them are visible to the stamp arms. The tautology is pinned as a fact so the green is not read as coverage. 6. MambaRadixCache got the sgl-project#1201 assertion but not the sgl-project#1201 fix. It caches `req_to_token_pool` as a plain attribute and has no `bind_req_pool_owner`, so `_restamp_phase_handles`' hasattr guard skipped it and the handle stayed on the outgoing pool -- and `assert_req_pool_identity`, added to both classes, then raises ReqPoolRebindRefused at the END of every cutover. Reachable: registry.py hands back this class for a hybrid-SSM model WITHOUT hierarchical cache. The boot went from flipping wrongly to not flipping. REPAIR: the same owner property and bind hook UnifiedRadixCache got. 7. The sgl-project#1203 A1 group vote moved a non-pure function onto a new cadence. `seam_transport_premise_holds_locally` writes the edge-triggered `_seam_premise_refused_announced` latch and emits SEAM TRANSPORT REFUSED; the vote runs once per loop iteration while the consumer returns at `_active_phase != PHASE_TP` before it asks. The latch was therefore set in the PP phase, swallowing the gate's own first engagement in TP. REPAIR: `announce=False` for the vote; the gate's own read still announces. Also: the anchor at overlap_utils.py:81-83 had drifted onto the line that REFUTES its sentence -- scheduler.py:13174-13177 is now the overlap call inside `if self.enable_overlap:`, while the non-overlap spec branch is the sibling `elif not batch.spec_algorithm.is_none():` arm. The claim is true; the number sent the next reader to the wrong branch. Named by branch, no number. WHAT KILLED WHICH MUTANT (planted in place, restored by sha256): M1 retirer over-clears (frees a row a live req owns, the corrupting direction) -> test_an_mbs_only_resident_is_retired_by_the_retraction M2 host bound back on the group abandon book (disarms sgl-project#721) -> test_the_host_headroom_bound_reads_its_own_defer_count M3 a cleared vote refunds its own budget (the A5 wedge) -> test_this_rank_clearing_does_not_refund_the_budget + test_a_taking_turns_run_still_reaches_the_limit M4 refund moved back into the gate's cleared-vote branch -> test_a_completed_flip_returns_every_budget_beside_the_abandon_book M5 sgl-project#1204 discards a measured backing again (claims not-poorest, the fleet-death direction) -> test_a_readable_backing_reaches_the_group_min M6 MambaRadixCache loses the rebind hook -> test_both_classes_expose_the_rebind_hook + test_the_handle_follows_the_owner_and_falls_back_without_one M7 the seam stops handing the probe the outgoing map -> test_the_seam_hands_the_probe_the_outgoing_map M8 the A1 vote announces again -> test_the_vote_leaves_the_announcement_latch_alone 8 planted, 8 killed, 0 survived. VERIFIED: 146 passed + 7 subtests over the six touched test files plus test_1157_seam_witness.py and test_seam_census_double_905.py (the joint pin over both sgl-project#1202 functions), hermetic, CUDA_VISIBLE_DEVICES="" with the serving venv. Import smoke over all seven edited modules. ruff parity with the parent on the touched files: 31 findings both sides, all pre-existing. NOT VERIFIED: nothing on metal. No boot, no GPU claimed, no collective. The sgl-project#1204 and A5 repairs are DESK-PROVEN only -- neither has run against a real reduce or a real host-RAM shortfall. REFUTED, and left untouched: (a) `assert_req_pool_identity`'s tree-cache arm is not decorative -- it is a construction invariant for UnifiedRadixCache BECAUSE the owner property is the fix there, and a live check for every other holder class, which is exactly what caught finding 6; (b) `seam_refused` was already read as a delta across the fence run (hicache_flip_writeback.py:700), not as an absolute, so only its direction and its label were wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Checklist