Skip to content

Move sampler into CUDA graph - #1201

Merged
merrymercy merged 14 commits into
mainfrom
sampler-cudagraph
Aug 26, 2024
Merged

merrymercy merged 14 commits into
mainfrom
sampler-cudagraph

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

Motivation

Modifications

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@hnyls2002
hnyls2002 marked this pull request as draft August 25, 2024 01:07
@hnyls2002
hnyls2002 marked this pull request as ready for review August 26, 2024 08:00
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

For some unknown torch compile bugs, it is fixed in a strange method

if is_torch_compile:
# FIXME: Temporary workaround for unknown bugs in torch.compile
logits.add_(0)

Similar issues:

pytorch/pytorch#129685

https://github.com/pytorch/pytorch/pull/124119/files#diff-809c39aeafb3acc92289f42a63e670a8719d4ce5627d5f88820142d80edf8d2a

@hnyls2002
hnyls2002 requested a review from merrymercy August 26, 2024 08:41
@merrymercy
merrymercy merged commit 75ce37f into main Aug 26, 2024
@merrymercy
merrymercy deleted the sampler-cudagraph branch August 26, 2024 14:02
zhyncs added a commit that referenced this pull request Aug 28, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Co-authored-by: Yineng Zhang <me@zhyncs.com>
AniZpZ pushed a commit to AniZpZ/sglang that referenced this pull request Apr 14, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…lders; the cutover moved one

WHAT WAS RED (parent a890b8a, test_1201_phase_stamped_handles.py):
13 failed, 3 passed. The 3 green are the default-path parity cases (an
unowned cache still answers with its constructor pool; no counter to carry
is not an error; a plain ReqToTokenPool without the hooks is not an error).
After the cut: 16 passed; with the two sibling files
(test_1040_req_pool_per_phase, test_cutover_participants_859) 82 passed.

THE DEFECT. ReqToTokenPool is stamped onto four holders at CONSTRUCTION and
rebind_req_pool_for_cutover moves exactly one, the scheduler's:
  * UnifiedRadixCache.req_to_token_pool (unified_radix_cache.py:429)
  * the pool's tree_cache back-reference (memory_pool.py:1871), whose only
    two bind_tree_cache callers are tree-cache constructors
    (unified_radix_cache.py:514-515, mamba_radix_cache.py:502-503), so the
    incoming pool keeps tree_cache=None for the whole phase and the
    evict-then-retry at memory_pool.py:2019-2024 never arms -- the sgl-project#581/sgl-project#773
    regression that unified_radix_cache.py:507-513 was written to close
  * the pool's layer_transfer_counter (memory_pool.py:1876)
  * FutureMap.pool (overlap_utils.py:339), NOT moved by this cut
Rank-uniform, so no ballot, digest or MIN can catch it: every rank is wrong
the same way. Both phases' pools hold the same row count, so the divergence
lands IN RANGE. Loud half: common.py:1849 -> free_slot's double-return
refusal (memory_pool.py:492-497). Silent half, which runs FIRST:
common.py:1836 reads the wrong pool's req_to_token row and hands those
indices to token_to_kv_pool_allocator.free().

THE CUT.
(a) UnifiedRadixCache.req_to_token_pool is a read-at-use property onto a
    registered owner, following kv_session_offload.py:2921-2933, whose
    docstring names this exact hazard. Unbound = byte-identical fallback to
    the constructor pool, so no non-flip boot changes.
(b) _restamp_phase_handles re-stamps tree_cache, the pool's back-reference
    and the layer-transfer counter onto the incoming pool at the seam.
(c) assert_req_pool_identity refuses at the end of the cutover
    (phase_flip_runtime, after the sgl-project#719 HiCache rebind, outside its
    try/except) when any holder still names a different pool.
(d) cutover_participants: two new REGISTRY rows
    (request_pool_phase_ownership, req_pool_back_references) and
    req_to_token_pool added to MUTATED_STATE -- a HANDLE the cutover
    REPLACES, next to ten quantities it recomputes. future_map is on the
    registry as an explained gap, not moved.

BRIEF PREMISE FALSIFIED -- (c) as briefed cannot be built.
"Turn the two silent returns at memory_pool.py:1995-1998 / :3992-3994 into
refusals" would refuse the DEFAULT path. layer_transfer_counter is None on
every boot without a hierarchical cache: it is only ever set by
register_layer_transfer_counter, whose callers are all HiCache/cache-
controller paths, and swa_memory_pool.py:124-125 registers None on purpose;
register_layer_transfer_counter's own docstring calls mamba_transfer_frame
=None "the historic no-wait behaviour, unchanged". So None is a legitimate
steady state, not a defect. What IS a defect is the handle failing to cross
the seam, and that is decidable where both pools are in hand. The refusal
therefore moved to the seam and the carry was built; the join sites are
untouched.

ANCHOR DRIFT (three, all reported rather than worked around):
  * "phase_req_pool_binding.py:180 is the only runtime rebind" -- :180 is
    the def line at the tree; the assignment is at :230, moved by CUT C
    (a890b8a). devindex still prints :180 because its store is pinned one
    commit back.
  * memory_pool.py:3992-3994 -- the guard is at :3993-3994; :3992 is a
    comment line.
  * memory_pool.py:496 -- the enclosing method free_slot is right, but :496
    is a fragment of the double-return message; the line every finish hits
    is :506 (free -> free_slot), and :492-497 is the guard.
All other anchors verified verbatim at the tree: unified_radix_cache.py:429
/507-513/514-515, mamba_radix_cache.py:502-503, memory_pool.py:1871/1876/
1995-1998/2019-2024, mamba_component.py:480, common.py:1780/1836/1847/1849,
kv_session_offload.py:2921-2924, cutover_participants.py:92/314-325.

MUTANTS (scratch copies in /tmp/wt1201-orig, all restored):
  1 DANGEROUS DIRECTION -- bind_req_pool_owner stores None, i.e. the
    pre-cut cached reference, which answers wrongly instead of refusing.
    Killed by 4: test_a_bound_cache_follows_the_owner,
    test_the_cutover_leaves_cache_and_scheduler_on_one_pool,
    test_the_kv_read_sees_the_rows_the_running_phase_wrote (the silent one),
    test_a_clean_cutover_passes.
  2 DANGEROUS DIRECTION -- bind_tree_cache re-stamp removed, so the pool's
    evict-then-retry stays disarmed silently. Killed by 2:
    test_the_incoming_pool_is_bound_to_the_tree_cache,
    test_a_clean_cutover_passes.
  3 assert_req_pool_identity returns early (#505a shape). Killed by 2:
    test_a_cache_left_on_the_outgoing_pool_is_refused,
    test_a_pool_left_bound_to_nothing_is_refused.
  4 counter carry dropped. Killed by 1:
    test_the_layer_transfer_counter_is_carried_to_the_incoming_pool.

VERIFIED: red-then-green tallies above; ruff parity on all four touched
files (unified_radix_cache.py had 1 pre-existing F401 typing.Sequence before
and after, zero new); import smoke of all four modules; no external writer
to a tree cache's req_to_token_pool and no __dict__ access to it (grep, none);
UnifiedRadixCache has no subclasses (devindex symbol_profile), so the
property cannot be shadowed.

NOT VERIFIED: no boot. Whether the layer-transfer counter carry ever fires
on the standing flip boot is UNMEASURED -- it depends on whether the
assembler registers a counter on both stacks' request pools, which only a
boot decides. MambaRadixCache.req_to_token_pool (mamba_radix_cache.py:498)
is the same cached-reference shape and is NOT converted here.

ANSWER TO THE COMPLETENESS CRITIC, recorded as asked: this registry axis
covers HANDLES THAT MOVED -- objects the cutover replaces and every holder
that cached one. It does NOT cover WHAT WAS STILL OWED at the arm:
launched-unreaped passes, outstanding receives, ring slots, lap counters.
Nothing here detects a debt; MUTATED_STATE's read-window axis and the new
handle axis are both about identity, not about obligations in flight. That
second axis remains unbuilt and unclaimed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…, in the phase that is about to end

WHAT WAS RED (parent 643237b, test_1201_future_map_phase.py): 14 failed,
0 passed. Reasons verified individually, not inferred from the tally:
ImportError on build_future_map; the registry entry's hook/probe still None;
MUTATED_STATE has no "future_map"; the source-order test cannot find the call.
After the cut: 14 passed. With the three immediate siblings
(test_1201_phase_stamped_handles, test_1040_req_pool_per_phase,
test_cutover_participants_859): 98 passed.

THE DEFECT, and it is on the LIVE path. FutureMap carries TWO boot stamps.
scheduler.py:820-823 parks a phase-flip instance's speculation at NONE because
the boot phase is PP prefill and has no draft worker (sgl-project#631); init_overlap then
builds the map out of that NONE (scheduler.py:2257) and nothing rebuilt it.
overlap_utils.py:107 branches on future_map.spec_algo.is_none(), and the
NON-overlap spec path reaches it -- scheduler.py:13174-13177, outside the
"if self.enable_overlap:" above it -- so on the standing boot form
(disable_overlap_schedule=True, speculative_algorithm='EAGLE') the frozen NONE
is read on every TP decode round and gathers last iteration's sampled token
into batch.input_ids on a batch whose input ids the spec worker owns. Nothing
raises: _assert_nonneg_and_invalidate is gated on SGLANG_IS_IN_CI
(overlap_utils.py:72), off on this rig. The second stamp is the request pool
(req_pool_size, max_context_len, ConfidenceRelay.pool) -- the fourth holder of
the sgl-project#1201 class, the one 643237b filed as an open gap.

THE CUT. build_future_map(scheduler) extracts the construction out of
init_overlap, duck-typed on five attributes so the cutover can call it without
importing the scheduler module. phase_flip_runtime calls it at the seam and
assert_future_map_identity(scheduler) beside assert_req_pool_identity. The
registry entry future_map_req_pool_holder now names both obligations and
"future_map" joins MUTATED_STATE on the HANDLE axis.

ORDERING IS THE LOAD-BEARING PART. The rebuild must run after BOTH swaps: the
algorithm swap (scheduler.spec_algorithm = want_spec_algo) and the request-pool
rebind (rebind_req_pool_for_cutover, ~330 lines later). Built between them it
carries the incoming algorithm and the OUTGOING pool. One test pins that source
order, and mutant 3 is exactly that mistake.

MUTANTS (scratch copies in /tmp/wt1201b3-orig, restored, md5 verified; grep
MUTANT over python/sglang/srt/managers clean):
  M1 DANGEROUS -- build_future_map hard-codes SpeculativeAlgorithm NONE, i.e.
     the defect kept: answers wrongly, never refuses. Killed by 5, including
     test_the_frozen_none_takes_the_branch_it_must_not, which asserts the
     silent branch is NOT taken rather than that a guard fires.
  M2 DANGEROUS -- the probe's pool check compares row counts instead of object
     identity, the in-range mistake this whole class is made of. Killed by 1,
     test_a_stale_pool_stamp_is_refused.
  M3 ORDERING -- the rebuild moved to just after the algorithm swap, before the
     pool rebind. Killed by 1, test_the_rebuild_runs_after_both_swaps.
  M4 DANGEROUS -- the probe swallows a stale algo stamp. Killed by 1,
     test_a_stale_algorithm_stamp_is_refused.

SIBLING SWEEP (analysis, not test code): every other reader of future_map reads
it at use off the scheduler (kv_session_offload.py:3899/:4411/:6040,
scheduler.py:13094, and the parameter-passing sites in disaggregation/ and
speculative/). No fifth cached holder on this axis.

WHAT I VERIFIED: hermetic CPU construction of FutureMap and of the whole suite
(CUDA_VISIBLE_DEVICES="", serving venv); import smoke on both edited runtime
modules; ruff parity -- 103 pre-existing findings in scheduler.py before and
after, zero new, the new test file clean.

WHAT I DID NOT VERIFY: nothing on metal. No boot, no GPU, no server. The cut is
DESK-PROVEN only. The rebuild allocates fresh device buffers per cutover and
that cost is unmeasured. The _confidence_budget_prepare handle next to the
future map in init_overlap is phase-stamped by the same argument (it caches the
draft worker's callable) and is NOT fixed here; it is gated on enable_overlap,
which is off on the standing form, so it is latent rather than live.

RETRACTED: test_1201_phase_stamped_handles's
test_the_future_map_holder_is_registered_as_an_open_gap asserted the gap is
OPEN. That was true one commit ago. Rewritten to assert the entry is complete,
retracted rather than deleted so the sequence stays readable.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 4, 2026
…ee adversarial lenses found in the six cuts

WHAT WAS RED, and every finding was re-verified at the tree before it was
acted on. Two of the eleven reported were REFUTED and are named at the end.

1. sgl-project#1202 added `mbs` to the residency AUTHORITY and not to the retirer.
   `_live_reqs` walks `mbs`; `consume_retracted_from_live_universe` did not.
   That is the W30 asymmetry re-opened, and it is the W27 death: the release
   frees a request the authority then re-enumerates, and
   `resident_mamba_slots` (gdn_flip_mover.py:620) raises KvReshardError on a
   live request with no mamba slot -- inside the no-return region, where the
   only outcome is a dead rank. The mbs-only shape is ordinary, not exotic:
   under event_loop_pp a freshly planned prefill batch sits in `mbs[slot]`
   while `running_mbs[slot]` holds the decode set.
   RED: the property test "authority and retirer walk the same containers"
   reported authority-only = ['mbs']; the concrete test read
   "0 != 1 : the mbs route retired no reference".
   Pinned as a PROPERTY over the two walks, so the next route added cannot
   reintroduce it. The stale present-tense collateral at
   scheduler_pp_mixin.py:5121 ("_live_reqs does NOT enumerate mbs", dated
   2026-09-04) is rewritten as the dated fact it now is.

2. sgl-project#1203 A5 spent three ESCALATION budgets in the group's abandon currency.
   A5's premise is right and is kept: a rank-local counter that zeroes itself
   when this rank's own objection clears can never reach its limit while
   ranks take turns objecting. But `flip_host_headroom_verdict` and
   `flip_seam_budget_verdict` do not merely stop deferring at the limit --
   they return (allow=True, escalated=True), "PROCEEDING WITH EYES OPEN"
   under a failed sgl-project#721 host floor -- and the writeback arm proceeds with an
   incomplete sgl-project#703 fence. `_seam_abandons_in_a_row` is incremented on EVERY
   reduced-fit abandon or frame divergence, so three unrelated abandons
   disarmed all three guards and the next GENUINE shortfall escalated on its
   first firing, having deferred zero times. A bound may be widened to the
   group's currency; an escalation that permits the hazard may not.
   REPAIR: each guard is bounded by its own count again, and that count is
   refunded ONLY by a completed flip (beside the abandon book, whose own
   comment already states the rule) or by the guard's own escalation. The
   whole reset policy is `flip_defer_budget_after`, one pure function, so the
   three copies cannot drift again.
   RED: 8 of 10 A5 tests failed at the parent.
   Three instruments that named the wrong counter are corrected with it:
   "consecutive" becomes "in this abandon run", and the host-RAM defer is
   booked BEFORE its own log line instead of after.

3. sgl-project#1204's abstain discarded a term the rank had successfully read. Both
   reads shared one `try`, so a rank whose `backed_rows()` succeeded and whose
   `live_floor_rows()` then raised contributed `_UNBOUNDED_ROWS` in slot 0 --
   the MIN identity, i.e. "I am not the poorest". Slot 0's MIN becomes the
   level the peers cap to and the ceiling `book_deferred_grow` clamps to, so
   erasing a poor rank makes the group agree an id level it cannot map, which
   `backed_rows`' docstring calls a cudaErrorIllegalAddress that kills every
   rank. The hang the commit removed would have been traded for a wrong group
   answer -- the corrupting direction bought to avoid the refusing one.
   REPAIR: abstain PER SLOT. Each slot carries this rank's own measurement or
   that slot's neutral, unchanged for a wholly unreadable relief.
   RED: "30 != 1099511627776".

4. sgl-project#1205's `seam_refused` measured a population its label does not name --
   the sgl-project#1205 defect class, committed inside the sgl-project#1205 fix. The fence read
   `seam_refusals()` with NO direction, summing "load" refusals raised on the
   controller's own background threads while the fence runs, and printed the
   delta as "of refused_silently, the share the FLIP SEAM refused".
   REPAIR: `seam_refusals("write")`, and the log line and the field comment
   now say what it is: process-wide write-direction refusals during the fence,
   not a subset of `refused_silently` and not per-node (consume_gate refuses a
   whole queue and counts one).
   RED: "1 != 3" when the direction-less read is put back.

5. `assert_future_map_identity` was REACHED and CANNOT-FAIL. Measured: four
   combinations of (algo NONE|EAGLE) x (pool pp|tp), ZERO fires.
   `create_future_map` passes `self` as `spec_algo` and the scheduler's pool
   straight into `ConfidenceRelay`, so both stamp arms are construction
   invariants at a call site 57 lines after the rebuild; the two suite tests
   that are green hand-mutate a map no code path produces.
   REPAIR: the registry's obligation is "a probe proving the rebuild ran", so
   the probe now takes `previous` -- the map held BEFORE the rebuild -- and
   refuses when the rebuild did not replace it. A memoised build, a deleted
   assignment or a rebuild moved out of the cutover all land there and none of
   them are visible to the stamp arms. The tautology is pinned as a fact so
   the green is not read as coverage.

6. MambaRadixCache got the sgl-project#1201 assertion but not the sgl-project#1201 fix. It caches
   `req_to_token_pool` as a plain attribute and has no `bind_req_pool_owner`,
   so `_restamp_phase_handles`' hasattr guard skipped it and the handle stayed
   on the outgoing pool -- and `assert_req_pool_identity`, added to both
   classes, then raises ReqPoolRebindRefused at the END of every cutover.
   Reachable: registry.py hands back this class for a hybrid-SSM model WITHOUT
   hierarchical cache. The boot went from flipping wrongly to not flipping.
   REPAIR: the same owner property and bind hook UnifiedRadixCache got.

7. The sgl-project#1203 A1 group vote moved a non-pure function onto a new cadence.
   `seam_transport_premise_holds_locally` writes the edge-triggered
   `_seam_premise_refused_announced` latch and emits SEAM TRANSPORT REFUSED;
   the vote runs once per loop iteration while the consumer returns at
   `_active_phase != PHASE_TP` before it asks. The latch was therefore set in
   the PP phase, swallowing the gate's own first engagement in TP.
   REPAIR: `announce=False` for the vote; the gate's own read still announces.

Also: the anchor at overlap_utils.py:81-83 had drifted onto the line that
REFUTES its sentence -- scheduler.py:13174-13177 is now the overlap call
inside `if self.enable_overlap:`, while the non-overlap spec branch is the
sibling `elif not batch.spec_algorithm.is_none():` arm. The claim is true; the
number sent the next reader to the wrong branch. Named by branch, no number.

WHAT KILLED WHICH MUTANT (planted in place, restored by sha256):
  M1 retirer over-clears (frees a row a live req owns, the corrupting
     direction) -> test_an_mbs_only_resident_is_retired_by_the_retraction
  M2 host bound back on the group abandon book (disarms sgl-project#721)
     -> test_the_host_headroom_bound_reads_its_own_defer_count
  M3 a cleared vote refunds its own budget (the A5 wedge)
     -> test_this_rank_clearing_does_not_refund_the_budget
      + test_a_taking_turns_run_still_reaches_the_limit
  M4 refund moved back into the gate's cleared-vote branch
     -> test_a_completed_flip_returns_every_budget_beside_the_abandon_book
  M5 sgl-project#1204 discards a measured backing again (claims not-poorest, the
     fleet-death direction)
     -> test_a_readable_backing_reaches_the_group_min
  M6 MambaRadixCache loses the rebind hook
     -> test_both_classes_expose_the_rebind_hook
      + test_the_handle_follows_the_owner_and_falls_back_without_one
  M7 the seam stops handing the probe the outgoing map
     -> test_the_seam_hands_the_probe_the_outgoing_map
  M8 the A1 vote announces again
     -> test_the_vote_leaves_the_announcement_latch_alone
8 planted, 8 killed, 0 survived.

VERIFIED: 146 passed + 7 subtests over the six touched test files plus
test_1157_seam_witness.py and test_seam_census_double_905.py (the joint pin
over both sgl-project#1202 functions), hermetic, CUDA_VISIBLE_DEVICES="" with the serving
venv. Import smoke over all seven edited modules. ruff parity with the parent
on the touched files: 31 findings both sides, all pre-existing.

NOT VERIFIED: nothing on metal. No boot, no GPU claimed, no collective. The
sgl-project#1204 and A5 repairs are DESK-PROVEN only -- neither has run against a real
reduce or a real host-RAM shortfall.

REFUTED, and left untouched: (a) `assert_req_pool_identity`'s tree-cache arm
is not decorative -- it is a construction invariant for UnifiedRadixCache
BECAUSE the owner property is the fix there, and a live check for every other
holder class, which is exactly what caught finding 6; (b) `seam_refused` was
already read as a delta across the fence run (hicache_flip_writeback.py:700),
not as an absolute, so only its direction and its label were wrong.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants