Skip to content

[HiCache] Fence load-back behind the forward stream - #36738

Merged
xiezhq-hermann merged 3 commits into
mainfrom
hicache-fence-load-back-behind-forward-stream
Aug 28, 2026
Merged

xiezhq-hermann merged 3 commits into
mainfrom
hicache-fence-load-back-behind-forward-stream

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Overlap scheduling frees a finished or retracted request's KV pages while the
round already launched for it is still writing them. A consumer that reuses
those pages from the compute stream is ordered after those writes implicitly,
so it sees correct data.

The hicache load-back H2D has no such ordering: it writes the reallocated pages
from the transfer stream. In-flight writes from the earlier round can therefore
race the restore and corrupt a restored prefix.

Modifications

  • managers/cache_controller.py: add HiCacheController.load_fence_stream
    (default None). In start_loading, when it is set, wait the transfer
    engine's host_to_device_stream on it before submitting the load's copies.
  • managers/scheduler.py: after the tree cache is built, point
    cache_controller.load_fence_stream at the model runner's forward_stream
    when hierarchical cache is enabled.

HybridCacheController subclasses HiCacheController, so it inherits both the
new attribute and the fenced start_loading; the guard also tolerates tree
caches with no controller, since BasePrefixCache defaults cache_controller
to None.

The stream-binding change in #36572 is what actually places these copies on the
fenced transfer stream. Without it this wait is a harmless no-op, so the two can
merge in either order.

Accuracy Tests

N/A as a standalone numerical check — the change adds stream ordering and does
not alter any computation. Its purpose is to remove a data race that corrupted
restored prefixes; the corruption is timing-dependent and does not reproduce
deterministically in a unit test.

Speed Tests and Profiling

One wait_stream per load-back submission, not per copy. It can delay a
load-back until the in-flight forward completes, which is the intended ordering.

Original commits

  • 585c6a7524

Checklist


CI States

Latest PR Test (Base): ✅ Run #33147439310
Latest PR Test (Extra): ✅ Run #33150271970
Latest PR Test (AMD ROCm 7.2): ❌ Run #33147439350

Overlap scheduling frees a finished or retracted request's KV pages while
its already-launched round still writes them. Consumers that reuse those
pages from the compute stream are ordered after those writes implicitly,
but the hicache load-back H2D writes from the transfer stream with no such
ordering, so in-flight writes can race the restore and corrupt a restored
prefix.

Wait on the forward stream before submitting a load's copies. The
stream-binding change in #36572 is what places these copies on the fenced
transfer stream; without it this wait is a harmless no-op, so the two can
merge in either order.
@xiezhq-hermann

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR labels Aug 27, 2026
test_hicache_staged_write_back_dispatch drives start_loading through
mock.Mock(spec=HybridCacheController) and through instances built with
__new__, neither of which runs __init__. An instance-only attribute is
therefore absent from the spec and the bare instance, so the new fence
check raised AttributeError.

Declaring it on the class gives every construction path the None default
and keeps the member visible in the class definition.
Replaces the class-level load_fence_stream declaration with the
instance attribute plus an explicit None on the spec'd controller mock,
matching the internal branch so the two stay byte-identical. The two
__new__ controllers in that test only exercise the write path, so they
never reach the fence.
@xiezhq-hermann xiezhq-hermann added the ready-to-merge The PR is ready to merge after the CI is green. label Aug 28, 2026
@xiezhq-hermann
xiezhq-hermann merged commit f611e0c into main Aug 28, 2026
187 of 219 checks passed
@xiezhq-hermann
xiezhq-hermann deleted the hicache-fence-load-back-behind-forward-stream branch August 28, 2026 18:12
wojciak pushed a commit to wojciak/sglang that referenced this pull request Aug 30, 2026
Adapt merged sgl-project#36738 and close its stream-binding dependency locally without wrapping every JIT module. Bind TVM-FFI only for HiCache kernel transfers, restore the prior per-thread stream afterward, and order H2D restore after the model forward stream so completion events cover the actual copies.

Signed-off-by: John Pezzulli <38448408+jpezzulli@users.noreply.github.com>
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 25, 2026
…d-back indices without host waits (both switches, default off)

User order 24.09.: HiCache work "darf nie den laufenden prefill oder decode
beeintraechtigen". Two things of it sat in the path between two D rounds.

1. Every scheduler round: check_hicache_events -> drain_storage_control_queues
   MIN-reduces the storage queue sizes over the attention group. On group D
   (TP 3) that is one gloo all_reduce per decode round, waited through
   bounded_wait -> ParkedWait (a new waiter thread per call), and it
   lock-steps the three schedulers once per round -- for queues that are empty
   through a decode phase (xsn420, 100k decode: 873 rounds on TP0, 7 WT-ACKs).
   SGLANG_HICACHE_DRAIN_AGREE_EVERY=N: the agreement runs every round while it
   drains and every N-th round while the group's MIN is zero. Rank-uniform by
   construction: the round counter advances once per check_hicache_events
   (calls are rank-uniform -- the sgl-project#1028 census reads the same n on all three D
   ranks, and today's every-round collective would deadlock otherwise), and
   "hot" is the MIN just agreed. Between agreements nobody drains. Engages only
   where the drain takes a collective (group P, TP 1, keeps its every-round
   local drain). drain_storage_control_queues now returns whether the MIN was
   non-zero; every other caller ignores it.

2. The load-back of a request admitted between two D rounds: since sgl-project#36738
   (6816062) the load stream waits for the forward stream, so every
   host-blocking index copy on the load path holds the scheduler thread until
   the running decode forward is done (xsn420 12:37:44: START-LOADING tokens=6,
   total 54 ms, mamba 49 ms, mid-decode). SGLANG_HICACHE_LOAD_ASYNC_INDEX=1:
   ArenaMambaHostPool's state load keeps the device rows on the card (selected
   there when not every row is an arena row) instead of `.cpu()` + pageable
   `.to(dev)`; ArenaMHAHostPool._transfer moves its indices through pinned
   memory with non_blocking (the xsn338 pattern). Same bytes, same rows, same
   stream order; only the host wait is gone. The draft tier's miss path
   (`_load_arena` with row_slot) is left to SGLANG_WEG2_HICACHE_DRAFT_TIER (B),
   not touched here.

3. SGLANG_HICACHE_ROUND_TIMING=N (instrument): check_hicache_events wall time
   per round split into checks (reap + ack polls) and drain (the agreement),
   summarised every N rounds as HICACHE-ROUND-TIMING.

Unset, all three are the unchanged path (the default-off test pins it).

Named, not changed: move_indices (direct, layer_first) `.cpu()` on the
schedule stream (ordered behind the forward's read-done since sgl-project#31687),
_arena_load_guard's int(device_indices.min/max) on device rows, the
between-block synchronize() of a multi-block state load, transfer_kv_direct's
`.cpu()` (sgl-kernel, staging rows), and the CPU gather of the state load
(index_select arena -> pinned stage) which stays on the scheduler thread --
taking it off needs the admission deferred until an asynchronous gather lands.

Measured (desk, 3 gloo CPU ranks on loopback, 3000 rounds, idle-priority test
cgroup on the loaded box, bench_drain_collective.py): the agreement as today
(bounded_wait/ParkedWait every round) 0.58 ms mean / 0.45 p50 / 2.3 p99 per
round and rank; plain wait() 0.55 / 0.38 / 2.5; gate N=8 on an idle group
0.071 mean / 1.0 p99; none 0. So the HiCache agreement is ~0.5-0.6 ms of the
~13 ms a D round spends outside the GPU timer -- a small, not the dominant,
term; the lock-step it imposes is not in that figure.

Tests (incg/test27b, CPU, files one by one):
- test_hicache_dround_27b.py (new): 13 passed -- unset = every round (the
  default-off identity), P (TP 1) never gated, the drain reports the MIN it
  drained with the same counts, three simulated D ranks enter the agreement on
  identical rounds with identical MINs and drain everything, idle cadence
  1, 9, 17, ..., hot every round, the timer only measures, _transfer hands the
  kernel the same int64 indices, the state loader lands bit-identical bytes.
- test_weg2_state_load_xsn335.py 2 passed; test_arena_draft_window_per_layer_
  xsn267.py 5 passed; test_arena_load_guard_xsn277.py 4 passed;
  test_hicache_bounded_waits_630.py 14 passed.
- test_arena_host_pool_1424.py: 3 failed / 5 passed, IDENTICAL on the base
  e15e760 (pre-existing, not this change).

NF line (H62, onto 88dec66): picked with -x, conflicts resolved line by
line against the NF instruments, nothing else changed.
- arena_mamba_pool.py import: NF's `_arena_page_load_mode` (H12 dma mode)
  kept beside the three new helpers.
- arena_pool.py: NF's `planned_arena_slots` / `planned_id_space_tokens`
  (ENV_ARENA_KV_PAGE_BYTES) kept, the three helpers follow them.
- unified_radix_cache.py check_hicache_events: the H49 `_HOST_COST.drain_ms`
  timer now wraps the gated-or-ungated drain (a skipped round books ~0 there,
  so DECODE-HOST-PERIOD drain_ms / allreduce_n show the cadence directly);
  H49's hc_ms / hc_calls stay where they were, the ROUND_TIMING note follows.
NF D (TP 3) takes the collective path (`_drain_agreement_is_collective` True),
group P (TP 1 / PP 3) never engages the gate. The NF load-back paths of their
own (H12 "dma" whole-page / state load, `_transfer_paged`) are not touched;
only `_transfer` and the mamba state load's index crossing follow the switch.

NF tests (CPU, incg.sh, clone, files one by one): test_hicache_dround_27b.py
13 passed. Selection for the three changed modules (gate_changed rule): 28
files run -- green: 1201, 1068 drop-tree, 1427 x2, xsn267, x59, xsn277, h12,
x62, x76, h2, dround_27b, x25, 773, 1417, 935, 966, 1068 ring, radix bench,
h47, xsn305, task3, h19, bounded_waits_630. Red, IDENTICAL on the base
88dec66 (pre-existing, not this change): test_1203_group_branches_on_one_
rank (2), test_protected_residue_orphaned_938 (2), test_arena_host_pool_1424
(3); test_unified_radix_cache_unittest.py OOM-killed in the 3-GiB test cgroup
at result 1155/1502 on both trees, 176 F at identical positions. Not run: the
nine server/GPU KL tests under test/registered/radix_cache/unified_radix_tree
(no GPU for this job).

(cherry picked from commit 0619f12)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…cks, chronologisch)

Grundlage: Präsenz-Scan aller 230 27B-Commits seit 76f8deb gegen diesen Baum
(Stichprobe der hinzugefügten Zeilen je Commit); die 94 fehlenden minus die bewusst
anders gewählten Formen (76e87ac/4ae11ababd -> S3 form.calibration_identity;
479f6ec/d7f588e017/d0fba8955f/34892e3017 -> S2 NF-Formen; 7f81f09/3c14481318 ->
S4/S7a; 3dbb790 line_gate_27b (Werkzeug, Schritt 9); 8604d13 W100-by-name
(Nutzer: bleibt aus); 6545e2c flashinfer-Pin in pyproject (Image-Frage, nicht Baum)).
Liste: 92bbccb eb5d044 829ebd0 431fcbc ef4d11f 6816062 a233e50 2cc593c f0c8451 87cc4fb c529777 31f2dbe c50085a 3301a96 036b368 e1d1fe9 03c68af 6dddc2e 06932b5 87389c4 58a7490 f85ac55 fc64aa5 5aa24dd 97c0e9a 159333c d9f1532 f3c685b 8550655 e50fb59 db2c2ef f09dc0d c255e10 51b810e 28a55a2 34965fc ff3d9cc 340a018 bee5e10 67b6352 fdade85 ed6630d f1c9a43 b434831 517f26d 0b6b60b a40837f 644de86 aff2b7c 197b701 856024b 238512a 9738626 b857a22 1f8c24d d294b3e 810239d b429dfd e714c95 9efd974 3d63e0a d3cfcf3 fee6134 7985b56 49a14e9 fc45706 19c720e 5306bee 6f1235a c98eaa3 93bc802 328349e f9fb3a2 572af73 94fa8b4 d342caa 2fd7d7e 3ebbb96 871d55f 78c2f16 3babf51 196f6a8 e70af54 22eccfc

Inhalt: Upstream-Ports (sgl-project#33758 sgl-project#37818 sgl-project#36738 sgl-project#33459/sgl-project#30096 sgl-project#34446 sgl-project#36267 sgl-project#33778 sgl-project#34859
sgl-project#36415 sgl-project#35255/sgl-project#36638 sgl-project#39858/sgl-project#40259 sgl-project#31417 sgl-project#34892 sgl-project#32225 sgl-project#30832/sgl-project#36626 sgl-project#39574 sgl-project#29579
sgl-project#31468 sgl-project#32575 sgl-project#31648); xsn409/410-Wake-Verdikte; Vision-Linie V1-V3b + xsn438
(SGLANG_WEG2_VISION_FLIP_URGENT); D-Planer L6 (159333c); DFLASH-Window-Pool
sync-frei, PLAN_SYNC_FREE, D-Kollektive (vocab-argmax, a2a-Merge, deferred rebuild),
#DGAP/D_DEFER_SEQ_LENS_CPU; Mamba-Anker Raster 4096 + Per-Path-Cap + Inner-Release;
P-TRIM (--p-trim-end-anchor); FP8 uniform Marlin; ModelOpt/NVFP4 RadixArk; GGUF G1-G6
+ F1/F2; native-mixed sgl-project#38 (sm_8x W4A8, sm_12x CUTLASS/W4A16); RC1-Capture-Set; sgl-project#49
Agent-Turns; dynchunk (--p-chunk-policy, --p-chunk-dynamic-min-tokens).

Auflösungen (Gabel -> Form, Grund):
- L6 d_operating_point_rows: 27B (d) "Token-Vektor auf jeder Position aus der Kapazität"
  nur bei TP-symmetrischem D (Profil d_layout paged_dcp); sonst NF-sgl-project#1293-Pin + NF-Anker-
  Klausel. mamba_ssm_dtype aus EARLY_READ_FACTS nur bei Profil early_read_flags.
  Overhead-Kalibrierung liest mit form.CalibrationIdentity statt LineIdentity.
- RC1 Capture-Set: neuer RecordKey-Term d_capture_set (qwen27b), Leser
  CalibrationIdentity.d_max_running_requests; nextflash unverändert.
- URC Carrier-Hold: 27B _weg2_carrier_hold entfällt (S2 NF-Rotation), Inner-Release und
  Per-Path-Cap bleiben (Env, Default aus; 27b.env setzt sie).
- scheduler_pp_mixin/overlap_utils/batch_result_processor: NF H49/H58 und 27B #PGAP/#DGAP
  komponiert (beide Instrumente getrennt schaltbar).
- schedule_policy: P-TRIM-Kurzschluss vor NF H63-Fold/QSA-Korn; sgl-project#36415 Hoist + NF
  computed_input_len.
- gdn_backend sgl-project#33778: 27B-strided-Verify; NF-Ring flacht beide Layouts ab.
- flashinfer_backend: RC9-Datei + NF-Form-A-Waiver (27B-intern mehrfach gegabelt).
- checkpoint_census: GGUF-Leser + NF exclude_segments (PLE) in einer Aggregation.
- xchg_manifest: FLAT_SEGMENTS in beiden (dst/src) NF-Breitenbedingungen ausgenommen.
- vram_peak_window: NF-Kumulativ-Peak liest über den 27B-Fast-Read.
- FP8: 8c86eb8-Rest nachgezogen (private Workspace-Registry entfernt, wie RC9).
- argv_d: vision= an allen drei Aufrufstellen (inkl. NF --d-only).
- census_checkpoint_decision (W161) jetzt für beide Profile aktiv.

Gates: py_compile aller geänderten Dateien; ruff F821/F811 ohne neue Funde gegenüber
dem Vorgänger (PendingSeqLensCpu ist String-Annotation wie in RC9); dup_defs_gate 0 neu.
Tests: 73 portierte Testdateien, Lauf nach dem Ruhefenster (Boot aktiv).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail hicache Hierarchical Caching for SGLang ready-to-merge The PR is ready to merge after the CI is green. run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant