Repository navigation
[HiCache] Fence load-back behind the forward stream - #36738
Merged
xiezhq-hermann merged 3 commits intoAug 28, 2026
Merged
Conversation
Overlap scheduling frees a finished or retracted request's KV pages while its already-launched round still writes them. Consumers that reuse those pages from the compute stream are ordered after those writes implicitly, but the hicache load-back H2D writes from the transfer stream with no such ordering, so in-flight writes can race the restore and corrupt a restored prefix. Wait on the forward stream before submitting a load's copies. The stream-binding change in #36572 is what places these copies on the fenced transfer stream; without it this wait is a harmless no-op, so the two can merge in either order.
xiezhq-hermann
requested review from
Ying1123,
hnyls2002 and
merrymercy
as code owners
August 27, 2026 22:17
Collaborator
Author
|
/tag-and-rerun-ci |
test_hicache_staged_write_back_dispatch drives start_loading through mock.Mock(spec=HybridCacheController) and through instances built with __new__, neither of which runs __init__. An instance-only attribute is therefore absent from the spec and the bare instance, so the new fence check raised AttributeError. Declaring it on the class gives every construction path the None default and keeps the member visible in the class definition.
3 of 5 tasks
Replaces the class-level load_fence_stream declaration with the instance attribute plus an explicit None on the spec'd controller mock, matching the internal branch so the two stay byte-identical. The two __new__ controllers in that test only exercise the write path, so they never reach the fence.
xiezhq-hermann
deleted the
hicache-fence-load-back-behind-forward-stream
branch
August 28, 2026 18:12
wojciak
pushed a commit
to wojciak/sglang
that referenced
this pull request
Aug 30, 2026
Adapt merged sgl-project#36738 and close its stream-binding dependency locally without wrapping every JIT module. Bind TVM-FFI only for HiCache kernel transfers, restore the prior per-thread stream afterward, and order H2D restore after the model forward stream so completion events cover the actual copies. Signed-off-by: John Pezzulli <38448408+jpezzulli@users.noreply.github.com>
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 25, 2026
…d-back indices without host waits (both switches, default off) User order 24.09.: HiCache work "darf nie den laufenden prefill oder decode beeintraechtigen". Two things of it sat in the path between two D rounds. 1. Every scheduler round: check_hicache_events -> drain_storage_control_queues MIN-reduces the storage queue sizes over the attention group. On group D (TP 3) that is one gloo all_reduce per decode round, waited through bounded_wait -> ParkedWait (a new waiter thread per call), and it lock-steps the three schedulers once per round -- for queues that are empty through a decode phase (xsn420, 100k decode: 873 rounds on TP0, 7 WT-ACKs). SGLANG_HICACHE_DRAIN_AGREE_EVERY=N: the agreement runs every round while it drains and every N-th round while the group's MIN is zero. Rank-uniform by construction: the round counter advances once per check_hicache_events (calls are rank-uniform -- the sgl-project#1028 census reads the same n on all three D ranks, and today's every-round collective would deadlock otherwise), and "hot" is the MIN just agreed. Between agreements nobody drains. Engages only where the drain takes a collective (group P, TP 1, keeps its every-round local drain). drain_storage_control_queues now returns whether the MIN was non-zero; every other caller ignores it. 2. The load-back of a request admitted between two D rounds: since sgl-project#36738 (6816062) the load stream waits for the forward stream, so every host-blocking index copy on the load path holds the scheduler thread until the running decode forward is done (xsn420 12:37:44: START-LOADING tokens=6, total 54 ms, mamba 49 ms, mid-decode). SGLANG_HICACHE_LOAD_ASYNC_INDEX=1: ArenaMambaHostPool's state load keeps the device rows on the card (selected there when not every row is an arena row) instead of `.cpu()` + pageable `.to(dev)`; ArenaMHAHostPool._transfer moves its indices through pinned memory with non_blocking (the xsn338 pattern). Same bytes, same rows, same stream order; only the host wait is gone. The draft tier's miss path (`_load_arena` with row_slot) is left to SGLANG_WEG2_HICACHE_DRAFT_TIER (B), not touched here. 3. SGLANG_HICACHE_ROUND_TIMING=N (instrument): check_hicache_events wall time per round split into checks (reap + ack polls) and drain (the agreement), summarised every N rounds as HICACHE-ROUND-TIMING. Unset, all three are the unchanged path (the default-off test pins it). Named, not changed: move_indices (direct, layer_first) `.cpu()` on the schedule stream (ordered behind the forward's read-done since sgl-project#31687), _arena_load_guard's int(device_indices.min/max) on device rows, the between-block synchronize() of a multi-block state load, transfer_kv_direct's `.cpu()` (sgl-kernel, staging rows), and the CPU gather of the state load (index_select arena -> pinned stage) which stays on the scheduler thread -- taking it off needs the admission deferred until an asynchronous gather lands. Measured (desk, 3 gloo CPU ranks on loopback, 3000 rounds, idle-priority test cgroup on the loaded box, bench_drain_collective.py): the agreement as today (bounded_wait/ParkedWait every round) 0.58 ms mean / 0.45 p50 / 2.3 p99 per round and rank; plain wait() 0.55 / 0.38 / 2.5; gate N=8 on an idle group 0.071 mean / 1.0 p99; none 0. So the HiCache agreement is ~0.5-0.6 ms of the ~13 ms a D round spends outside the GPU timer -- a small, not the dominant, term; the lock-step it imposes is not in that figure. Tests (incg/test27b, CPU, files one by one): - test_hicache_dround_27b.py (new): 13 passed -- unset = every round (the default-off identity), P (TP 1) never gated, the drain reports the MIN it drained with the same counts, three simulated D ranks enter the agreement on identical rounds with identical MINs and drain everything, idle cadence 1, 9, 17, ..., hot every round, the timer only measures, _transfer hands the kernel the same int64 indices, the state loader lands bit-identical bytes. - test_weg2_state_load_xsn335.py 2 passed; test_arena_draft_window_per_layer_ xsn267.py 5 passed; test_arena_load_guard_xsn277.py 4 passed; test_hicache_bounded_waits_630.py 14 passed. - test_arena_host_pool_1424.py: 3 failed / 5 passed, IDENTICAL on the base e15e760 (pre-existing, not this change). NF line (H62, onto 88dec66): picked with -x, conflicts resolved line by line against the NF instruments, nothing else changed. - arena_mamba_pool.py import: NF's `_arena_page_load_mode` (H12 dma mode) kept beside the three new helpers. - arena_pool.py: NF's `planned_arena_slots` / `planned_id_space_tokens` (ENV_ARENA_KV_PAGE_BYTES) kept, the three helpers follow them. - unified_radix_cache.py check_hicache_events: the H49 `_HOST_COST.drain_ms` timer now wraps the gated-or-ungated drain (a skipped round books ~0 there, so DECODE-HOST-PERIOD drain_ms / allreduce_n show the cadence directly); H49's hc_ms / hc_calls stay where they were, the ROUND_TIMING note follows. NF D (TP 3) takes the collective path (`_drain_agreement_is_collective` True), group P (TP 1 / PP 3) never engages the gate. The NF load-back paths of their own (H12 "dma" whole-page / state load, `_transfer_paged`) are not touched; only `_transfer` and the mamba state load's index crossing follow the switch. NF tests (CPU, incg.sh, clone, files one by one): test_hicache_dround_27b.py 13 passed. Selection for the three changed modules (gate_changed rule): 28 files run -- green: 1201, 1068 drop-tree, 1427 x2, xsn267, x59, xsn277, h12, x62, x76, h2, dround_27b, x25, 773, 1417, 935, 966, 1068 ring, radix bench, h47, xsn305, task3, h19, bounded_waits_630. Red, IDENTICAL on the base 88dec66 (pre-existing, not this change): test_1203_group_branches_on_one_ rank (2), test_protected_residue_orphaned_938 (2), test_arena_host_pool_1424 (3); test_unified_radix_cache_unittest.py OOM-killed in the 3-GiB test cgroup at result 1155/1502 on both trees, 176 F at identical positions. Not run: the nine server/GPU KL tests under test/registered/radix_cache/unified_radix_tree (no GPU for this job). (cherry picked from commit 0619f12)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…cks, chronologisch) Grundlage: Präsenz-Scan aller 230 27B-Commits seit 76f8deb gegen diesen Baum (Stichprobe der hinzugefügten Zeilen je Commit); die 94 fehlenden minus die bewusst anders gewählten Formen (76e87ac/4ae11ababd -> S3 form.calibration_identity; 479f6ec/d7f588e017/d0fba8955f/34892e3017 -> S2 NF-Formen; 7f81f09/3c14481318 -> S4/S7a; 3dbb790 line_gate_27b (Werkzeug, Schritt 9); 8604d13 W100-by-name (Nutzer: bleibt aus); 6545e2c flashinfer-Pin in pyproject (Image-Frage, nicht Baum)). Liste: 92bbccb eb5d044 829ebd0 431fcbc ef4d11f 6816062 a233e50 2cc593c f0c8451 87cc4fb c529777 31f2dbe c50085a 3301a96 036b368 e1d1fe9 03c68af 6dddc2e 06932b5 87389c4 58a7490 f85ac55 fc64aa5 5aa24dd 97c0e9a 159333c d9f1532 f3c685b 8550655 e50fb59 db2c2ef f09dc0d c255e10 51b810e 28a55a2 34965fc ff3d9cc 340a018 bee5e10 67b6352 fdade85 ed6630d f1c9a43 b434831 517f26d 0b6b60b a40837f 644de86 aff2b7c 197b701 856024b 238512a 9738626 b857a22 1f8c24d d294b3e 810239d b429dfd e714c95 9efd974 3d63e0a d3cfcf3 fee6134 7985b56 49a14e9 fc45706 19c720e 5306bee 6f1235a c98eaa3 93bc802 328349e f9fb3a2 572af73 94fa8b4 d342caa 2fd7d7e 3ebbb96 871d55f 78c2f16 3babf51 196f6a8 e70af54 22eccfc Inhalt: Upstream-Ports (sgl-project#33758 sgl-project#37818 sgl-project#36738 sgl-project#33459/sgl-project#30096 sgl-project#34446 sgl-project#36267 sgl-project#33778 sgl-project#34859 sgl-project#36415 sgl-project#35255/sgl-project#36638 sgl-project#39858/sgl-project#40259 sgl-project#31417 sgl-project#34892 sgl-project#32225 sgl-project#30832/sgl-project#36626 sgl-project#39574 sgl-project#29579 sgl-project#31468 sgl-project#32575 sgl-project#31648); xsn409/410-Wake-Verdikte; Vision-Linie V1-V3b + xsn438 (SGLANG_WEG2_VISION_FLIP_URGENT); D-Planer L6 (159333c); DFLASH-Window-Pool sync-frei, PLAN_SYNC_FREE, D-Kollektive (vocab-argmax, a2a-Merge, deferred rebuild), #DGAP/D_DEFER_SEQ_LENS_CPU; Mamba-Anker Raster 4096 + Per-Path-Cap + Inner-Release; P-TRIM (--p-trim-end-anchor); FP8 uniform Marlin; ModelOpt/NVFP4 RadixArk; GGUF G1-G6 + F1/F2; native-mixed sgl-project#38 (sm_8x W4A8, sm_12x CUTLASS/W4A16); RC1-Capture-Set; sgl-project#49 Agent-Turns; dynchunk (--p-chunk-policy, --p-chunk-dynamic-min-tokens). Auflösungen (Gabel -> Form, Grund): - L6 d_operating_point_rows: 27B (d) "Token-Vektor auf jeder Position aus der Kapazität" nur bei TP-symmetrischem D (Profil d_layout paged_dcp); sonst NF-sgl-project#1293-Pin + NF-Anker- Klausel. mamba_ssm_dtype aus EARLY_READ_FACTS nur bei Profil early_read_flags. Overhead-Kalibrierung liest mit form.CalibrationIdentity statt LineIdentity. - RC1 Capture-Set: neuer RecordKey-Term d_capture_set (qwen27b), Leser CalibrationIdentity.d_max_running_requests; nextflash unverändert. - URC Carrier-Hold: 27B _weg2_carrier_hold entfällt (S2 NF-Rotation), Inner-Release und Per-Path-Cap bleiben (Env, Default aus; 27b.env setzt sie). - scheduler_pp_mixin/overlap_utils/batch_result_processor: NF H49/H58 und 27B #PGAP/#DGAP komponiert (beide Instrumente getrennt schaltbar). - schedule_policy: P-TRIM-Kurzschluss vor NF H63-Fold/QSA-Korn; sgl-project#36415 Hoist + NF computed_input_len. - gdn_backend sgl-project#33778: 27B-strided-Verify; NF-Ring flacht beide Layouts ab. - flashinfer_backend: RC9-Datei + NF-Form-A-Waiver (27B-intern mehrfach gegabelt). - checkpoint_census: GGUF-Leser + NF exclude_segments (PLE) in einer Aggregation. - xchg_manifest: FLAT_SEGMENTS in beiden (dst/src) NF-Breitenbedingungen ausgenommen. - vram_peak_window: NF-Kumulativ-Peak liest über den 27B-Fast-Read. - FP8: 8c86eb8-Rest nachgezogen (private Workspace-Registry entfernt, wie RC9). - argv_d: vision= an allen drei Aufrufstellen (inkl. NF --d-only). - census_checkpoint_decision (W161) jetzt für beide Profile aktiv. Gates: py_compile aller geänderten Dateien; ruff F821/F811 ohne neue Funde gegenüber dem Vorgänger (PendingSeqLensCpu ist String-Annotation wie in RC9); dup_defs_gate 0 neu. Tests: 73 portierte Testdateien, Lauf nach dem Ruhefenster (Boot aktiv).
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Overlap scheduling frees a finished or retracted request's KV pages while the
round already launched for it is still writing them. A consumer that reuses
those pages from the compute stream is ordered after those writes implicitly,
so it sees correct data.
The hicache load-back H2D has no such ordering: it writes the reallocated pages
from the transfer stream. In-flight writes from the earlier round can therefore
race the restore and corrupt a restored prefix.
Modifications
managers/cache_controller.py: addHiCacheController.load_fence_stream(default
None). Instart_loading, when it is set, wait the transferengine's
host_to_device_streamon it before submitting the load's copies.managers/scheduler.py: after the tree cache is built, pointcache_controller.load_fence_streamat the model runner'sforward_streamwhen hierarchical cache is enabled.
HybridCacheControllersubclassesHiCacheController, so it inherits both thenew attribute and the fenced
start_loading; the guard also tolerates treecaches with no controller, since
BasePrefixCachedefaultscache_controllerto
None.The stream-binding change in #36572 is what actually places these copies on the
fenced transfer stream. Without it this wait is a harmless no-op, so the two can
merge in either order.
Accuracy Tests
N/A as a standalone numerical check — the change adds stream ordering and does
not alter any computation. Its purpose is to remove a data race that corrupted
restored prefixes; the corruption is timing-dependent and does not reproduce
deterministically in a unit test.
Speed Tests and Profiling
One
wait_streamper load-back submission, not per copy. It can delay aload-back until the in-flight forward completes, which is the intended ordering.
Original commits
585c6a7524Checklist
CI States
Latest PR Test (Base): ✅ Run #33147439310
Latest PR Test (Extra): ✅ Run #33150271970
Latest PR Test (AMD ROCm 7.2): ❌ Run #33147439350