Repository navigation
Conversation
Issue the per-step ready-count all-reduce asynchronously on a dedicated gloo group in step N and consume it in step N+1, removing a blocking collective from the scheduler critical path. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… docs The deferred ready-count sync skips the storage-queue drain, so a storage backend attached at runtime would never have its queues serviced. Settle the in-flight sync and switch to the synchronous path once storage is on. Add CPU unit tests for the deferred-consume invariants (ack prefix kept across a step, remembered end-of-prefix after a drain, digest check, the storage-attach fallback) and document the flag, which the server-args docs coverage test requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
…ache-async-ready-sync # Conflicts: # docs/docs/advanced_features/hicache_best_practices.mdx
|
cc. @nvpohanh @YAMY1234 |
|
HiCache expert would be @hnyls2002 |
|
cc @nvjullin |
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
|
@wenscarl Could you check whether we'd still run into the problem described in the PR if we used writeback? Just want to check whether this overhead is specific to agg + lowlatency + write-through. Thanks! |
check_hicache_events runs a blocking gloo MIN-all-reduce of the write-back/load-back ready counts every scheduler step; on TP2 short-decode steps that CPU collective is a large share of the step. With the new flag (default False) the same reduction is issued asynchronously on its own gloo group in step N and consumed in step N+1: acks complete in queue order and queues only grow at the tail, so the agreed ready prefix stays valid one step later, and _deferred_consume_count re-finds the agreed last-ready ack by identity so a blocking drain in between is never double-consumed. A deferred digest pair keeps the write_back duplicate-reclaim victim-order assertion across ranks. Runtime storage attach settles the in-flight sync and falls back to synchronous for good (the deferred path skips the storage-queue drain). reset() drains the pending op. Union notes: lands after the sgl-project#46451 port in unified_radix_cache.py — the insertion sits between flush_pending_backups() (entry-199 batched-backup drain; its SGLANG_ENABLE_HICACHE_BATCHED_BACKUP default-OFF semantics are untouched: the new flag is an orthogonal memory-group bool, flag-off is the unchanged synchronous path) and the four-tuple _sync_hicache_ready_counts call our entry-217/218 region built on. The deferred gate refuses storage/buffer_only/PP/attn-CP/single-rank setups with a warning, matching the landed _sync_hicache_ready_counts payload (storage-queue sizes stay on the sync path). Ports the full head (29e8125 + 39a0c0d incl. the runtime-attach refactor + docs); the 244-line CPU test suite (7P) drives two simulated ranks through the real methods. Docs rows land next to the entry-219 mamba-sizing note. Fork-stub adaptation: our test_hicache_pp_sync_drain __new__-built stubs now pre-memoize _hcs_state=False (same as upstream's own async harness pre-setting the state) so the per-step gate never resolves the memory bag these stubs never publish; 4P. (cherry picked from commit 6ffe9cd) PR: sgl-project#45795 (open, ported) (cherry picked from commit b357ce2961bdfa723091e0b3bd44b989243756d8)
…),199,217,218,219,253,290d,293,294 Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the #46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py. What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired, the reason="deadline" bookkeeping, belief-heal suppression on deadline, the _invalidate_absent_from_hit_query move to the drain head + KV-verdict guard + deadline gate, the release_aborted_request / finish_storage_prefetch_admission wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800 (unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P). Re-planted here: - entry 294 (sgl-project#45795): the FULL deferred ready-count sync family (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback, _check_hicache_events_deferred one-op-per-step on its own gloo group, _consume_deferred_ready_sync with the digest MIN/-MAX assert, _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync collection error (the B3-caused 1err in the merge gates). - entries 199+218: the batched write-through backup family VERBATIM from the patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups, _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node, _parent_backed_up, _release_queued_backup, _backup_queued_batch, _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the node_ids= ack-list through _execute_kv_backup into the controller write (B3-2), the flush_pending_backups guard, and the writing_check(write_back) drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule). _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record). - entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache reads with the nonnegative-finite ValueError, wait_full, the retry cadence (host_capacity_deadline/next_retry/retries — fields planted on the operation in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with identity snapshots, the info.operation-is-not-operation late-query guards, and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical. - entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining prefetches a smaller endpoint would fit). - entry 219 producer: _record_host_pool_evictions on evict_host (before _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}. - entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through merge_source_transfers (every destination, not matching_sources[0]; the two AssertionErrors it subsumes are structurally guaranteed by the resolver). - entry 30 rider: load_back unwinds (drop both locks, report miss) when any component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba component sets -- component side rides B3-5's mamba.py cluster). - entry 253: the per-call-site file_storage_path fold DELETED here (it moved into _generate_storage_config, every attach path, B3-1-verified). - test_chunked_publish_release_disjointness.py (entry 307's tripwire): re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...) per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line is the documented cargo/torch-sys ENV class (fails identically at base). ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after): test_hicache_async_ready_sync 1err -> 7P/0F ** B3-caused error CLEARED ** test_hybrid_storage_sparse_allocation 12P/1F -> 13P/0F/14sub ** entry-290(d) closed ** test_storage_prefetch_lifecycle -> 16P/0F/13sub test_hicache_trailing_endpoint (gate) -> 17P/0F (unmoved by this cluster; controller-side B3-2) test_buffer_mode_sidecar -> 6P/0F/6sub test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green) test_mamba_path_state_cap -> 5P/5S (skips = honest, env) test_decode_retraction_backup 2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log) test_hicache_mamba_sizing -> 30P/0F/17sub (unmoved) URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S UNMOVED — 100% of the errors are 'No accelerator ... available' from the upstream kits' fixtures.py build_fixture calling get_device() (entry 310's "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's own per-file numbers (core 27P/465F etc.) match the damage totals exactly; this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here). Lint: ruff format + F821/E9 clean on both files; py_compile + import OK. No GPU executed; no push.
|
@YAMY1234 It isn't specific to write-through. With What is specific is the regime. It shows end-to-end only when the scheduler loop is on the critical path, i.e. low concurrency or a short GPU step, as in the 12-layer rows. In GPU-bound setups (40 and 78 layers) the overlap scheduler hides it. Happy to add a write-back run to the tables if that would help. |
Address review: drop the defensive getattr and the None/False/group tri-state. The dedicated gloo group is created once at the end of init_hicache (None = synchronous path), and the per-step check only handles a runtime storage attach. Trim the explanatory comment and add parametrized tests that every ineligible setup keeps the sync path without creating a group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the opt-in --hicache-async-ready-sync with --disable-hicache-async-ready-sync. Unsupported setups fall back to the synchronous sync with an info log instead of a warning. Correct the docs: the deferred ack only delays unlocking finished nodes, not the forward pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@wenscarl I am a little confused. So the gain from this PR only comes in low latency regime right? In that scenario why would we want to enable HiCache? |
|
@kpham-sgl Low-latency can still have a large prefix cache footprint if subsequent requests has a long time delay in between. For example, suppose there are 50 people chatting with agents, each taking 300s in between requests. It's plausible this saturates your server at low-latency, while needing 50*500k = 25M tokens of prefix cache. |
|
@nvjullin right i mean for AgentX. Real life makes sense |
|
It's exactly the agentX low-latency case concurrency=1. |
Adapt based on @nvrohanv 's draft.
Motivation
With
--enable-hierarchical-cache, the scheduler callscheck_hicache_eventsevery step. It counts each rank's completed write-back (D→H) and load-back (H→D) acks and MIN-all-reduces the counts over the attention-TP CPU group. That way every rank consumes the same acks and the radix trees stay identical. The all-reduce is a blocking gloo collective on the critical path of every step, decode included.The overlap scheduler hides it while the GPU step is long. Once the GPU step is short, as with fast decode kernels or small or sparse models at low concurrency, the CPU loop becomes the critical path. That one collective then adds directly to every decode step.
Modifications
Async ready-count sync, on by default.
--disable-hicache-async-ready-syncrestores the blocking per-step sync, which is what stock main does, for example when debugging.In the results below, flag on is this PR's default and flag off is
--disable-hicache-async-ready-sync, which is identical to stock main. The data was collected while the flag was still opt-in; the code paths are the same.async_op=Trueon a dedicated gloo group in step N and consumed in step N+1.--disable-hicache-async-ready-syncis the escape hatch.Files:
python/sglang/srt/mem_cache/unified_radix_cache.py: deferred-sync logic, roughly 130 lines.arg_groups/fields/memory.py,field_order.py: the--disable-hicache-async-ready-syncflag.docs/.../server_arguments.mdx,hicache_best_practices.mdx: docs. The server-args docs-coverage test requires the row.test/registered/unit/mem_cache/test_hicache_async_ready_sync.py: new CPU unit test, 15 cases.Related: #39903 (closed, similar pipelining idea). This version adds remembered-prefix handling for drains in between, the dedicated group, and the storage-attach fallback. Originally developed by Rohan Varma in an internal fork and ported to current main.
Accuracy / correctness
Setup: GB200 NVL4, TP4, HiCache
write_throughwith a 64 GB host pool. The device KV pool is capped at 65,536 tokens. 8 conversations × 4 turns of about 12K tokens each are visited round-robin, so revisits require host backup and load-back. Decoding is greedy with a fixed--random-seed.cached_tokens, flag on vs offTurn 1–3 hits are 11968, 12288 and 12544 tokens, which confirms load-back is exercised.
With the flag on, every rank logs that the deferred ready-count sync runs on its own gloo group over ranks [0, 1, 2, 3]. With it off, none does.
Unit tests (CPU, 2 simulated ranks with a fake async MIN all-reduce) cover these cases:
GSM8K on GLM-5.3-Flash-NVFP4, real weights + MTP (EAGLE 3/1/4; full 1319 questions, 5-shot, parallel 64; one boot per mode, 3 passes per boot; invalid answers 0 % throughout):
All three overlap within pass-to-pass noise of about ±0.6 points, so the flag has no measurable accuracy effect. An earlier 500-question pass suggested a 2-point gap (on 0.906/0.912 vs. off 0.932/0.926). It did not hold at full size. MTP accept length was 3.50–3.62 in every boot.
Note on GLM-5.3-Flash: token-exact comparison is not meaningful for this model. Two stock (flag-off) boots match each other on only 7/32 outputs and 28/32
cached_tokens, the same level as flag on vs. off (7/32 and 29/32). NVFP4 + MTP + hybrid linear attention is not run-to-run deterministic.Benchmark
All runs: GB200 NVL4, TP4, HiCache with a 64 GB host pool, and
--load-format dummy. Dummy weights keep the kernels and step time real while letting the FP8 geometry fit on 4 GPUs.The "12 layers" and "40 layers" setups are the same
zai-org/GLM-5.2-FP8architecture withnum_hidden_layersoverridden (the real model has 78). They are not different models. 12 layers stands in for a short GPU step, as from fast decode kernels or a small model. 40 layers is the largest depth that fits.nvidia/GLM-5.3-NVFP4runs at its full 78 layers.Each request uses a distinct 4K random prefix and 256 output tokens. Medians are of client-side streaming inter-token intervals, with no profiler attached.
Short GPU step, CPU-bound (GLM-5.2 geometry, 12 layers): the sync is on the critical path.
HiCache's per-step overhead drops from 0.47 ms to 0.08 ms, so the flag removes about 83 % of it. The bs=1 gap reproduced across two independent server boots (2.15 → 1.80 ms and 2.18 → 1.79 ms).
Real weights:
nvidia/GLM-5.3-Flash-NVFP4+ MTP (EAGLE 3/1/4): no reproducible latency change. Two server boots per mode:The first flag-on boot was faster, but the second matched flag off, so that gap is boot-to-boot variance.
Long GPU step, GPU-bound: the sync is already hidden. There is no gain and no regression.
On GLM-5.3 flag off equals HiCache off, so the sync is fully hidden there. The flag-on boot was faster than even HiCache off. Its trace shows that run's GPU kernels were about 0.2 ms per step faster: it was the first boot and ran a fresh FlashInfer autotune, while the others used the cache. That makes it boot-to-boot kernel variance, not this flag. The same pattern appeared on GLM-5.3-Flash, where a repeat flag-on boot matched flag off, so this row is treated as boot variance.
The flag pays off where the host is on the critical path, which matches the original internal measurements. When the GPU step is long, the overlap scheduler already hides the sync.
Where the time goes (torch profiler, TP0, bs=1)
check_hicache_eventson the scheduler thread, both setups_sync_hicache_ready_counts(blocking)gloo:all_reduce(about 0.38–0.39 ms)check_hicache_events, blocking12 layers (CPU-bound): the red blocking sync adds straight to every step, and with the flag the step converges to HiCache-off.
GLM-5.3-NVFP4, 78 layers (GPU-bound): the GPU runs back to back and the CPU stays ahead, so the same 0.5 ms sync is hidden under GPU work.
Each panel shows 4 consecutive bs=1 decode steps from TP0. They are rendered from the torch-profiler trace on a shared time axis.
_sync_hicache_ready_countsblock on the scheduler thread, waiting on the purple gloo all-reduce._check_hicache_events_deferred, and the all-reduce overlaps with the rest of the scheduler loop.Absolute times are inflated by Python-stack profiling; use the unprofiled tables above for numbers. The 40-layer timeline is here.
How to reproduce
Server (one of three modes)
With the flag on, each rank should log
HiCache async ready-count sync on its own gloo group over ranks [...].Correctness + bs=1 latency client
Trace
Unit test:
python -m pytest test/registered/unit/mem_cache/test_hicache_async_ready_sync.pyChecklist
base-a-test-cpu)server_arguments.mdx,hicache_best_practices.mdx)ruff(repo selection) /ruff format/isortclean🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #41839818374
Latest PR Test (Extra): ❌ Run #41839817909
Latest PR Test (AMD ROCm 10): ❌ Run #41839818071
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: