Skip to content

[Perf] hicache: Enable async ready-count sync by default - #45795

Open
wenscarl wants to merge 6 commits into
sgl-project:mainfrom
wenscarl:hicache-async-ready-sync
Open

wenscarl wants to merge 6 commits into
sgl-project:mainfrom
wenscarl:hicache-async-ready-sync

Conversation

@wenscarl

@wenscarl wenscarl commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Adapt based on @nvrohanv 's draft.

Motivation

With --enable-hierarchical-cache, the scheduler calls check_hicache_events every step. It counts each rank's completed write-back (D→H) and load-back (H→D) acks and MIN-all-reduces the counts over the attention-TP CPU group. That way every rank consumes the same acks and the radix trees stay identical. The all-reduce is a blocking gloo collective on the critical path of every step, decode included.

The overlap scheduler hides it while the GPU step is long. Once the GPU step is short, as with fast decode kernels or small or sparse models at low concurrency, the CPU loop becomes the critical path. That one collective then adds directly to every decode step.

Modifications

Async ready-count sync, on by default. --disable-hicache-async-ready-sync restores the blocking per-step sync, which is what stock main does, for example when debugging.

In the results below, flag on is this PR's default and flag off is --disable-hicache-async-ready-sync, which is identical to stock main. The data was collected while the flag was still opt-in; the code paths are the same.

  • The ready-count all-reduce is issued with async_op=True on a dedicated gloo group in step N and consumed in step N+1.
  • Safety rests on two invariants:
    1. Acks complete in queue order, and queues only grow at the tail. So a prefix that was ready on every rank in step N is still ready in step N+1.
    2. In step N each rank remembers the ack object that ends its ready prefix. In step N+1 it pops up to and including the agreed one. If a blocking write-back drain consumed it in between, it pops nothing, so there is no double consumption.
  • The duplicate-reclaim digest check rides along and fires one step later.
  • There is one async op per step on its own group, issued in the same order on every rank, so no new collective ordering is introduced.
  • Scope: host-cache path with more than one attention rank and no storage backend, PP or attention CP. Other setups keep the synchronous sync and log the reason once at info level.
  • Runtime storage attach: if a storage backend is attached later via the admin API, the in-flight sync is settled and the cache falls back to the synchronous path for good. The storage queues are only drained on that path. This fallback is added in this PR.
  • Cost: the forward pass and TTFT are not delayed, because loaded KV is ordered by CUDA events, not by the ack. What moves one step later is the bookkeeping: finished load-back and write-through nodes are unlocked, and become evictable, one step later. Under tight device memory, slightly fewer tokens are evictable for that step. The cache also holds one extra gloo group per attention TP group.
  • Validation scope: GB200, single node, TP4. PD decode, multi-node TP and non-NVIDIA backends are not yet covered; --disable-hicache-async-ready-sync is the escape hatch.

Files:

  • python/sglang/srt/mem_cache/unified_radix_cache.py: deferred-sync logic, roughly 130 lines.
  • arg_groups/fields/memory.py, field_order.py: the --disable-hicache-async-ready-sync flag.
  • docs/.../server_arguments.mdx, hicache_best_practices.mdx: docs. The server-args docs-coverage test requires the row.
  • test/registered/unit/mem_cache/test_hicache_async_ready_sync.py: new CPU unit test, 15 cases.

Related: #39903 (closed, similar pipelining idea). This version adds remembered-prefix handling for drains in between, the dedicated group, and the storage-attach fallback. Originally developed by Rohan Varma in an internal fork and ported to current main.

Accuracy / correctness

Setup: GB200 NVL4, TP4, HiCache write_through with a 64 GB host pool. The device KV pool is capped at 65,536 tokens. 8 conversations × 4 turns of about 12K tokens each are visited round-robin, so revisits require host backup and load-back. Decoding is greedy with a fixed --random-seed.

model setup output token ids, flag on vs off cached_tokens, flag on vs off
GLM-5.2 geometry, 12 layers (dummy) 32 / 32 identical 32 / 32 identical
GLM-5.2 geometry, 40 layers (dummy) 32 / 32 identical 32 / 32 identical
GLM-5.3-NVFP4, full 78 layers (dummy) 32 / 32 identical 32 / 32 identical
GLM-5.3-Flash-NVFP4, real weights + MTP (hybrid linear-attn) not deterministic run-to-run (see note) not deterministic run-to-run (see note)
  • Turn 1–3 hits are 11968, 12288 and 12544 tokens, which confirms load-back is exercised.

  • With the flag on, every rank logs that the deferred ready-count sync runs on its own gloo group over ranks [0, 1, 2, 3]. With it off, none does.

  • Unit tests (CPU, 2 simulated ranks with a fake async MIN all-reduce) cover these cases:

    • consume-count edge cases;
    • the result is consumed exactly one step later;
    • lockstep under rank skew, with new acks appended at the tail;
    • no double consumption after a blocking drain;
    • the digest-divergence assert;
    • the storage-attach fallback;
    • pending-sync drop on reset.
  • GSM8K on GLM-5.3-Flash-NVFP4, real weights + MTP (EAGLE 3/1/4; full 1319 questions, 5-shot, parallel 64; one boot per mode, 3 passes per boot; invalid answers 0 % throughout):

    pass 1 pass 2 pass 3 mean
    flag off (stock) 0.917 0.923 0.911 0.917
    flag on 0.915 0.925 0.922 0.921
    HiCache disabled 0.930 0.920 0.924 0.925

    All three overlap within pass-to-pass noise of about ±0.6 points, so the flag has no measurable accuracy effect. An earlier 500-question pass suggested a 2-point gap (on 0.906/0.912 vs. off 0.932/0.926). It did not hold at full size. MTP accept length was 3.50–3.62 in every boot.

  • Note on GLM-5.3-Flash: token-exact comparison is not meaningful for this model. Two stock (flag-off) boots match each other on only 7/32 outputs and 28/32 cached_tokens, the same level as flag on vs. off (7/32 and 29/32). NVFP4 + MTP + hybrid linear attention is not run-to-run deterministic.

Benchmark

All runs: GB200 NVL4, TP4, HiCache with a 64 GB host pool, and --load-format dummy. Dummy weights keep the kernels and step time real while letting the FP8 geometry fit on 4 GPUs.

The "12 layers" and "40 layers" setups are the same zai-org/GLM-5.2-FP8 architecture with num_hidden_layers overridden (the real model has 78). They are not different models. 12 layers stands in for a short GPU step, as from fast decode kernels or a small model. 40 layers is the largest depth that fits. nvidia/GLM-5.3-NVFP4 runs at its full 78 layers.

Each request uses a distinct 4K random prefix and 256 output tokens. Medians are of client-side streaming inter-token intervals, with no profiler attached.

Short GPU step, CPU-bound (GLM-5.2 geometry, 12 layers): the sync is on the critical path.

flag off (stock) flag on HiCache disabled (reference)
bs=1 decode step (8×4-turn trace, revisited ctx) 2.18 ms 1.79 ms (−18 %) 1.71 ms
ITL, concurrency 1 2.19 ms 1.79 ms (−18 %) 1.69 ms
ITL, concurrency 8 2.27 ms 1.85 ms (−19 %) 1.76 ms
ITL, concurrency 32 2.43 ms 2.02 ms (−17 %) 1.94 ms

HiCache's per-step overhead drops from 0.47 ms to 0.08 ms, so the flag removes about 83 % of it. The bs=1 gap reproduced across two independent server boots (2.15 → 1.80 ms and 2.18 → 1.79 ms).

Real weights: nvidia/GLM-5.3-Flash-NVFP4 + MTP (EAGLE 3/1/4): no reproducible latency change. Two server boots per mode:

flag off #1 flag off #2 flag on #1 flag on #2
bs=1 decode iteration 5.89 ms 5.87 ms 5.57 ms 5.89 ms
ITL, concurrency 1 / 8 / 32 6.12 / 8.50 / 10.47 5.94 / 8.35 / 10.46 5.50 / 8.11 / 10.26 5.81 / 8.81 / 10.09

The first flag-on boot was faster, but the second matched flag off, so that gap is boot-to-boot variance.

Long GPU step, GPU-bound: the sync is already hidden. There is no gain and no regression.

flag off flag on HiCache disabled
GLM-5.2 geometry, 40 layers: bs=1 step 4.48 ms 4.48 ms 4.48 ms
40 layers: ITL, concurrency 1 / 8 / 32 4.44 / 4.78 / 4.96 4.43 / 4.78 / 4.96 4.43 / 4.78 / 4.96
GLM-5.3-NVFP4, 78 layers: bs=1 step 6.95 ms 6.60 ms (boot variance, see note) 6.95 ms
78 layers: ITL, concurrency 1 / 8 / 32 6.82 / 7.56 / 8.41 6.48 / 7.11 / 8.20 (boot variance) 6.82 / 7.56 / 8.41

On GLM-5.3 flag off equals HiCache off, so the sync is fully hidden there. The flag-on boot was faster than even HiCache off. Its trace shows that run's GPU kernels were about 0.2 ms per step faster: it was the first boot and ran a fresh FlashInfer autotune, while the others used the cache. That makes it boot-to-boot kernel variance, not this flag. The same pattern appeared on GLM-5.3-Flash, where a repeat flag-on boot matched flag off, so this row is treated as boot variance.

The flag pays off where the host is on the critical path, which matches the original internal measurements. When the GPU step is long, the overlap scheduler already hides the sync.

Where the time goes (torch profiler, TP0, bs=1)

per decode step flag off flag on HiCache off
12 layers: step period (profiler-inflated) 5.42 ms 4.92 ms 4.78 ms
12 layers: GPU idle per step 3.69 ms 3.46 ms 3.26 ms
78-layer GLM-5.3: step period (profiler-inflated) 6.88 ms 6.59 ms 6.89 ms
78-layer GLM-5.3: GPU idle per step 0.10 ms 0.05 ms 0.10 ms
check_hicache_events on the scheduler thread, both setups 0.54–0.55 ms 0.12 ms n/a
— of which _sync_hicache_ready_counts (blocking) 0.53 ms n/a n/a
gloo:all_reduce (about 0.38–0.39 ms) inside check_hicache_events, blocking overlapped, returns before it completes n/a

12 layers (CPU-bound): the red blocking sync adds straight to every step, and with the flag the step converges to HiCache-off.

12-layer decode-step timelines: flag off vs flag on vs HiCache off

GLM-5.3-NVFP4, 78 layers (GPU-bound): the GPU runs back to back and the CPU stays ahead, so the same 0.5 ms sync is hidden under GPU work.

GLM-5.3 78-layer decode-step timelines: flag off vs flag on vs HiCache off

Each panel shows 4 consecutive bs=1 decode steps from TP0. They are rendered from the torch-profiler trace on a shared time axis.

  • Flag off: each step has a red _sync_hicache_ready_counts block on the scheduler thread, waiting on the purple gloo all-reduce.
  • Flag on: that block becomes a short green _check_hicache_events_deferred, and the all-reduce overlaps with the rest of the scheduler loop.

Absolute times are inflated by Python-stack profiling; use the unprofiled tables above for numbers. The 40-layer timeline is here.

How to reproduce

Server (one of three modes)
# MODE=on: this PR (default); MODE=off: add --disable-hicache-async-ready-sync (same as stock);
# drop the hicache flags for the HiCache-off reference
python -m sglang.launch_server \
  --model-path zai-org/GLM-5.2-FP8 --load-format dummy --trust-remote-code \
  --json-model-override-args '{"num_hidden_layers": 12}' \   # same model, depth reduced
  --tp-size 4 --page-size 64 --mem-fraction-static 0.8 \
  --max-total-tokens 65536 --random-seed 42 --port 30000 \
  --enable-hierarchical-cache --hicache-size 64 --hicache-write-policy write_through \
  --disable-hicache-async-ready-sync   # only for MODE=off

With the flag on, each rank should log HiCache async ready-count sync on its own gloo group over ranks [...].

Correctness + bs=1 latency client
import random, statistics, time, requests
URL = "http://127.0.0.1:30000"; rng = random.Random(1234)
def gen(ids, n, stream=False):
    return requests.post(URL + "/generate", stream=stream, json={"input_ids": ids, "stream": stream,
        "sampling_params": {"temperature": 0, "max_new_tokens": n, "ignore_eos": True}})
requests.post(URL + "/flush_cache")
convs = [[rng.randrange(100, 150000) for _ in range(12000)] for _ in range(8)]
outs = []
for turn in range(4):                      # 8 x 12K ctx > 64K device pool -> backup + load-back
    for c in range(8):
        r = gen(convs[c], 32).json(); outs.append((r["output_ids"], r["meta_info"]["cached_tokens"]))
        convs[c] += r["output_ids"] + [rng.randrange(100, 150000) for _ in range(256)]
print(outs)                                # compare across MODE=on / MODE=off
for c in (0, 3, 5, 7, 1):                  # bs=1 decode on revisited contexts
    t, gaps = None, []
    for line in gen(convs[c], 512, stream=True).iter_lines():
        if line.startswith(b"data:") and line.strip() != b"data: [DONE]":
            now = time.perf_counter(); gaps += [(now - t) * 1000] if t else []; t = now
    print(f"median step {statistics.median(gaps[16:]):.2f} ms")
Trace
SGLANG_TORCH_PROFILER_DIR=/tmp/trace  # set on the server
curl -s localhost:30000/start_profile -H 'Content-Type: application/json' \
  -d '{"num_steps": 40, "activities": ["CPU", "GPU"]}'
# then send one bs=1 request with ~400 output tokens; open TP-0.trace.json.gz in Perfetto and look at
# check_hicache_events -> _sync_hicache_ready_counts (off) vs _check_hicache_events_deferred (on)

Unit test: python -m pytest test/registered/unit/mem_cache/test_hicache_async_ready_sync.py

Checklist

  • Unit tests added (CPU-only, registered in base-a-test-cpu)
  • Docs updated (server_arguments.mdx, hicache_best_practices.mdx)
  • ruff (repo selection) / ruff format / isort clean
  • AI assistance was used (Claude Code)

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #41839818374
Latest PR Test (Extra): ❌ Run #41839817909
Latest PR Test (AMD ROCm 10): ❌ Run #41839818071

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Issue the per-step ready-count all-reduce asynchronously on a dedicated gloo
group in step N and consume it in step N+1, removing a blocking collective
from the scheduler critical path.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… docs

The deferred ready-count sync skips the storage-queue drain, so a storage
backend attached at runtime would never have its queues serviced. Settle
the in-flight sync and switch to the synchronous path once storage is on.

Add CPU unit tests for the deferred-consume invariants (ack prefix kept
across a step, remembered end-of-prefix after a drain, digest check, the
storage-attach fallback) and document the flag, which the server-args
docs coverage test requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 483c9cf.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang labels Oct 26, 2026
…ache-async-ready-sync

# Conflicts:
#	docs/docs/advanced_features/hicache_best_practices.mdx
@wenscarl

wenscarl commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

cc. @nvpohanh @YAMY1234
@Fridge003 plz ping a hicache expert to review. Thanks!

@nvpohanh

Copy link
Copy Markdown
Collaborator

HiCache expert would be @hnyls2002

@nvpohanh

Copy link
Copy Markdown
Collaborator

cc @nvjullin

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

@YAMY1234

YAMY1234 commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

@wenscarl Could you check whether we'd still run into the problem described in the PR if we used writeback? Just want to check whether this overhead is specific to agg + lowlatency + write-through. Thanks!

detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
check_hicache_events runs a blocking gloo MIN-all-reduce of the
write-back/load-back ready counts every scheduler step; on TP2 short-decode
steps that CPU collective is a large share of the step. With the new flag
(default False) the same reduction is issued asynchronously on its own
gloo group in step N and consumed in step N+1: acks complete in queue
order and queues only grow at the tail, so the agreed ready prefix stays
valid one step later, and _deferred_consume_count re-finds the agreed
last-ready ack by identity so a blocking drain in between is never
double-consumed. A deferred digest pair keeps the write_back
duplicate-reclaim victim-order assertion across ranks. Runtime storage
attach settles the in-flight sync and falls back to synchronous for good
(the deferred path skips the storage-queue drain). reset() drains the
pending op.

Union notes: lands after the sgl-project#46451 port in unified_radix_cache.py — the
insertion sits between flush_pending_backups() (entry-199 batched-backup
drain; its SGLANG_ENABLE_HICACHE_BATCHED_BACKUP default-OFF semantics are
untouched: the new flag is an orthogonal memory-group bool, flag-off is
the unchanged synchronous path) and the four-tuple
_sync_hicache_ready_counts call our entry-217/218 region built on. The
deferred gate refuses storage/buffer_only/PP/attn-CP/single-rank setups
with a warning, matching the landed _sync_hicache_ready_counts payload
(storage-queue sizes stay on the sync path). Ports the full head
(29e8125 + 39a0c0d incl. the runtime-attach refactor + docs); the
244-line CPU test suite (7P) drives two simulated ranks through the real
methods. Docs rows land next to the entry-219 mamba-sizing note.
Fork-stub adaptation: our test_hicache_pp_sync_drain __new__-built stubs
now pre-memoize _hcs_state=False (same as upstream's own async harness
pre-setting the state) so the per-step gate never resolves the memory bag
these stubs never publish; 4P.

(cherry picked from commit 6ffe9cd)
PR: sgl-project#45795 (open, ported)
(cherry picked from commit b357ce2961bdfa723091e0b3bd44b989243756d8)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…),199,217,218,219,253,290d,293,294

Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the
#46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py.

What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk
verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait
msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired
REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired,
the reason="deadline" bookkeeping, belief-heal suppression on deadline, the
_invalidate_absent_from_hit_query move to the drain head + KV-verdict guard +
deadline gate, the release_aborted_request / finish_storage_prefetch_admission
wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800
(unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT
obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P).

Re-planted here:
- entry 294 (sgl-project#45795): the FULL deferred ready-count sync family
  (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback,
  _check_hicache_events_deferred one-op-per-step on its own gloo group,
  _consume_deferred_ready_sync with the digest MIN/-MAX assert,
  _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events
  branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync
  collection error (the B3-caused 1err in the merge gates).
- entries 199+218: the batched write-through backup family VERBATIM from the
  patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups,
  _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node,
  _parent_backed_up, _release_queued_backup, _backup_queued_batch,
  _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the
  node_ids= ack-list through _execute_kv_backup into the controller write
  (B3-2), the flush_pending_backups guard, and the writing_check(write_back)
  drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule).
  _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE
  (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record).
- entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache
  reads with the nonnegative-finite ValueError, wait_full, the retry cadence
  (host_capacity_deadline/next_retry/retries — fields planted on the operation
  in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with
  identity snapshots, the info.operation-is-not-operation late-query guards,
  and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical.
- entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the
  endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining
  prefetches a smaller endpoint would fit).
- entry 219 producer: _record_host_pool_evictions on evict_host (before
  _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}.
- entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through
  merge_source_transfers (every destination, not matching_sources[0]; the two
  AssertionErrors it subsumes are structurally guaranteed by the resolver).
- entry 30 rider: load_back unwinds (drop both locks, report miss) when any
  component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba
  component sets -- component side rides B3-5's mamba.py cluster).
- entry 253: the per-call-site file_storage_path fold DELETED here (it moved
  into _generate_storage_config, every attach path, B3-1-verified).
- test_chunked_publish_release_disjointness.py (entry 307's tripwire):
  re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...)
  per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line
  is the documented cargo/torch-sys ENV class (fails identically at base).

ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after):
  test_hicache_async_ready_sync          1err -> 7P/0F   ** B3-caused error CLEARED **
  test_hybrid_storage_sparse_allocation  12P/1F -> 13P/0F/14sub  ** entry-290(d) closed **
  test_storage_prefetch_lifecycle                -> 16P/0F/13sub
  test_hicache_trailing_endpoint (gate)          -> 17P/0F (unmoved by this cluster; controller-side B3-2)
  test_buffer_mode_sidecar                        -> 6P/0F/6sub
  test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green)
  test_mamba_path_state_cap                      -> 5P/5S (skips = honest, env)
  test_decode_retraction_backup             2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log)
  test_hicache_mamba_sizing                      -> 30P/0F/17sub (unmoved)
  URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S
    UNMOVED — 100% of the errors are 'No accelerator ... available' from the
    upstream kits' fixtures.py build_fixture calling get_device() (entry 310's
    "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's
    own per-file numbers (core 27P/465F etc.) match the damage totals exactly;
    this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here).
Lint: ruff format + F821/E9 clean on both files; py_compile + import OK.
No GPU executed; no push.
@wenscarl

Copy link
Copy Markdown
Collaborator Author

@YAMY1234 It isn't specific to write-through. With --enable-hierarchical-cache, check_hicache_events → _sync_hicache_ready_counts runs the MIN all-reduce on every scheduler step, whatever --hicache-write-policy is set to. The policy only changes when backups are issued (on insert for write-through, on eviction for write-back). That changes how many acks are queued, not whether the collective runs. In the traces above, the blocking sync costs about 0.53 ms per step during steady bs=1 decode, when there are almost no new write acks. So the cost is the gloo round-trip itself, not the ack volume. It isn't agg-only either: the PD decode path calls the same check_hicache_events (disaggregation/decode.py).

What is specific is the regime. It shows end-to-end only when the scheduler loop is on the critical path, i.e. low concurrency or a short GPU step, as in the 12-layer rows. In GPU-bound setups (40 and 78 layers) the overlap scheduler hides it. Happy to add a write-back run to the tables if that would help.

Address review: drop the defensive getattr and the None/False/group
tri-state. The dedicated gloo group is created once at the end of
init_hicache (None = synchronous path), and the per-step check only
handles a runtime storage attach. Trim the explanatory comment and add
parametrized tests that every ineligible setup keeps the sync path
without creating a group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wenscarl
wenscarl requested a review from nvpohanh November 2, 2026 18:34
Replace the opt-in --hicache-async-ready-sync with
--disable-hicache-async-ready-sync. Unsupported setups fall back to the
synchronous sync with an info log instead of a warning. Correct the docs:
the deferred ack only delays unlocking finished nodes, not the forward pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wenscarl wenscarl changed the title [Perf] hicache: Add opt-in async ready-count sync [Perf] hicache: Enable async ready-count sync by default Nov 4, 2026
@kpham-sgl

Copy link
Copy Markdown
Collaborator

@wenscarl I am a little confused. So the gain from this PR only comes in low latency regime right? In that scenario why would we want to enable HiCache?

@nvjullin

nvjullin commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

@kpham-sgl Low-latency can still have a large prefix cache footprint if subsequent requests has a long time delay in between. For example, suppose there are 50 people chatting with agents, each taking 300s in between requests. It's plausible this saturates your server at low-latency, while needing 50*500k = 25M tokens of prefix cache.

@kpham-sgl

Copy link
Copy Markdown
Collaborator

@nvjullin right i mean for AgentX. Real life makes sense

@wenscarl

Copy link
Copy Markdown
Collaborator Author

It's exactly the agentX low-latency case concurrency=1.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants