Repository navigation
Conversation
|
@hnyls2002 Can you help review this PR? This is a follow-up to #41520. I'd like to check which What I see
Net effect on main: selective backs up a prefix only when a second request uses it. On Qwen3.5-397B-A17B-FP8 AgentX (TP4 MI355X, concurrency=80), most prefixes are evicted before that:
Under the new counting, threshold 1 backs up at the first insert, which is the same timing as Which behavior do you want for
Many thanks! |
This comment was marked as outdated.
This comment was marked as outdated.
77b53ec to
5c8c069
Compare
|
Required checks failed on 9cd838b; this PR cannot be merged until they pass. PR title (run)
|
Motivation
write_through_selectivecan lose a reusable prefix before HiCache copies it to host memory. This causes a large cache-hit and throughput regression under GPU KV-cache pressure.Root cause
#41520's final PR body says that "
write_through_thresholddefaults to 1 for both write-through policies, keeping the first-finished-request backup timing ofwrite_through_selective."However, commit 21f8950063 changed both initialization and runtime policy switching from threshold 1 to
1 if policy == "write_through" else 2. The same commit replaced documentation saying that selective write-through backs up a prefix "once the first request using it finishes" with "after the access frequency exceeds a threshold." The merged implementation therefore giveswrite_through_selectivethreshold 2, which contradicts the final PR body.The first finished request therefore raises a prefix to
hit_count == 1without backing it up. If the GPU cache evicts that prefix before a second request finishes, the prefix never reaches host memory. In the Qwen3.5 AgentX concurrency-80 replay, host KV use fell to 32%, cache hit rate fell to 51%, and output rate fell to 646 token/s.Modifications
write_through_thresholdto 1 for bothwrite_throughandwrite_through_selectiveduring HiCache initialization.write_backthreshold at 2.Accuracy Tests
Qwen3.5-397B-A17B-FP8, TP4 on four MI355X GPUs, EAGLE with real draft acceptance, FP8 KV cache, and
write_through_selective. Image:lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261008.The targeted CPU regression test also passes:
python3 -m pytest -q test/registered/unit/mem_cache/test_unified_hicache_transfer_lifecycle.py -k selective_write_through_uses_first_finished_request_thresholdSpeed Tests and Profiling
Qwen3.5-397B-A17B-FP8 AgentX, TP4 on four MI355X GPUs, concurrency 80, EAGLE (3 steps, top-k 1, 4 draft tokens), FP8 KV cache, HiCache
write_through_selective, 400 GB host cache per worker, and a 6.8M-token GPU KV cap. Both arms use the same20261006image and source stack; the fix arm changes only the selective write-through threshold from 2 to 1. AgentX ran in exclusive mode.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #38522920551
Latest PR Test (Extra): ❌ Run #38522920519
Latest PR Test (AMD ROCm 10): ❌ Run #38522920548