Skip to content

[AMD] Restore first-request backup for HiCache selective writes - #43969

Closed
chuyeh wants to merge 2 commits into
sgl-project:mainfrom
chuyeh:fix/hicache-selective-threshold-1
Closed

chuyeh wants to merge 2 commits into
sgl-project:mainfrom
chuyeh:fix/hicache-selective-threshold-1

Conversation

@chuyeh

@chuyeh chuyeh commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

write_through_selective can lose a reusable prefix before HiCache copies it to host memory. This causes a large cache-hit and throughput regression under GPU KV-cache pressure.

Root cause

#41520's final PR body says that "write_through_threshold defaults to 1 for both write-through policies, keeping the first-finished-request backup timing of write_through_selective."

However, commit 21f8950063 changed both initialization and runtime policy switching from threshold 1 to 1 if policy == "write_through" else 2. The same commit replaced documentation saying that selective write-through backs up a prefix "once the first request using it finishes" with "after the access frequency exceeds a threshold." The merged implementation therefore gives write_through_selective threshold 2, which contradicts the final PR body.

The first finished request therefore raises a prefix to hit_count == 1 without backing it up. If the GPU cache evicts that prefix before a second request finishes, the prefix never reaches host memory. In the Qwen3.5 AgentX concurrency-80 replay, host KV use fell to 32%, cache hit rate fell to 51%, and output rate fell to 646 token/s.

Modifications

  • Set write_through_threshold to 1 for both write_through and write_through_selective during HiCache initialization.
  • Apply the same threshold when the write policy changes through runtime storage attachment.
  • Keep the write_back threshold at 2.
  • Add a CPU regression test for initialization and runtime policy switching.

Accuracy Tests

Qwen3.5-397B-A17B-FP8, TP4 on four MI355X GPUs, EAGLE with real draft acceptance, FP8 KV cache, and write_through_selective. Image: lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261008.

Test Questions Accuracy Invalid Latency Output throughput
GSM8K 5-shot 1,319 0.951 0.011 51.2 s 4,206 token/s

The targeted CPU regression test also passes:

python3 -m pytest -q test/registered/unit/mem_cache/test_unified_hicache_transfer_lifecycle.py -k selective_write_through_uses_first_finished_request_threshold

Speed Tests and Profiling

Qwen3.5-397B-A17B-FP8 AgentX, TP4 on four MI355X GPUs, concurrency 80, EAGLE (3 steps, top-k 1, 4 draft tokens), FP8 KV cache, HiCache write_through_selective, 400 GB host cache per worker, and a 6.8M-token GPU KV cap. Both arms use the same 20261006 image and source stack; the fix arm changes only the selective write-through threshold from 2 to 1. AgentX ran in exclusive mode.

Metric Merged code (threshold 2) This PR (threshold 1) Fix vs. merged
Output rate 646 token/s 2,584 token/s +300.0%
Interactivity p50 10.2 token/s/user 45.2 token/s/user +343.1%
ITL p50 97.7 ms 22.1 ms -77.4%
Cache hit rate 51% 94% +43 pp
Host KV use 32% 100% +68 pp
Completed requests 3,079 10,162 +230.0%

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #38522920551
Latest PR Test (Extra): ❌ Run #38522920519
Latest PR Test (AMD ROCm 10): ❌ Run #38522920548

@chuyeh

chuyeh commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

@hnyls2002 Can you help review this PR? This is a follow-up to #41520. I'd like to check which write_through_selective behavior you intended, because the merged code and the #41520 description seem to disagree.

What I see

Net effect on main: selective backs up a prefix only when a second request uses it. On Qwen3.5-397B-A17B-FP8 AgentX (TP4 MI355X, concurrency=80), most prefixes are evicted before that:

main (threshold 2) this PR (threshold 1)
Output rate 646 tok/s 2,584 tok/s
Cache hit rate 51% 94%
Host KV use 32% 100%

Under the new counting, threshold 1 backs up at the first insert, which is the same timing as write_through. So this PR makes the backup mechanism of two policies identical.

Which behavior do you want for write_through_selective?

  1. Back up only on reuse (current main). Then I'll close this PR and switch our recipe to write_through.
  2. Back up when the first request finishes (the [mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes #41520 description). Threshold 1 alone doesn't give that; selective would need to count hits only at the finishing insert. I can send that instead.
  3. Same as write_through. Then this PR is enough.

Many thanks!

@yichiche yichiche added the run-ci CI: run the baseline test suite on this PR label Oct 13, 2026
@chuyeh

This comment was marked as outdated.

@chuyeh
chuyeh force-pushed the fix/hicache-selective-threshold-1 branch from 77b53ec to 5c8c069 Compare October 14, 2026 04:15
@github-actions

Copy link
Copy Markdown
Contributor

Required checks failed on 9cd838b; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

@chuyeh chuyeh closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants