Repository navigation
[Fix] gdn: Preserve FP32 prefix-cache snapshots - #48682
wirybeaver wants to merge 1 commit into
Conversation
3aa6a07 to
3cf208b
Compare
|
@YAMY1234 @yizhang2077 Could you take a review when you are available? |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
BBuf
left a comment
There was a problem hiding this comment.
[BBuf Review Agent] LGTM. Please fix the existing CPU tracking test and wait for related CI.
|
/tag-and-rerun-ci |
|
[BBuf Review Agent] Could you rebase this onto the latest main? It conflicts with main now. |
|
[BBuf Review Agent] @wirybeaver Taking the approval back for now, it was given too early. The fix itself looks right, but it shouldn't sit behind a new opt-in env ( Could you drop the env and pick the FP32 snapshot path from the dtypes instead (FP32 state pool, lower-precision activations, tracking on), keeping the current path everywhere else? A before/after prefill number on a GDN model with prefix caching on would show whether the extra FP32 buffer costs anything. The CPU tracking fixture from the review still needs its fix, and the branch needs a rebase. |
[BBuf Review Agent] Withdrawn: the fix should not depend on a new opt-in env, see the comment on the PR.
3cf208b to
b24266a
Compare
b24266a to
660fe93
Compare
|
@BBuf Removed the env flag; FP32 snapshots are now selected from the state/activation dtypes when tracking is enabled. Also fixed the CPU tracking fixture and rebased. Added H200 Drama3 c16 results with prefix caching enabled: mean prefill batch wall time increased 0.85–0.96%, while corpus WER decreased 0.25–0.29 percentage points. Throughput was roughly unchanged. The full setup and per-visit results are in the PR description. |
|
[BBuf Review Agent] @wirybeaver Checked 660fe93: eager and graph prefill now select FP32 snapshots from the dtypes with tracking enabled, and the CPU fixture uses matching QKV/conv dtypes. That addresses my earlier code concerns. |
|
/tag-and-rerun-ci |
BBuf
left a comment
There was a problem hiding this comment.
[BBuf Review Agent] One thing to fix before this can go in.
python/sglang/srt/layers/attention/linear/gdn_backend.py:1632This also runs on the default Qwen3.5 / Qwen3-Next prefill (Triton GDN with radix cache), not only for Drama3, and it allocates an fp32 snapshot buffer per layer on top of the bf16 one. Could you post Qwen3-Next on H200 with the default radix cache, main vs this PR: prefill throughput frompython -m sglang.benchmark.servingat c1 and c128, peak memory, and the KV pool size from the startup log? Please include a batch big enough to take the eager path (more than 128 requests).
[BBuf Review Agent] Withdrawn: there are things to fix first, see the review comments.
660fe93 to
de75223
Compare
|
@BBuf This was opt-in in the original version. I've restored that scope with |
de75223 to
73b65a8
Compare
|
[BBuf Review Agent] @wirybeaver Checked 73b65a8: the flag defaults off, and both eager and graph prefill gate the extra FP32 snapshot buffer on it, with dtype and tracking checks still in place. That addresses the default-path memory/perf concern. |
|
/rerun-failed-ci |
2 similar comments
|
/rerun-failed-ci |
|
/rerun-failed-ci |
73b65a8 to
22b4013
Compare
|
/rerun-failed-ci |
Motivation
With FP32 GDN states and BF16 activations, unaligned prefix-cache snapshots currently pass through a BF16 chunk-state buffer. Copying it into an FP32 cache cannot recover the discarded precision, so a prefix hit resumes from a rounded state.
Downstream integration: sglang-omni#3234 carries this fix alongside a pinned upstream GDN graph runtime. Stacked sglang-omni#3235 enables Drama3 prompt-only prefix caching and depends on this fix to restore FP32 checkpoints without BF16 rounding.
Modification
Add the explicit enable_gdn_fp32_snapshots parameter (CLI: --enable-gdn-fp32-snapshots), defaulting to false. When enabled, tracking an FP32 state pool with FP16/BF16 activations preserves FP32 snapshots. Pass the selection and snapshot buffers to graph metadata and the kernel through explicit arguments. The default and other dtype combinations keep their existing snapshot path; no environment variable is used. Drama3 opts in through its engine configuration.
Connect the existing kernel's selected FP32 snapshot output to eager and captured Triton prefill. Convert only at the final state-pool write. Keep the upstream graph and metadata architecture; this extracts the precision fix from #46619 without its graph refactor.
Add one regression to the existing GPU suite, checking eager and graph snapshots against the actual FP32 state at a chunk boundary. Eager/graph parity alone misses their shared rounding defect.
Benchmark Result
A → B: +0.33%
A → B: +0.09%
A → B: +0.85%
A → B: -0.2913 pp
A → B: +0.48%
A → B: +4.06%
A → B: +0.96%
A → B: -0.2549 pp
H200, Fish Audio Drama3 streaming, SeedTTS English. Both arms enable the prompt-only prefix cache from sglang-omni#3235, use upstream GDN prefill CUDA graphs, FP32 live recurrent states and BF16 activations. A uses the original BF16 boundary-snapshot intermediate; B changes only snapshot export to preserve FP32. This compares snapshot precision, not cache off versus on.
Each workload ran A → B → B → A, restarting the server every visit. The main stages in each c16 visit were:
cache_salt=payload.request_id, giving every request a unique cache namespace across all stages. Identical inputs therefore cannot reuse previous requests' prefixes. Cache lookup, snapshot export, insertion and eviction remain enabled. Each timed visit had zero cached tokens, prefilled all 42,876 tokens, and exported 253 chunk-boundary snapshot rows plus three final-state rows. Here the first 256-request pass is cache-fill warmup, not priming for later hits.Conclusion
With FP32 snapshots, observed mean prefill batch wall time increased by less than 1% (0.85–0.96%), throughput was roughly unchanged, and corpus WER was 0.25–0.29 percentage points lower. The wall-time difference includes batching and scheduling effects; it does not isolate snapshot overhead.
Per-visit results
CI States
Latest PR Test (Base): ❌ Run #44202051566
Latest PR Test (Extra): ⊘ Skipped -- add the
run-ci-nv-extralabel to opt in.Latest PR Test (AMD ROCm 10): ❌ Run #44202051235
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: