Skip to content

[Fix] gdn: Preserve FP32 prefix-cache snapshots - #48682

Open
wirybeaver wants to merge 1 commit into
sgl-project:mainfrom
wirybeaver:fix/gdn-fp32-prefix-snapshots
Open

wirybeaver wants to merge 1 commit into
sgl-project:mainfrom
wirybeaver:fix/gdn-fp32-prefix-snapshots

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With FP32 GDN states and BF16 activations, unaligned prefix-cache snapshots currently pass through a BF16 chunk-state buffer. Copying it into an FP32 cache cannot recover the discarded precision, so a prefix hit resumes from a rounded state.

Downstream integration: sglang-omni#3234 carries this fix alongside a pinned upstream GDN graph runtime. Stacked sglang-omni#3235 enables Drama3 prompt-only prefix caching and depends on this fix to restore FP32 checkpoints without BF16 rounding.

Modification

Add the explicit enable_gdn_fp32_snapshots parameter (CLI: --enable-gdn-fp32-snapshots), defaulting to false. When enabled, tracking an FP32 state pool with FP16/BF16 activations preserves FP32 snapshots. Pass the selection and snapshot buffers to graph metadata and the kernel through explicit arguments. The default and other dtype combinations keep their existing snapshot path; no environment variable is used. Drama3 opts in through its engine configuration.

Connect the existing kernel's selected FP32 snapshot output to eager and captured Triton prefill. Convert only at the final state-pool write. Keep the upstream graph and metadata architecture; this extracts the precision fix from #46619 without its graph refactor.

Add one regression to the existing GPU suite, checking eager and graph snapshots against the actual FP32 state at a chunk boundary. Eager/graph parity alone misses their shared rounding defect.

Benchmark Result

Workload Concurrency Throughput (req/s) P50 TTFA (ms) Mean prefill batch wall time (ms) Corpus WER (%)
Hot cache c16 A/B: 12.98 / 13.02
A → B: +0.33%
A/B: 162.55 / 162.70
A → B: +0.09%
A/B: 14.24 / 14.36
A → B: +0.85%
A/B: 1.4567% / 1.1653%
A → B: -0.2913 pp
Cache miss c16 A/B: 12.36 / 12.42
A → B: +0.48%
A/B: 172.40 / 179.40
A → B: +4.06%
A/B: 19.70 / 19.89
A → B: +0.96%
A/B: 1.6752% / 1.4202%
A → B: -0.2549 pp

H200, Fish Audio Drama3 streaming, SeedTTS English. Both arms enable the prompt-only prefix cache from sglang-omni#3235, use upstream GDN prefill CUDA graphs, FP32 live recurrent states and BF16 activations. A uses the original BF16 boundary-snapshot intermediate; B changes only snapshot export to preserve FP32. This compares snapshot precision, not cache off versus on.

Each workload ran A → B → B → A, restarting the server every visit. The main stages in each c16 visit were:

Stage Inputs Purpose
Substantial warmup 128 separate samples Warm up the server/runtime; excluded from results
Cache priming / cache-fill warmup 256 evaluation inputs Populate the cache in the hot-cache workload; exercise cache writes in the miss workload
Timed Resend the same 256 inputs Measure performance; retain audio from visits 3 (B) and 4 (A) for WER
  • Hot cache: requests share the normal cache namespace, so timed requests can reuse prefixes inserted during priming. Each timed visit reused 35,136 tokens and prefilled the remaining 7,740 tokens. It restored cached snapshots, exported no new chunk-boundary snapshot rows, and wrote three final-state rows.
  • Cache miss: both arms apply the same experiment-only cache_salt=payload.request_id, giving every request a unique cache namespace across all stages. Identical inputs therefore cannot reuse previous requests' prefixes. Cache lookup, snapshot export, insertion and eviction remain enabled. Each timed visit had zero cached tokens, prefilled all 42,876 tokens, and exported 253 chunk-boundary snapshot rows plus three final-state rows. Here the first 256-request pass is cache-fill warmup, not priming for later hits.

Conclusion

With FP32 snapshots, observed mean prefill batch wall time increased by less than 1% (0.85–0.96%), throughput was roughly unchanged, and corpus WER was 0.25–0.29 percentage points lower. The wall-time difference includes batching and scheduling effects; it does not isolate snapshot overhead.

Per-visit results
Workload Visit Throughput (req/s) P50 TTFA (ms) Mean prefill batch wall time (ms) Corpus WER (%)
Hot cache 1: A 12.76 159.80 14.37 Not scored
Hot cache 2: B 12.81 163.60 14.25 Not scored
Hot cache 3: B 13.23 161.80 14.46 1.1653%
Hot cache 4: A 13.19 165.30 14.10 1.4567%
Cache miss 1: A 12.46 171.30 19.53 Not scored
Cache miss 2: B 12.45 181.80 19.85 Not scored
Cache miss 3: B 12.39 177.00 19.93 1.4202%
Cache miss 4: A 12.26 173.50 19.87 1.6752%

CI States

Latest PR Test (Base): ❌ Run #44202051566
Latest PR Test (Extra): ⊘ Skipped -- add the run-ci-nv-extra label to opt in.
Latest PR Test (AMD ROCm 10): ❌ Run #44202051235

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@wirybeaver

Copy link
Copy Markdown
Contributor Author

@YAMY1234 @yizhang2077 Could you take a review when you are available?

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

This comment has been minimized.

@wirybeaver wirybeaver changed the title [Fix] Add opt-in FP32 GDN prefix-cache snapshots [Fix] Preserve FP32 GDN prefix snapshots with an opt-in flag Nov 8, 2026
@github-actions

This comment has been minimized.

@wirybeaver wirybeaver changed the title [Fix] Preserve FP32 GDN prefix snapshots with an opt-in flag [Fix] gdn: Add opt-in FP32 prefix-cache snapshots Nov 8, 2026
BBuf
BBuf previously approved these changes Nov 15, 2026

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] LGTM. Please fix the existing CPU tracking test and wait for related CI.

@BBuf

BBuf commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] Could you rebase this onto the latest main? It conflicts with main now.

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 15, 2026
@BBuf

BBuf commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @wirybeaver Taking the approval back for now, it was given too early. The fix itself looks right, but it shouldn't sit behind a new opt-in env (SGLANG_ENABLE_GDN_FP32_PREFIX_SNAPSHOTS): with it off, a model with FP32 states and BF16 activations keeps restoring rounded snapshots, which is the bug this PR describes.

Could you drop the env and pick the FP32 snapshot path from the dtypes instead (FP32 state pool, lower-precision activations, tracking on), keeping the current path everywhere else? A before/after prefill number on a GDN model with prefix caching on would show whether the extra FP32 buffer costs anything.

The CPU tracking fixture from the review still needs its fix, and the branch needs a rebase.

@BBuf
BBuf dismissed their stale review November 15, 2026 09:01

[BBuf Review Agent] Withdrawn: the fix should not depend on a new opt-in env, see the comment on the PR.

@wirybeaver
wirybeaver force-pushed the fix/gdn-fp32-prefix-snapshots branch from 3cf208b to b24266a Compare November 15, 2026 19:32
@wirybeaver
wirybeaver force-pushed the fix/gdn-fp32-prefix-snapshots branch from b24266a to 660fe93 Compare November 15, 2026 20:12
@wirybeaver wirybeaver changed the title [Fix] gdn: Add opt-in FP32 prefix-cache snapshots [Fix] gdn: Preserve FP32 prefix-cache snapshots Nov 15, 2026
@wirybeaver

Copy link
Copy Markdown
Contributor Author

@BBuf Removed the env flag; FP32 snapshots are now selected from the state/activation dtypes when tracking is enabled. Also fixed the CPU tracking fixture and rebased.

Added H200 Drama3 c16 results with prefix caching enabled: mean prefill batch wall time increased 0.85–0.96%, while corpus WER decreased 0.25–0.29 percentage points. Throughput was roughly unchanged. The full setup and per-visit results are in the PR description.

@BBuf

BBuf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @wirybeaver Checked 660fe93: eager and graph prefill now select FP32 snapshots from the dtypes with tracking enabled, and the CPU fixture uses matching QKV/conv dtypes. That addresses my earlier code concerns.

BBuf
BBuf previously approved these changes Nov 16, 2026

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] LGTM.

@BBuf

BBuf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] One thing to fix before this can go in.

  • python/sglang/srt/layers/attention/linear/gdn_backend.py:1632 This also runs on the default Qwen3.5 / Qwen3-Next prefill (Triton GDN with radix cache), not only for Drama3, and it allocates an fp32 snapshot buffer per layer on top of the bf16 one. Could you post Qwen3-Next on H200 with the default radix cache, main vs this PR: prefill throughput from python -m sglang.benchmark.serving at c1 and c128, peak memory, and the KV pool size from the startup log? Please include a batch big enough to take the eager path (more than 128 requests).

@BBuf
BBuf dismissed their stale review November 16, 2026 23:49

[BBuf Review Agent] Withdrawn: there are things to fix first, see the review comments.

@wirybeaver
wirybeaver force-pushed the fix/gdn-fp32-prefix-snapshots branch from 660fe93 to de75223 Compare November 17, 2026 02:02
@wirybeaver

Copy link
Copy Markdown
Contributor Author

@BBuf This was opt-in in the original version. I've restored that scope with --enable-gdn-fp32-snapshots (default off), using a parameter rather than an environment variable. Drama3 enables it explicitly; Qwen3.5 / Qwen3-Next retain the existing snapshot path by default, without the additional FP32 snapshot buffer. The dtype and tracking checks still apply when enabled.

@wirybeaver
wirybeaver force-pushed the fix/gdn-fp32-prefix-snapshots branch from de75223 to 73b65a8 Compare November 17, 2026 02:07
@BBuf

BBuf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @wirybeaver Checked 73b65a8: the flag defaults off, and both eager and graph prefill gate the extra FP32 snapshot buffer on it, with dtype and tracking checks still in place. That addresses the default-path memory/perf concern.

@wirybeaver
wirybeaver requested a review from BBuf November 17, 2026 02:14
@BBuf

BBuf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

2 similar comments
@BBuf

BBuf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@wirybeaver
wirybeaver force-pushed the fix/gdn-fp32-prefix-snapshots branch from 73b65a8 to 22b4013 Compare November 17, 2026 23:44
@BBuf

BBuf commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants