[Tracking] Qwen3.5-397B (G)B200 Functional Support and Optimizations

[nvpohanh]

Qwen3.5-397B (G)B200 Functional Support and Optimizations

Tracking issue for Qwen3.5-397B support in SGLang on (G)B200.

---

Progress Tracker

no-MTP+Agg

Category	Precision	Item	Status
Functional	FP8	DEP (high-throughput) works functionally	✅ DONE
Functional	FP8	DEP (high-throughput) has good accuracy	✅ DONE
Functional	FP8	TP (low-latency) works functionally	✅ DONE
Functional	FP8	TP (low-latency) has good accuracy	✅ DONE
Functional	NVFP4	DEP (high-throughput) works functionally	✅ DONE
Functional	NVFP4	DEP (high-throughput) has good accuracy	✅ DONE
Functional	NVFP4	TP (low-latency) works functionally	✅ DONE
Functional	NVFP4	TP (low-latency) has good accuracy	✅ DONE
Baseline Perf	FP8	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	FP8	TP (low-latency) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	TP (low-latency) uses correct backends/kernels	✅ DONE
Cookbook	(all)	Update SGLang cookbook	🔄 IN PROGRESS
Perf Analysis	FP8	Round 1 perf analysis	🔄 IN PROGRESS
Perf Analysis	NVFP4	Round 1 perf analysis	🔄 IN PROGRESS
Perf Optimization	FP8	Round 1 perf optimizations	🔄 IN PROGRESS
Perf Optimization	NVFP4	Round 1 perf optimizations	🔄 IN PROGRESS

MTP+Agg

Category	Precision	Item	Status
Functional	FP8	DEP (high-throughput) works functionally	✅ DONE
Functional	FP8	DEP (high-throughput) has good accuracy	✅ DONE
Functional	FP8	TP (low-latency) works functionally	✅ DONE
Functional	FP8	TP (low-latency) has good accuracy	✅ DONE
Functional	NVFP4	DEP (high-throughput) works functionally	✅ DONE
Functional	NVFP4	DEP (high-throughput) has good accuracy	✅ DONE
Functional	NVFP4	TP (low-latency) works functionally	✅ DONE
Functional	NVFP4	TP (low-latency) has good accuracy	✅ DONE
Baseline Perf	FP8	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	FP8	TP (low-latency) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	TP (low-latency) uses correct backends/kernels	✅ DONE
Cookbook	(all)	Update SGLang cookbook	🔄 IN PROGRESS
Perf Analysis	FP8	Round 1 perf analysis	🔄 IN PROGRESS
Perf Analysis	NVFP4	Round 1 perf analysis	🔄 IN PROGRESS
Perf Optimization	FP8	Round 1 perf optimizations	🔄 IN PROGRESS
Perf Optimization	NVFP4	Round 1 perf optimizations	🔄 IN PROGRESS

no-MTP+Disagg

Category	Precision	Item	Status
Functional	FP8	DEP (high-throughput) works functionally	✅ DONE
Functional	FP8	DEP (high-throughput) has good accuracy	✅ DONE
Functional	FP8	TP (low-latency) works functionally	✅ DONE
Functional	FP8	TP (low-latency) has good accuracy	✅ DONE
Functional	NVFP4	DEP (high-throughput) works functionally	✅ DONE
Functional	NVFP4	DEP (high-throughput) has good accuracy	✅ DONE
Functional	NVFP4	TP (low-latency) works functionally	✅ DONE
Functional	NVFP4	TP (low-latency) has good accuracy	✅ DONE
Baseline Perf	FP8	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	FP8	TP (low-latency) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	TP (low-latency) uses correct backends/kernels	✅ DONE
Cookbook	(all)	Update SGLang cookbook	🔄 IN PROGRESS
Perf Analysis	FP8	Round 1 perf analysis	🔄 IN PROGRESS
Perf Analysis	NVFP4	Round 1 perf analysis	🔄 IN PROGRESS
Perf Optimization	FP8	Round 1 perf optimizations	🔄 IN PROGRESS
Perf Optimization	NVFP4	Round 1 perf optimizations	🔄 IN PROGRESS

MTP+Disagg

Category	Precision	Item	Status
Functional	FP8	DEP (high-throughput) works functionally	✅ DONE
Functional	FP8	DEP (high-throughput) has good accuracy	✅ DONE
Functional	FP8	TP (low-latency) works functionally	✅ DONE
Functional	FP8	TP (low-latency) has good accuracy	✅ DONE
Functional	NVFP4	DEP (high-throughput) works functionally	✅ DONE
Functional	NVFP4	DEP (high-throughput) has good accuracy	✅ DONE
Functional	NVFP4	TP (low-latency) works functionally	✅ DONE
Functional	NVFP4	TP (low-latency) has good accuracy	✅ DONE
Baseline Perf	FP8	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	FP8	TP (low-latency) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	DEP (high-throughput) uses correct backends/kernels	✅ DONE
Baseline Perf	NVFP4	TP (low-latency) uses correct backends/kernels	✅ DONE
Cookbook	(all)	Update SGLang cookbook	🔄 IN PROGRESS
Perf Analysis	FP8	Round 1 perf analysis	🔄 IN PROGRESS
Perf Analysis	NVFP4	Round 1 perf analysis	🔄 IN PROGRESS
Perf Optimization	FP8	Round 1 perf optimizations	🔄 IN PROGRESS
Perf Optimization	NVFP4	Round 1 perf optimizations	🔄 IN PROGRESS

---

Weekly Progress

2026-05-08

• B200/B300 Agg FP8/FP4 performance configs were updated; the latest B200 FP4 Agg 8k/1k MTP chart shows a substantial improvement over the prior run.
• Pending optimizations remain in review: [NVIDIA] [GDN] Enable FlashInfer MTP verify on SM100+ (Blackwell) #23273 for the GDN MTP kernel update, Ameyn/wide vec t1 flashinfer-ai/flashinfer#3147 for the GDN decode kernel, and [NVIDIA] [GDN] Add FlashInfer prefill support for SM100+ (Blackwell) #22921 for GDN prefill kernel integration.

2026-04-17

• MoE kernels: cutedsl MoE V2 (Add dedicated FlashInferCuteDslMoE layer for standard-path FP4 MoE #21339) fixed (needs FlashInfer v0.6.8 for autotuner fix Prevent MoE autotuner buffer overflow on large token buckets flashinfer-ai/flashinfer#3025); A2A + trtllm_routed MoE in review ([NVIDIA] Support flashinfer a2a with flashinfer_trtllm_routed moe #22394); A2A + cutedsl v2 MoE in review (feat: Support flashinfer_cutedsl MoE runner with flashinfer alltoall backend #22669, perf debugging in progress)
• SmallEP optimization: reduce_scatterv merged (Replace all-reduce + dp_scatter with reduce_scatterv for DP attention #22642) — fuses all-reduce + dp_scatter into single reduce_scatterv for DP+EP MoE. -13.6% NCCL time, +7.7% e2e throughput on Qwen3.5-397B DP4EP4
• GDN prefill kernel: merged into FlashInfer v0.6.7.post3 ([feat] Add blackwell GDN prefill kernel flashinfer-ai/flashinfer#3001), SGLang integration [NVIDIA] [GDN] Add FlashInfer prefill support for SM100+ (Blackwell) #22921 in review — ~6.8% prefill-only throughput gain on Blackwell (ISL=1000, OSL=1, FP8 DEP4 on GB200: 63,603 → 67,923)
• GDN decode MTP sm100 kernel: SGLang integration in progress, debugging accuracy issue
• Regression fixes: Dynamo hang race condition fixed (fix(deps): pin mio after lockfile drift ai-dynamo/dynamo#8080); Disagg Wide-EP ~2x throughput regression resolved ([Performance Regression] ~2x throughput drop in disaggregated PD mode with Wide-EP (DeepSeek-R1 FP4, GB200) between SGLang v0.5.8 and latest nightly #22095, fix: dp_rank always 0 in non-KV router mode ai-dynamo/dynamo#7984); Small-EP 6x prefill regression resolved ([Performance][Small-EP] 6x prefill throughput regression with Expert Parallelism on DeepSeek R1 between v0.5.9 and dev #22379, fixed in FI v0.6.7.post3)
• Disagg/NIXL: GPU staging buffer merged ([Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890), heterogeneous TP staging in draft ([Disagg][NIXL] Add staging buffer support for heterogeneous TP KV transfer #22536)
• MNNVL AllReduce: [Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787 merged but reverted (Revert "[Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api" #20792) due to GPT-OSS hang — PCG-related, tracking in Hang in oneshotAllreduceFusionKernel During Piecewise CUDA Graph Replay flashinfer-ai/flashinfer#3053

2026-04-08

• NVFP4 now fully functional across all variants — SGLang upgraded to FlashInfer v0.6.7.post2 and released v0.5.10
• All Functional and Baseline Perf tasks marked DONE for all variants and precisions
• Cookbook, Perf Analysis all now IN PROGRESS (including previously blocked Disagg variants)
• GDN optimization: decode with BF16 state + MTP merged (feat(gdn): add BF16 state kernel with MTP support beyond T>4 with intermediate caching. flashinfer-ai/flashinfer#2679); prefill with BF16 state in review ([feat] Add blackwell GDN prefill kernel flashinfer-ai/flashinfer#3001)
• TPx_TPy kv-transfer: Mooncake PR [Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890 merged; NIXL migration in progress ([Disagg][NIXL] Fix heterogeneous TP KV transfer for non-MLA models (same logic with mooncake, Step 1/2 for Qwen3.5 support) #22145 merged, ongoing dev branches)
• MNNVL AllReduce: still reverted due to hang; debugging in progress
• Dynamo perf regression found ([Performance Regression] ~2x throughput drop in disaggregated PD mode with Wide-EP (DeepSeek-R1 FP4, GB200) between SGLang v0.5.8 and latest nightly #22095); fix PR in review (fix: dp_rank always 0 in non-KV router mode ai-dynamo/dynamo#7984)

2026-03-30

• NVFP4 Functional: pending FlashInfer v0.6.7 upgrade (chore: bump flashinfer version to 0.6.7 #21422 in review)
• GDN kernel optimization: in progress ([RFC] Unified GDN Decode/Prefill API flashinfer-ai/flashinfer#2687)
• Disagg perf fix (prefill/decode TP size mismatch): Mooncake PR in review ([Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890). Ongoing discussion about how to do the same for NIXL.
• MNNVL AllReduce: PR ([Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787) was merged but reverted in Revert "[Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api" #20792 due to GPT-OSS hang. Will be debugged and re-submitted.
• cutedsl MoE backend: in progress

2026-03-17

• FP8 Functional: DONE for all variants (MTP+Disagg FP8 now done)
• NVFP4 Functional: all in progress across all variants
• Baseline Perf: all variants now IN PROGRESS
• Disagg perf fix (prefill/decode TP size mismatch): PR in review ([Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890)
• GDN kernel optimization: in progress ([RFC] Unified GDN Decode/Prefill API flashinfer-ai/flashinfer#2687)
• MNNVL AllReduce: PR in review ([Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787)
• cutedsl MoE backend: in progress, resolving FlashInfer issues

2026-03-12

• Bug tracking: all known bugs listed in [Tracking] Qwen3.5 bugs #20069 — prioritizing B200-relevant ones; others handled by community contributors
• Spec V2 (MTP) support PR [Qwen3.5] Enable MTP spec_v2 and add test for nvidia/Qwen3.5-397B-A17B-NVFP4 #19391 has been merged
• Disagg TP size mismatch fix PR [Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890 is in review (only affects non-MLA networks)
• GDN (Gated Delta Net) kernel: working to unify prefill/decode to both use FlashInfer — tracked in [RFC] Unified GDN Decode/Prefill API flashinfer-ai/flashinfer#2687
• MNNVL AllReduce PR [Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787 is in review
• Cookbook update on hold pending discussion on whether to target main branch or latest release (Spec V2 only in main)

2026-03-06 (update)

• Agg functional support (no-MTP+Agg, MTP+Agg): DONE for both FP8 and NVFP4
• MTP enabled via merged PR: [Qwen3.5] Enable MTP spec_v2 and add test for nvidia/Qwen3.5-397B-A17B-NVFP4 #19391
• Disagg functional support: IN PROGRESS (debugging TP size mismatch in Prefill/Decode)
• Agg cookbook, IBDB pipeline, perf analysis: all IN PROGRESS
• Perf optimizations (Agg + Disagg): IN PROGRESS — GDN (gated delta net) kernel optimizations ongoing

2026-03-06

• Created and initialized all tasks

---

Related GitHub Issues

• [NVIDIA] [GDN] Enable FlashInfer MTP verify on SM100+ (Blackwell) #23273 - GDN MTP kernel update (in review)
• Ameyn/wide vec t1 flashinfer-ai/flashinfer#3147 - FlashInfer GDN decode kernel (in review)
• [NVIDIA] [GDN] Add FlashInfer prefill support for SM100+ (Blackwell) #22921 - GDN prefill kernel integration (in review)
• [Tracking] Qwen3.5/Qwen3-Next Optimizations #18590 - Known perf optimizations for Qwen3.5-397B on SGLang
• [Qwen3.5] Enable MTP spec_v2 and add test for nvidia/Qwen3.5-397B-A17B-NVFP4 #19391 - Enable MTP for Qwen3.5-397B (merged)
• [Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890 - Fix Disagg perf issue when prefill/decode TP sizes do not match (merged)
• [Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787 - MNNVL AllReduce support (merged, then reverted in Revert "[Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api" #20792)
• [Tracking] Qwen3.5 bugs #20069 - Qwen3.5-397B bug tracking
• Revert "[Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api" #20792 - Revert MNNVL AllReduce due to GPT-OSS hang
• chore: bump flashinfer version to 0.6.7 #21422 - FlashInfer v0.6.7 upgrade
• [Disagg][NIXL] Fix heterogeneous TP KV transfer for non-MLA models (same logic with mooncake, Step 1/2 for Qwen3.5 support) #22145 - NIXL backend migration (merged)
• [Performance Regression] ~2x throughput drop in disaggregated PD mode with Wide-EP (DeepSeek-R1 FP4, GB200) between SGLang v0.5.8 and latest nightly #22095 - Dynamo perf regression
• [RFC] Unified GDN Decode/Prefill API flashinfer-ai/flashinfer#2687 - FlashInfer GDN prefill kernel support
• feat(gdn): add BF16 state kernel with MTP support beyond T>4 with intermediate caching. flashinfer-ai/flashinfer#2679 - GDN decode with BF16 state + MTP support (merged)
• [feat] Add blackwell GDN prefill kernel flashinfer-ai/flashinfer#3001 - GDN prefill with BF16 state (in review)
• fix: dp_rank always 0 in non-KV router mode ai-dynamo/dynamo#7984 - Dynamo perf regression fix (in review)
• Add dedicated FlashInferCuteDslMoE layer for standard-path FP4 MoE #21339 - cutedsl MoE V2 (fixed)
• [NVIDIA] Support flashinfer a2a with flashinfer_trtllm_routed moe #22394 - A2A + trtllm_routed MoE integration (in review)
• feat: Support flashinfer_cutedsl MoE runner with flashinfer alltoall backend #22669 - A2A + cutedsl v2 MoE for Qwen3.5 (in review)
• Replace all-reduce + dp_scatter with reduce_scatterv for DP attention #22642 - SmallEP reduce_scatterv optimization (merged)
• [NVIDIA] [GDN] Add FlashInfer prefill support for SM100+ (Blackwell) #22921 - GDN prefill kernel SGLang integration (in review)
• [Performance][Small-EP] 6x prefill throughput regression with Expert Parallelism on DeepSeek R1 between v0.5.9 and dev #22379 - Small-EP 6x prefill throughput regression fix (merged)
• [Disagg][NIXL] Add staging buffer support for heterogeneous TP KV transfer #22536 - Heterogeneous TP staging buffer (draft)
• Prevent MoE autotuner buffer overflow on large token buckets flashinfer-ai/flashinfer#3025 - FlashInfer autotuner fix for cutedsl MoE (merged, v0.6.8)
• Hang in oneshotAllreduceFusionKernel During Piecewise CUDA Graph Replay flashinfer-ai/flashinfer#3053 - MNNVL AllReduce PCG hang tracking
• fix(deps): pin mio after lockfile drift ai-dynamo/dynamo#8080 - Dynamo hang race condition fix (merged)