Skip to content

[Kernel] Add Qwen3.8 NVFP4 skinny GEMM for SM120 - #36043

Closed
BBuf wants to merge 2 commits into
sgl-project:mainfrom
BBuf:agent/qwen38-nvfp4-skinny-gemm
Closed

BBuf wants to merge 2 commits into
sgl-project:mainfrom
BBuf:agent/qwen38-nvfp4-skinny-gemm

Conversation

@BBuf

@BBuf BBuf commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Qwen3.8-27B NVFP4 decode on high-SM-count SM120 GPUs spends a significant amount of time in FlashInfer's general block-scaled GEMM. Its decode and DSpark verification shapes have only 1-16 rows, so the general kernel pays unnecessary tile padding and scheduling overhead.

Modifications

  • Add an NVRTC-compiled SM120 NVFP4 skinny GEMM specialized for the Qwen3.8 gate/up, down, and lm_head shapes.
  • Dispatch the kernel behind SGLANG_ENABLE_QWEN38_FP4_SKINNY_GEMM (off by default), with strict shape, dtype, stride, device, SM-count, and architecture guards.
  • Keep the down projection on FlashInfer for M>1 by default. Its Stream-K accumulation order can change speculative acceptance decisions even when the output passes the normal numerical tolerance. Down M=1 remains enabled.
  • Add correctness, CUDA Graph replay, safe-dispatch, and microbenchmark coverage.

Accuracy Tests

Tested on an NVIDIA RTX PRO 6000 Blackwell Server Edition:

TestNvfp4SkinnySm120.test_cuda_graph_replay                              PASS
TestNvfp4SkinnySm120.test_matches_flashinfer                              PASS
TestNvfp4SkinnySm120.test_safe_roles_keep_down_verification_on_flashinfer PASS

The FlashInfer comparison covers all three shapes at M=1, 9, and 16.

DSpark E2E validation used three fixed seeds. The accept lengths are unchanged after applying the safe down-projection routing:

Seed Baseline This PR
1234 3.14 3.14
42 2.86 2.86
7 2.90 2.90

Speed Tests and Profiling

RTX PRO 6000 kernel microbenchmarks against flashinfer.mm_fp4(backend="auto"):

Shape M FlashInfer (us) This PR (us) Speedup
gate/up 1 104.774 98.914 1.0592x
gate/up 9 104.624 99.877 1.0475x
gate/up 16 104.502 100.168 1.0433x
down 1 69.755 56.534 1.2339x
down 9 71.936 57.105 1.2597x
down 16 71.692 57.643 1.2437x
lm_head 1 540.978 522.787 1.0348x
lm_head 9 548.902 526.659 1.0422x
lm_head 16 552.271 530.363 1.0413x

Down M=9/16 numbers show the raw kernel result; those two paths are disabled by the safe default because of the acceptance sensitivity described above.

Qwen3.8-27B end-to-end output throughput:

  • Normal decode: 67.24/67.52/67.87 -> 68.27/68.68/68.53 tok/s (1.41% geometric-mean improvement).
  • DSpark: 135.13/114.77/143.35 -> 137.05/116.36/145.15 tok/s (1.35% geometric-mean improvement), with identical accept lengths.

Checklist


CI States

Latest PR Test (Base): ❌ Run #32962658503
Latest PR Test (Extra): ❌ Run #32962658497
Latest PR Test (AMD ROCm 7.2): ❌ Run #32962658549

@github-actions github-actions Bot added quant LLM Quantization blackwell SM100/SM120 jit-kernel labels Aug 23, 2026
@BBuf
BBuf marked this pull request as ready for review August 23, 2026 13:25
@BBuf BBuf added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) bypass-fastfail labels Aug 23, 2026
@BBuf

BBuf commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci extra

@BBuf
BBuf force-pushed the agent/qwen38-nvfp4-skinny-gemm branch from fe71f76 to 25685b1 Compare August 23, 2026 13:36
@BBuf
BBuf marked this pull request as draft August 23, 2026 15:12
@BBuf
BBuf marked this pull request as ready for review August 23, 2026 16:52
@BBuf

BBuf commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

@BBuf BBuf added the mergeable label Aug 23, 2026
@BBuf
BBuf marked this pull request as draft August 25, 2026 06:57
@BBuf

BBuf commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this draft in favor of the stronger KDA-Pilot implementation from BBuf/KDA-Pilot#195. I will re-port that optimized SM120 NVFP4 GEMM into a dedicated python/sglang/kernels/kda_kernels package with explicit Humanize2 / Kernel Design Agents provenance, then revalidate DSpark end-to-end throughput and acceptance length on RTX PRO 6000 before opening the replacement PR.

@BBuf BBuf closed this Aug 28, 2026
@BBuf
BBuf deleted the agent/qwen38-nvfp4-skinny-gemm branch August 28, 2026 11:21
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
…+ re-plant era CLOSED at b86b7c9 (125/125 markers; grep census ZERO verified at this tree): header STACK epoch 321-2026-10-30 -> 329-2026-11-10 (supersession note kept) + +merge6-era-closeout-2026-11-10 trailer line + entries-count 1-329; tip/HEAD re-pinned (era CODE TIP b86b7c9 NAMED with the full wave chain, superseded values dea2641/e9f0ae1838/065fdcdf33/b7c656f69f/5cde7a5ede kept, docs tip recorded as this wave) + a prominent ** LIVE/BOOT STATE ** line (main checkout detached @ 2d87757 = the ONLY boot-validated state, 9b 16:58:52 warm all-needles-hit per prod-health-2026-11-10.txt; era branch CPU-green but NOT BOOTED -- switching prod = operator-chosen checkout + the entry-326 boot-watch). ENTRY 324: imports+bootfix trunk wave -- sgl-project#48781 require_attn_tp_gather->dp_attention relocation invisible to markers, base_runner deletion-only fix unmasking the 145E/32E/31E collection cascade (1dcc2d6); THEN the operator-live crash: maybe_init_router_registration read tokenizer_ipc_name before init_ipc_channels at OUR fork's entry-201 init order, call moved back to upstream's position, RED(2F)/GREEN(2P) test_tokenizer_manager_init_order.py, family scan clean + report-only carry (10a1c08). ENTRY 325: RE-PLANT ERA TABLE b1 12 / b3 18 / b4 39 / b5 17 / b2 23 / b6 16 = 125/125 per-wave dispositions (compressed from the commit bodies, all shas cited) + the NINE ERA CATCHES with FULL CREDIT: _rope_cache_key 10/11-arg universal get_rope crash (15 spec F were its fingerprints), PLE file-backend env registration gap (AttributeError every boot), COW-catcher doubled-staging-arm cascade bug, stat_loggers DI test restored from its corrupted merge hybrid, w13 refined-grid mis-key, the budget-contract decision (fork accounting RESTORED, evidence 106.9GB/100GB = the entry-317 leak shape; INDEXER collective order verified), verify-gate SUBSUME pick (sgl-project#44047 eats sgl-project#42779, both intents preserved), mm_process_config ambient-widths root-cause (validators now state the full 8-way layout), TLFIX twin re-pinned 282/285. ENTRY 326: BATTERY RE-PIN TABLE -- era pins managers 1307P/0F, spec 406P/0F, server_args 496P/0F, sampling 252P/0F, distributed 213P/0F, mem_cache 2965P/24F (the ACCEPTED device-only 24-node list over 8 files), cargo 966P/0F/1-ignored, five registration/wheel gates rc=0 + the PATH-TO-BOOTABLE statement: CPU-proven vs GPU-window debt -- first-boot needles incl. the sampling.cuh-vs-0.7.2 JIT arbitration (if it JIT-fails, wheel-trio-then-pin per S-FI073, never a lone re-pin), autotune caches SHOULD hit warm (no retune owed on 0.7.2), entry-322 verify-at-boot quartet, and the still-riding 321-era watches (sgl-project#45472/sgl-project#46643 TTFT+starvation, first-delta tool-stream latency). ENTRY 327: WHEELS & VENV RECORD -- pairing matrix (FI 0.7.2 HELD, sgl-kernel 0.4.10/0.4.10 era-consistent with the RETIRED-rule cross-ref, torch/cutlass-dsl MATCH), blake3 1.0.11 INSTALLED with ZERO-DRIFT PROOF (0 battery lines vs ~380; matrix re-read unchanged; negative-shim prefix-sim causality) + no-pip/uv-note (operator lane), the two tree-side wheel reverts with reachability + residual-symbol evidence, check_dockerfile_wheel_pins re-green, FI-0.7.3 adoption contract restated with post-era facts (upstream code now expects 0.7.3 in one reported JIT path; tree HELD 0.7.2; first boot arbitrates), dist-name/module-name note sglang_kernel/sgl_kernel. ENTRY 328: INCIDENT RECORD 2026-11-10 ~16:4x-16:58, honest narrative -- re-plant waves landed into the EDITABLE-IMPORT working tree while :30000/:30001 ran; 9b died ~pre-16:50 (cause UNVERIFIABLE from the user acct, /root logs unreadable; worker-respawn-importing-changed-code is the LEADING MECHANISM, stated AS SUCH); the operator restart hit the init-order crash = the era's flagship find; recovery = merge6-era-backup @ b069e6a + detached checkout 2d87757, 9b re-booted WARM 16:58:52 all needles hit, 9d NEVER dropped (Nov 3, HTTP 200); NEW ERA LAW: working tree = prod import path, integration by branch ref-move ONLY; b2's emergency-freeze WIP 7ccf76a (author 16:55:02) kept in range and completed as C1 -- provenance note. ENTRY 329: BACKFILL AUDIT RECORD -- window 2026-08-10..2026-10-27, universe 2714 = 1830 in-tree / 255 already-documented / 628 NEW REJECTS / 1 NEW-INTERESTING; method one-liners; the qwen4-main-squashed side-branch class discovery (REJECT-MERGED-TO-SIDE-BRANCH x44 -- future sweeps MUST filter by base; orchestrator's "206" NOT FOUND in the artifacts, 44 recorded as verified); sgl-project#36043 NVFP4 skinny GEMM SM120 full paragraph (closed-unmerged, absent-tree-verified, W4A4 M<=16 = our exact decode cell, BENCHMARK-BEFORE-PORT -> S-36043-BENCH filed); artifact paths. EVALUATED: the 671-line DOCS-BLOCK.txt inserted VERBATIM as its own dated block (165 dropped list-commas tidied at insertion, noted once at the block head; nothing else reflowed). DATED CORRECTIONS where the waves superseded 322/323-era operational text: the header PROD-BOOT-FORBIDDEN bullet and entry 323's WHEEL DRIFT + STATE lines get [11-10 ERA-CLOSE CORRECTION] annotations (classes closed, wheel landed, ban now purely the operator gate); marker-zone math re-verified against the wave ledgers (121-at-merge vs 125 recount stands correct; planted == consumed per zone). WATCH LIST: era-close block appended (sgl-project#36043 bench-gated; sgl-project#42790/sgl-project#40916 arrived-via-merge6 noted; sgl-project#43703 revert-trap cross-ref from entry 323; sgl-project#45520 STILL-OPEN re-confirmed at era close -- zero in-window refs, b5 verified the fail-safe layer intact; bootfix's report-only get_tokenizer_worker_class/entrypoints wiring carry). Docs-only: NO .py touched; check_registered_tests rc=0 re-run against this tree. NOTHING PUSHED; the live tree was NOT touched by this wave (ref-move integration follows with HEAD/status equality proofs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 bypass-fastfail jit-kernel mergeable quant LLM Quantization run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant