Skip to content

[Kernel] Fused Triton gather + dequant for ModelOpt NVFP4 embeddings - #42097

Open
benboren wants to merge 5 commits into
sgl-project:mainfrom
benboren:fused-modelopt-nvfp4-embedding
Open

benboren wants to merge 5 commits into
sgl-project:mainfrom
benboren:fused-modelopt-nvfp4-embedding

Conversation

@benboren

@benboren benboren commented Oct 1, 2026 •

Copy link
Copy Markdown

Motivation

#34222 made NVFP4 token embeddings servable and noted that "the gain is purely
footprint", with no compute win. This PR adds the compute side.

ModelOptNvFp4EmbeddingMethod dequantizes on gather. Each lookup runs more
than a dozen PyTorch kernels: two row gathers, nibble splitting, an E2M1 table
lookup, a sign select, two scale multiplies, and a cast. Decode-sized lookups
are launch-bound, and every intermediate tensor is a round trip through HBM. On
H100 that costs about 26 µs per lookup under CUDA graphs and about 250 µs
without them.

This PR replaces that sequence with one Triton kernel, at 1–7 µs for 1–256
tokens. The output is bit-identical to the existing path.

Modifications

Kernel: sglang.kernels.ops.embeddings.nvfp4_embedding

  • One program handles one token and up to 1024 columns. It gathers the packed
    E2M1 row, decodes nibbles (low nibble first), applies the E4M3 group scale and
    the FP32 global scale, and writes BF16 or FP16 output.
  • It multiplies in the same order as the fallback,
    value * (group_scale * global_scale), so the results are bit-exact rather
    than merely close.
  • Row offsets are int64, so offsets past 2^31 bytes do not overflow. Negative ids
    wrap the way torch indexing does. Any id shape or stride is accepted
    (scalar, N-D, non-contiguous, empty).
  • Registered as a lazy KernelSpec (embeddings.nvfp4_embedding, Triton
    backend) with CapabilityRequirement.cuda(min_sm=(8, 9)). Triton enables
    the E4M3 scale dtype only from SM 8.9.

Dispatch: modelopt_quant.py

  • process_weights_after_loading decides once whether the loaded table can
    use the kernel. It checks: CUDA (not ROCm), SM 8.9+, BF16/FP16 output, a
    packed uint8 table, E4M3 group scales, a scalar FP32 global scale, a
    consistent scale shape, contiguous tensors on one device. The result is
    stored as layer.use_fused_nvfp4_embedding.
  • Each lookup checks only that flag and that the ids are int32/int64 on the
    table's device. Because the per-call path has no device query or cache,
    --enable-torch-compile traces it with fullgraph=True and the Triton kernel
    appears in the graph (tested).
  • Everything else (CPU, ROCm, pre-Ada GPUs, FP8 output, unexpected layouts)
    keeps the existing PyTorch path unchanged.

Tests and benchmark

  • test/registered/unit/layers/quantization/test_nvfp4_embedding.py (CPU):
    • Dispatch tests for every fallback reason.
    • Tests that the load-time decision is wired into the layer.
    • A test that the dispatch's SM floor agrees with the registered
      KernelSpec capabilities for SM 7.5 through 12.1.
  • test/registered/kernels/ops/embeddings/test_nvfp4_embedding.py (CUDA,
    base-b-kernel-unit, 1-gpu-large), all comparing against the fallback with
    atol=0, rtol=0:
    • hidden sizes 32, 160, and 2560;
    • BF16 and FP16 output;
    • int32 and int64 ids;
    • every id layout listed above;
    • E4M3 extreme scales (smallest subnormal, 448);
    • torch.compile with fullgraph=True;
    • CUDA-graph replay with updated ids.
  • test/registered/kernels/benchmark/embeddings/bench_nvfp4_embedding.py
    (base-b-kernel-benchmark) uses the shared run_benchmark helpers. It asserts
    bit-exactness before timing.

Accuracy Tests

The kernel is bit-identical to the existing PyTorch path (atol=0, rtol=0).
That path is itself checked against an independent per-element reference in the
CPU unit test. Model outputs therefore cannot change.

Results on an NVIDIA H100 PCIe (SM 9.0, driver 580.126.09, torch 2.13.0+cu130,
Triton 3.7.1):

python test/registered/unit/layers/quantization/test_nvfp4_embedding.py   # 8 passed
python test/registered/unit/kernels/test_kernels_namespace.py             # 59 passed
python test/registered/unit/layers/quantization/test_modelopt_nvfp4.py    # 7 passed
python test/registered/kernels/ops/embeddings/test_nvfp4_embedding.py     # 20 passed

Diff coverage (diff-cover, branch coverage): modelopt_quant.py is 100% and
embeddings/__init__.py is 100%. The only uncovered lines are the
@triton.jit bodies, which run on the GPU where Python coverage cannot trace
them.

To confirm the dispatch tests are not vacuous, I injected four bugs, one at a
time. Each one turned at least one test red:

  • lowering the SM floor to 8.0;
  • not recording the load-time decision;
  • skipping the ids-device check;
  • dropping the group-divisibility check.

Speed Tests and Profiling

python test/registered/kernels/benchmark/embeddings/bench_nvfp4_embedding.py

The setup is a 131,072-row table on an H100 PCIe, timed in µs per lookup.
"Previous" is the existing PyTorch path. "BF16" is F.embedding on a resident
BF16 copy of the same table, which needs 3.6x the memory and is shown for
context.

With CUDA graphs (run_benchmark):

tokens hidden fused previous speedup resident BF16
1 160 1.16 25.70 22.1x 1.81
1 7168 1.65 34.19 20.7x 2.00
32 2560 1.80 33.53 18.6x 1.64
256 160 1.59 34.63 21.8x 1.72
256 7168 5.97 68.49 11.5x 3.08
2048 2560 16.46 204.70 12.4x 5.88
8192 160 6.68 61.14 9.2x 6.53
8192 7168 149.68 2427.41 16.2x 131.81

Without CUDA graphs (run_benchmark_no_cudagraph), the previous path is
launch-bound at about 240–270 µs for up to 256 tokens. The fused kernel takes
4.7–9.4 µs, which is 27–51x faster.

Compared with a resident BF16 table, the fused kernel is faster at 1–8 tokens
(for example, 1.16 vs 1.81 µs). From 32 to 256 tokens it is up to 1.9x slower,
and at 2048×2560 it is 2.8x slower (16.5 vs 5.9 µs). BF16 is a plain row copy,
while this kernel also decodes every element. In exchange, the table takes
0.5625 instead of 2 bytes per element. These are operator-level numbers, not
end-to-end serving claims.

Related

#41624 adds a separate NVFP4 row decode for Qwen4's per-layer embeddings
(qwen4_exp.py, offloaded tables). The two don't share a code path: this PR
covers ModelOptNvFp4EmbeddingMethod, which serves the token embedding. If
useful, that gather could later call this registered kernel instead of a
private one.

Developed at SwarmOne using SwarmSim-powered
optimization; correctness and performance were then validated on H100 with the
tests and benchmark above.

Checklist


CI States

Latest PR Test (Base): ❌ Run #40221124445
Latest PR Test (Extra): ❌ Run #40221124401
Latest PR Test (AMD ROCm 10): ❌ Run #40221124479

ModelOptNvFp4EmbeddingMethod.embedding dequantizes on gather with more than a
dozen PyTorch kernels per lookup. Replace that with one Triton kernel,
embeddings.nvfp4_embedding, registered in the kernel registry with an
SM 8.9 floor (Triton enables E4M3 only from SM 8.9).

The kernel reproduces the fallback's multiplication order, so the output is
bit-identical. The table's eligibility (CUDA, SM 8.9+, packed layout,
BF16/FP16 output) is decided once in process_weights_after_loading. Each
lookup then checks only that flag and the ids' dtype and device. That keeps
the per-call path free of device queries under torch.compile. Every other
case keeps the existing PyTorch path.

On H100, under CUDA graphs, lookups are 9-22x faster than the previous path
across 1-8192 tokens and hidden sizes 160-7168.
@github-actions

Copy link
Copy Markdown
Contributor

Required checks failed on 4aaa705; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 jit-kernel quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant