Repository navigation
Conversation
ModelOptNvFp4EmbeddingMethod.embedding dequantizes on gather with more than a dozen PyTorch kernels per lookup. Replace that with one Triton kernel, embeddings.nvfp4_embedding, registered in the kernel registry with an SM 8.9 floor (Triton enables E4M3 only from SM 8.9). The kernel reproduces the fallback's multiplication order, so the output is bit-identical. The table's eligibility (CUDA, SM 8.9+, packed layout, BF16/FP16 output) is decided once in process_weights_after_loading. Each lookup then checks only that flag and the ids' dtype and device. That keeps the per-call path free of device queries under torch.compile. Every other case keeps the existing PyTorch path. On H100, under CUDA graphs, lookups are 9-22x faster than the previous path across 1-8192 tokens and hidden sizes 160-7168.
|
Required checks failed on 4aaa705; this PR cannot be merged until they pass. PR title (run)
|
Motivation
#34222 made NVFP4 token embeddings servable and noted that "the gain is purely
footprint", with no compute win. This PR adds the compute side.
ModelOptNvFp4EmbeddingMethoddequantizes on gather. Each lookup runs morethan a dozen PyTorch kernels: two row gathers, nibble splitting, an E2M1 table
lookup, a sign select, two scale multiplies, and a cast. Decode-sized lookups
are launch-bound, and every intermediate tensor is a round trip through HBM. On
H100 that costs about 26 µs per lookup under CUDA graphs and about 250 µs
without them.
This PR replaces that sequence with one Triton kernel, at 1–7 µs for 1–256
tokens. The output is bit-identical to the existing path.
Modifications
Kernel:
sglang.kernels.ops.embeddings.nvfp4_embeddingE2M1 row, decodes nibbles (low nibble first), applies the E4M3 group scale and
the FP32 global scale, and writes BF16 or FP16 output.
value * (group_scale * global_scale), so the results are bit-exact ratherthan merely close.
wrap the way
torchindexing does. Any id shape or stride is accepted(scalar, N-D, non-contiguous, empty).
KernelSpec(embeddings.nvfp4_embedding, Tritonbackend) with
CapabilityRequirement.cuda(min_sm=(8, 9)). Triton enablesthe E4M3 scale dtype only from SM 8.9.
Dispatch:
modelopt_quant.pyprocess_weights_after_loadingdecides once whether the loaded table canuse the kernel. It checks: CUDA (not ROCm), SM 8.9+, BF16/FP16 output, a
packed uint8 table, E4M3 group scales, a scalar FP32 global scale, a
consistent scale shape, contiguous tensors on one device. The result is
stored as
layer.use_fused_nvfp4_embedding.table's device. Because the per-call path has no device query or cache,
--enable-torch-compiletraces it withfullgraph=Trueand the Triton kernelappears in the graph (tested).
keeps the existing PyTorch path unchanged.
Tests and benchmark
test/registered/unit/layers/quantization/test_nvfp4_embedding.py(CPU):KernelSpeccapabilities for SM 7.5 through 12.1.test/registered/kernels/ops/embeddings/test_nvfp4_embedding.py(CUDA,base-b-kernel-unit,1-gpu-large), all comparing against the fallback withatol=0, rtol=0:torch.compilewithfullgraph=True;test/registered/kernels/benchmark/embeddings/bench_nvfp4_embedding.py(
base-b-kernel-benchmark) uses the sharedrun_benchmarkhelpers. It assertsbit-exactness before timing.
Accuracy Tests
The kernel is bit-identical to the existing PyTorch path (
atol=0, rtol=0).That path is itself checked against an independent per-element reference in the
CPU unit test. Model outputs therefore cannot change.
Results on an NVIDIA H100 PCIe (SM 9.0, driver 580.126.09, torch 2.13.0+cu130,
Triton 3.7.1):
Diff coverage (
diff-cover, branch coverage):modelopt_quant.pyis 100% andembeddings/__init__.pyis 100%. The only uncovered lines are the@triton.jitbodies, which run on the GPU where Python coverage cannot tracethem.
To confirm the dispatch tests are not vacuous, I injected four bugs, one at a
time. Each one turned at least one test red:
Speed Tests and Profiling
The setup is a 131,072-row table on an H100 PCIe, timed in µs per lookup.
"Previous" is the existing PyTorch path. "BF16" is
F.embeddingon a residentBF16 copy of the same table, which needs 3.6x the memory and is shown for
context.
With CUDA graphs (
run_benchmark):Without CUDA graphs (
run_benchmark_no_cudagraph), the previous path islaunch-bound at about 240–270 µs for up to 256 tokens. The fused kernel takes
4.7–9.4 µs, which is 27–51x faster.
Compared with a resident BF16 table, the fused kernel is faster at 1–8 tokens
(for example, 1.16 vs 1.81 µs). From 32 to 256 tokens it is up to 1.9x slower,
and at 2048×2560 it is 2.8x slower (16.5 vs 5.9 µs). BF16 is a plain row copy,
while this kernel also decodes every element. In exchange, the table takes
0.5625 instead of 2 bytes per element. These are operator-level numbers, not
end-to-end serving claims.
Related
#41624 adds a separate NVFP4 row decode for Qwen4's per-layer embeddings
(
qwen4_exp.py, offloaded tables). The two don't share a code path: this PRcovers
ModelOptNvFp4EmbeddingMethod, which serves the token embedding. Ifuseful, that gather could later call this registered kernel instead of a
private one.
Developed at SwarmOne using SwarmSim-powered
optimization; correctness and performance were then validated on H100 with the
tests and benchmark above.
Checklist
CI States
Latest PR Test (Base): ❌ Run #40221124445
Latest PR Test (Extra): ❌ Run #40221124401
Latest PR Test (AMD ROCm 10): ❌ Run #40221124479