Repository navigation
Conversation
Ying1123
approved these changes
Jan 16, 2024
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Mar 28, 2025
pi314ever
pushed a commit
to pi314ever/sglang
that referenced
this pull request
May 7, 2025
* support hpu fix import error and improve contiguous pa clean up clean up clean up style pre-commit clean up enable warmup add moe layer fix accuracy disable radix cache automatically remove sort clean up fix fix model type refine style style add hpu test add change try directly install vllm-fork clean up unnecessary changes fix awq address comments fix acc address comments fix warmup fix cpu scheduler fix style fix device remove block scales refactor refactor fix fix remove change fix attn mask fix attn bias add to device fix style address comments address comments Update python/sglang/srt/model_executor/forward_batch_info.py Co-authored-by: JieXin Liang <Alcanderian@users.noreply.github.com> Update python/sglang/srt/model_executor/model_runner.py Co-authored-by: JieXin Liang <Alcanderian@users.noreply.github.com> address comments optimize allocator and rope add profile add heap based allocator * fix hardcoded dtype * fix style * fix embedding model * fix style
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
[Checkpoint] Fix the search procedure
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
update demos
5 tasks done
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
May 28, 2025
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Jun 4, 2025
pengxin99
pushed a commit
to pengxin99/sglang
that referenced
this pull request
Jun 19, 2025
5 tasks done
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Jul 8, 2025
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Jul 10, 2025
yichiche
pushed a commit
to yichiche/sglang
that referenced
this pull request
Jul 30, 2025
yichiche
pushed a commit
to yichiche/sglang
that referenced
this pull request
Aug 7, 2025
yichiche
pushed a commit
to yichiche/sglang
that referenced
this pull request
Aug 11, 2025
5 tasks done
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 29, 2026
…project#15) The compressed-tensors pydantic validator rejects num_bits=2 for type=FLOAT (it only supports 4-bit nvfp4 and 8-bit fp8), so the per-target weight_quant on a HIGGS checkpoint can't carry the discriminator directly. Switch _is_higgs_2bit_moe to consult the top-level ``quantization_config.format`` string first (``higgs-dense-2bit-moe`` / ``higgs_dense_2bit_moe``) and fall back to the per-args sniff for forward compatibility. Also re-order the predicate so HIGGS detection runs before the ``weight_quant is None`` early-return — a stricter HIGGS checkpoint may omit the per-target args entirely and rely solely on the format string. Verified on B200 decode pod: HIGGS format string -> True; NVFP4 format string -> False; underscore variant accepted; existing NVFP4 detection unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 29, 2026
…gl-project#15 iter2) iter1 landed the HIGGS-2bit-MoE codec + CompressedTensors scheme as a stepping-stone (commits 9fc19b3, 100bd91): on-disk storage savings + a dequant-and-NVFP4-quantize post-load path that's bit-identical to the W4A4 NVFP4 trtllm path at runtime. The opt-in BF16-runtime fallback (SGLANG_OPT_USE_HIGGS_MOE_2BIT_BF16_RUNTIME) was the path that preserved the HIGGS 8x weight-memory savings, but it called the eager PyTorch codec ``dequantize_higgs_moe_weights`` on every apply step. Two real problems with that: (1) Eager dequant cost at DeepSeek-V3.2-REAP per-rank shapes (E=32, H=7168, I=2048) is *96.5 ms per MoE layer per decode step* on B200 — measured by sweeping the codec's per-call cost against the PYTHONPATH-mounted repo. Across 58 MoE layers that's 5.6 seconds of weight dequant *every decode step*, on top of the actual GEMMs. Per the brief and sibling subagent sgl-project#19's profiling, the expert weight HBM traffic is the binding constraint on the HIGGS path relative to NVFP4 — so the eager codec didn't just leave perf on the table, it actively made the BF16 runtime path unusable. (2) The iter1 ``HiggsMoE2BitConfig`` validator rejected non-power-of-2 ``in_dim`` outright. DeepSeek-V3.2's hidden_size is 7168 = 7 * 2^10, not a power of two, so the iter1 path could never load the actual target model. The FWHT only requires a *block size* to be a power of two; the iter1 codec already supported sub-row blocking but the post_init guard was over-restrictive. This commit lands the iter2 perf work: * ``python/sglang/jit_kernel/higgs_moe_2bit_dequant.py``: new Triton kernel ``_higgs_moe_dequant_unpack_kernel`` that reads HIGGS-packed weights ([E, out_dim, slot_bytes] uint8), decodes each 4-bit codebook index via the EDEN2-16 lookup, multiplies by the per-row / per-block FP16 scale, and emits a contiguous [E, out_dim, in_dim] BF16 tile. Output layout matches the eager codec exactly: per byte j the four positions 4j..4j+3 hold (ce_x, ce_y, co_x, co_y) for the even and odd pairs in that byte. Vectorized BF16 store via ``tl.join`` cascade (parity outer, axis inner) so each CTA emits a single [ROWS, BLOCK_K] write instead of four strided writes — that single change took the unpack kernel from 2.55 ms (w13 prod shape) to 0.65 ms (~3.9x). Heuristics tuned by sweep: ROWS_PER_PROGRAM=4, num_warps=4 is best across both w13 ([E,2I,H], block=1024) and w2 ([E,H,I], block=2048) at DSv3.2 per-rank scale. The end-to-end fast dequant ``higgs_moe_2bit_dequant_fast`` composes the Triton unpack with the existing fast-Hadamard CUDA kernel (``hadamard_transform`` from ``python/sglang/jit_kernel/hadamard.py``, Tri Dao's). The per-block inverse FWHT runs at orthonormal scale ``1/sqrt(block_size)`` on a ``[E*out_dim*nb, block_size]`` reshape; correctness round-trips to cos sim 0.9448 vs the original BF16 weight (bit-identical to the eager codec's 0.9448). * ``higgs_moe_2bit_weights.py``: relax the in_dim power-of-2 check. Only the FWHT block size must be a power of two and divide in_dim. Power-of-2 in_dim's still default to single-block (unchanged behaviour); non-power-of-2 in_dim's require an explicit block_size. Verified: in_dim=2048 default block=2048 nb=1 still works; in_dim=7168 block=1024 nb=7 now works; in_dim=7168 default block still raises ("block_size must be a power of two"). * ``compressed_tensors_higgs_dense_2bit_moe.py``: route ``_dequant_to_bf16`` through the fast kernel on CUDA tensors; CPU fallback still uses the eager codec for parity. New ``_resolve_block_size`` picks the largest power-of-2 divisor of ``in_dim`` when ``SGLANG_OPT_HIGGS_MOE_2BIT_BLOCK_SIZE`` is unset, so DSv3.2-REAP shapes load cleanly without env knobs. Composition rule honored: this is the dispatch — there's no config-branched alternate. The BF16 runtime fallback already calls ``_dequant_to_bf16``, so updating that helper is enough to wire the fast path through ``_apply_bf16_runtime``. * ``test/srt/test_higgs_moe_2bit_dequant_fast.py``: unit + scheme + bench coverage. Asserts ``higgs_moe_2bit_dequant_fast`` matches the eager ``dequantize_higgs_moe_weights`` within BF16 round-off on small toy, single-block (in_dim=2048), and sub-row-blocked (in_dim=7168 block=1024) shapes; asserts the scheme helper routes through the fast kernel; logs the per-call ms at DSv3.2 per-rank shapes for benchmark visibility. Measured kernel-level speedup (B200, DSv3.2 per-rank, best of 10): shape (E, OUT, IN, block) eager iter2 speedup ------------------------------------- -------- ------- ------- w13 (32, 4096, 7168, block=1024) 62.59 ms 2.21 ms 28.3x w2 (32, 7168, 2048, block=2048) 33.92 ms 1.02 ms 33.2x total per MoE layer 96.52 ms 3.23 ms 29.9x x58 MoE layers per decode step 5598 ms 187 ms Unpack-only (no FWHT, for arithmetic): w13 0.65 ms, w2 0.30 ms — at 2.8 / 3.4 TiB/s = 35% / 42% of B200 HBM peak. The remaining 1.57 ms (w13) / 0.72 ms (w2) is the existing fast-Hadamard CUDA kernel which itself runs at ~2.2 TiB/s on these shapes. End-to-end TPOT delta vs the active FP8 trtllm-DSA deploy (4091 agg tok/s, 25.9 ms TPOT): * The default iter1 dispatch (post-load dequant -> NVFP4-quantize -> trtllm fp4 MoE) is unchanged by this commit. Runtime is still bit-identical to the W4A4 NVFP4 path, so default-deploy TPOT delta is zero. * The BF16 runtime path (opt-in via the existing env var) is what changed. iter1 cost at production shape: ~5.6 sec / decode step for dequant alone -> totally unusable. iter2 cost: 187 ms / decode step for dequant. That's still ~7-8x the FP8 baseline TPOT, so the BF16 runtime path is not yet production-deploy-ready — but it's now within striking distance of an A/B comparison, which iter1 wasn't. The remaining gap is the actual BF16 expert GEMM cost (``trtllm_bf16_moe``), which has different shape constraints than the NVFP4 kernel. * The actual e2e perf win arrives in iter3, which the brief and sibling subagent sgl-project#19's profiling both flag as a CuTe-fused HIGGS-dequant + BMM kernel. That kernel would consume HIGGS-packed weights inline (skip the BF16 materialization), reading ~2.25x less weight memory per call than the NVFP4 path. With weight HBM as the binding constraint at decode, that's worth a measurable TPOT reduction relative to FP8 trtllm-DSA. iter2 unblocks iter3 by delivering a tested codec primitive that the CuTe kernel can use as an oracle. What's left for iter3 (documented in scheme docstring + NOTE markers at the call site): (a) Fuse the FWHT into the same Triton (or CuTe) kernel as the unpack+scale step. That eliminates the 235 MiB BF16 round-trip between unpack and hadamard, projected ~2x further on dequant cost (~1.5 ms / layer). Triton's lack of bit-shuffle primitives for FWHT_1024 makes a CuTe-DSL implementation the natural fit (quack has a reference at width up to 32K). (b) The actual perf-win kernel: HIGGS-2bit -> BF16 dequant fused with the per-expert BMM, mirroring W4A16 CuTe kernels. Inputs: HIGGS-packed [E, out_dim, slot] uint8, activation [tokens, hidden] bf16. Output: per-expert BMM result. Avoids materializing the BF16 working tile entirely — weight HBM read drops 2.25x vs NVFP4. The CuTe-DSL reference set (cutest, quack, cutlass cute, blackwell tutorial, CuTe DSL docs, CuTe paper) is mirrored at ``/home/spencergarnets/higgs_cute_rewrite_plan/references``. Verification on the decode pod (PYTHONPATH-mounted override; pod is under sgl-project#14 debug so we don't restart it): $ pytest -xvs test/srt/test_higgs_moe_2bit_dequant_fast.py 6 passed, 2 warnings in 10.44s TestHiggsMoeDequantFast::test_dsv32_w13_shape_sub_row_blocking PASS TestHiggsMoeDequantFast::test_dsv32_w2_shape PASS TestHiggsMoeDequantFast::test_medium_pow2_single_block PASS TestHiggsMoeDequantFast::test_small_pow2 PASS TestHiggsMoeDequantFastViaScheme:: test_scheme_helper_routes_through_fast_kernel PASS TestHiggsMoeDequantFastBench::test_bench_dsv32_per_rank PASS [higgs_moe_dequant_fast] per layer (best of 10): 3.229 ms (x58 layers: 187.3 ms) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
pushed a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 29, 2026
…+ cached slice Adds SGLang JIT NVFP4 quantize with linear (non-swizzled) SF layout - drop-in for flashinfer.fp4_quantize used by trtllm_fp4_block_scale_moe on the DeepSeek-V3.2-REAP NVFP4 MoE deploy. Env-var SGLANG_USE_SGL_NVFP4_QUANT=1 enables; default off. Caches w13_input_scale_quant[:1] slice on the layer to skip a re-slice each decode step. Iter2 will fuse this kernel with upstream RMSNorm path when allreduce-fusion is not engaged.
SpencerGarnets
pushed a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…r NVFP4 quantize) kernel
Adds a single-CTA-per-row kernel that collapses the upstream
`fused_add_rmsnorm` BF16 write+read of hidden_states with the
downstream `scaled_fp4_quant_linear` BF16 read. On B200 at production
hidden_size=7168 this saves the ~1.8 MB/layer of HBM traffic plus one
launch-overhead amortization per layer on the non-allreduce-fusion path
(small/medium batch decode, or when the closed flashinfer
allreduce-fusion is disabled).
Public API: `sglang.jit_kernel.nvfp4.fused_rmsnorm_scaled_fp4_quant_linear`
(also re-exported as `sglang.jit_kernel.norm.fused_add_rmsnorm_to_nvfp4_linear`
for callers already importing the fused_add_rmsnorm pair).
Correctness: 100% bit-exact residual + FP4 + SF byte match against the
unfused pair at m=1,8,16,32,64,128,256 for hidden=7168
(test/srt/test_nvfp4_fused_rmsnorm_quant.py).
Bench at hidden=7168, 58 layers/decode-step on B200 (CUDA-graph
captured, single GPU):
m= 1: unfused=4.38us fused=4.13us fused+pdl=3.55us saved=0.25us/layer step=14.4us
m= 8: unfused=4.43us fused=4.14us fused+pdl=3.57us saved=0.29us/layer step=16.8us
m= 16: unfused=4.43us fused=4.19us fused+pdl=3.78us saved=0.24us/layer step=13.9us
m= 32: unfused=4.47us fused=4.19us fused+pdl=3.82us saved=0.27us/layer step=15.7us
m= 64: unfused=4.54us fused=4.23us fused+pdl=3.81us saved=0.31us/layer step=17.7us
m=128: unfused=4.63us fused=4.31us fused+pdl=3.91us saved=0.33us/layer step=19.0us
m=256: unfused=5.63us fused=6.49us (regression; compute-bound)
Iter2 secondary vector (PDL): `enable_pdl=True` plumbs through
`LaunchKernel.enable_pdl`. Bench shows additional 0.32-0.55 us/layer
saved on top of the fuse — kernel dispatches in the shadow of the
preceding global store (allreduce store or attention output write).
PDL gain is largest at low m where launch overhead dominates.
Iter2 stretch (NOT in this commit, documented in
nvfp4_fused_rmsnorm_quant_kernels.cuh tail): vendor a copy of
trtllm_fused_moe_kernel_launcher.cu, relax the MxE2m1 activation check,
rebuild via SGLang's JIT pipeline. That would unlock the 58
bmm_Bfloat16_E2m1E2m1_* in-CUBIN variants consuming BF16 activations
directly and eliminate the entire fp4_quantize step including this
fused kernel.
Foundation fixes (load-bearing):
* `#pragma once` on nvfp4_quant.cuh, nvfp4_quant_kernels.cuh, and
nvfp4_quant_linear_kernels.cuh. Without these the linear quant
iter1 module fails to compile when the fused kernel pulls in
`cvt_warp_fp16_to_fp4` via `#include nvfp4_quant_kernels.cuh`
(double-definition errors). Confirmed iter1 `_jit_nvfp4_quant_linear_module()`
was broken at HEAD pre-commit; now compiles clean.
m=256 regression: at that batch size the fused kernel's register
pressure (one CTA per row with deeper per-thread work) costs more than
the saved HBM traffic. Production target QPS keeps decode batch
mostly in m=1..128, where the fused path wins consistently. Iter3
vectors below address m>=256.
Deploy-delta projection (DP=TP=8, 58 MoE layers, B200 8 TB/s HBM):
- Per-step TPOT win at m=1..128: 14-19 us (~0.3-0.4% at 5ms TPOT
target, ~2.3-3.8% at 500us TPOT target)
- With PDL: 18-49 us / step
- Iter1 deploy delta was ~0 (foundation); this is the real perf win.
Iter3 vectors:
- Cooperative SMEM staging for m>=256 to amortize register pressure.
- Multi-stream pipelining (route the residual write to a separate
stream from the FP4 store path).
- Stretch goal (launcher patch): vendor + JIT-rebuild
trtllm_fused_moe_kernel_launcher.cu, drop the MxE2m1 act check,
invoke bmm_Bfloat16_E2m1E2m1_* directly with BF16 activations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
pushed a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
… apply_weights consumer + RMSNorm helper) Secondary vector of iter3 — completes the kernel-side iter3 wide+SMEM landed via 2f0dcfd (mistagged inside a DSA-indexer commit body but functionally correct; the iter3 wide variant is at nvfp4_fused_rmsnorm_quant_kernels.cuh:cvt_fused_rmsnorm_to_fp4_linear_wide). Adds the activation surface so a deploy can flip the fused kernel on without further C++ changes: * environ.py — registers SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM (EnvBool, default False). Comment block documents the three preconditions: compressed_tensors_w4a4_nvfp4_moe + flashinfer_trtllm runner, should_allreduce_fusion=False, non-cast_x_before_out_mul RMSNorm. * compressed_tensors_w4a4_nvfp4_moe.apply_weights — when env is set AND the incoming hidden_states carries `_sglang_pre_quantized_fp4 = (fp4, sf)`, consume the stash and skip the redundant fp4_quantize call. Defensive del-after-consume guards against moe_cp re-runs reusing the stale buffer. * RMSNorm.forward_with_fused_nvfp4_quant — wrapper that calls fused_rmsnorm_scaled_fp4_quant_linear and returns (fp4, sf, residual). Use it from a hook that knows the global_scale (the MoE layer's cached w13_input_scale_quant_slice). No call site is plumbed inside DeepSeekV2 yet — see Iter4 vectors below for the DP-aware hookup. Why this is staged as activation-only: * The production deploy runs --dp 8 --tp 8 which hits the `context.attn_dp_size != 1` branch of `_gather_hidden_states_and_residual`. In that branch the local layernorm runs at line 995, then dp_gather_partial materializes a bf16 buffer at line 1008 before the MoE sees it. A naive fuse at line 995 would produce LOCAL (fp4, sf) but the gather is sized for bf16, so the trtllm consumer would see the wrong layout. The DP-aware fuse needs either a parallel fp4-aware gather or a post-gather quantize — that is iter4 work. * In the meantime, this commit lets a deploy verify the iter2/iter3 kernel against its own dispatcher by setting the env var, attaching `_sglang_pre_quantized_fp4` from a unit-test fixture, and confirming apply_weights consumes the stash without regression. Iter4 vectors (deferred): * DP-aware fuse in `_gather_hidden_states_and_residual` line 995: either (a) extend dp_gather_partial to gather fp4+sf instead of bf16, or (b) fuse layernorm + scaled_fp4_quant_linear inline AFTER the gather (line 1014, hidden_states already gathered). * Multi-stream pipelining of the residual write — schedule the p_res store on a side stream so the fp4 store can race ahead. * cast_x_before_out_mul correctness sweep — bench/test currently covers only Llama-style; add parameterized test for HF parity. * Stretch: vendor trtllm_fused_moe_kernel_launcher.cu, drop the MxE2m1 activation-dtype check, dispatch bmm_Bfloat16_E2m1E2m1_* directly with BF16 activations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
pushed a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…ches (m=384, 512) Adds m=384 and m=512 to the existing test_nvfp4_fused_rmsnorm_quant.py coverage so the iter3 wide+SMEM variant (auto-selected at m>=kWideMinM=192) gets exercised at multiple batch sizes, not just at the regression-driver m=256. m=256 was already covered by iter2's DECODE_BATCHES; with iter3 it now exercises the wide kernel. m=384 sits on the mid-batch slope where register-pressure was iter2's regression mode. m=512 brings prefill shapes into the same parametrize sweep so the m grid is contiguous across the decode/prefill boundary. Both correctness (test_fused_matches_unfused) and perf (test_perf_vs_unfused) tests inherit the new ALL_BATCHES sweep. No change to the unfused reference path — same fused_add_rmsnorm + scaled_fp4_quant_linear pair used as the baseline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
pushed a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
… dispatch
Hot fix for the wide-variant dispatch helper added in the iter3 wide
SMEM-staged kernel. The launcher uses `std::getenv` and `std::atoi` to
read the SGLANG_NVFP4_FUSED_WIDE_MIN_M tuning override. Neither symbol
is brought in by the pre-existing CUDA-runtime includes, so add
<cstdlib> explicitly. Without this nvcc on some toolchains errors out
("std::getenv was not declared in this scope").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…trized test
iter3 tertiary vector held over. The fused (residual-add + RMSNorm +
linear NVFP4 quantize) kernel takes a `cast_x_before_out_mul` flag that
controls whether the post-norm accumulator is rounded to dtype before
multiplying by the RMSNorm weight (HF parity) or left in fp32 until the
weight multiply (Llama parity).
Iter2 / iter3 only exercised the False (Llama-style) path because the
DeepSeek-V3.2-REAP deploy uses Llama-style RMSNorm. But the kernel
emits a different specialization for cast_x=True (the round trip
through dtype changes which fp4 bucket a value falls into at
~50% of the boundary cases), so the True path needs its own correctness
coverage before another deploy can rely on it.
Parametrize test_fused_matches_unfused on cast_x in {False, True}. The
unfused reference now threads cast_x through to fused_add_rmsnorm so
the comparison kernel and the fused kernel see the same rounding mode.
Existing thresholds (residual max < 0.05, FP4 vals > 95% match, SF
bytes > 95% match) carry forward unchanged -- the cast_x=True path
should pass the same gates.
The if __name__ == "__main__" smoke runner picks up both flag values.
No production code touched; deploy preconditions still require
cast_x_before_out_mul=False on the post-attention RMSNorm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…nt kernel into post-attn layernorm path Closes the iter3 deploy-wire stub. The iter3 commit (3a57f94) landed the activation surface — env var + RMSNorm.forward_with_fused_nvfp4_quant helper + apply_weights consumer hook in compressed_tensors_w4a4_nvfp4_moe — but the RMSNorm helper was NEVER called from the actual forward path, so deploy TPOT delta was 0. This commit plumbs the kernel into the real call site. Target call site: communicator.py CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual, the `not use_layer_norm_before_gather` branch where `hidden_states = layernorm(hidden_states)` is invoked after the DP-attn allgather + dp_scatter. The residual has already been folded into hidden_states at attn_tp_rank==0 (line 997), so the layernorm here is a pure RMSNorm — NO residual-add. New residual-less variant of the fused kernel: `fused_rmsnorm_only_to_fp4_linear(x, weight, gs, eps, *, enable_pdl)` → `(y_bf16, fp4, sf)` The kernel produces BOTH the post-norm BF16 hidden_states (consumed by the gate router, shared_experts, and topk path) AND the FP4 + linear-SF stash (consumed by the NVFP4 MoE expert apply_weights via the iter3 `_sglang_pre_quantized_fp4` hook in compressed_tensors_w4a4_nvfp4_moe.py). Both consumers are downstream of the wired call site, with no intervening allreduce / allgather / scatter that would invalidate the stash — the gathered hidden_states flows directly into DeepseekV2MoE.forward. Wire surface: * RMSNorm.forward_with_fused_nvfp4_quant_no_residual(x, gs, *, enable_pdl) — helper that calls the new kernel; matches the existing forward_with_fused_nvfp4_quant helper (residual variant) signature minus the residual arg. * LayerCommunicator.prepare_mlp(..., next_consumer_fp4_global_scale=) — new optional kwarg; the deepseek_v2 caller passes self.mlp.experts.w13_input_scale_quant_slice when the next layer is an NVFP4 MoE expert. Stored on CommunicateContext.next_consumer_fp4_global_scale as per-call scratch; cleared in a finally block. * communicator.py line 1042 (the `hidden_states = layernorm(hidden_states)` call): when (env flag set) AND (next_consumer_fp4_global_scale set) AND (layernorm has the new helper) AND (dtype is fp16/bf16), the layernorm call is replaced with the fused-quant path. The stash is attached as `y._sglang_pre_quantized_fp4 = (fp4, sf)` and the consumer dels the attr after reading (iter3 commit). Correctness (100% bit-exact vs flashinfer.rmsnorm + scaled_fp4_quant_linear pair): m= 1: y max-err=0.000000 FP4 match=100% SF match=100% m= 8: y max-err=0.000000 FP4 match=100% SF match=100% m= 16: y max-err=0.003906 FP4 match=100% SF match=100% m= 32: y max-err=0.007812 FP4 match=100% SF match=100% m= 64: y max-err=0.007812 FP4 match=100% SF match=100% m=128: y max-err=0.007812 FP4 match=100% SF match=100% m=256: y max-err=0.007812 FP4 match=100% SF match=100% m=512: y max-err=0.015625 FP4 match=100% SF match=100% (y max-err = BF16 ULP differences from rsqrt rounding order; mean-err column was 0.0 at every m.) Bench at hidden=7168, 58 MoE layers/decode-step on B200 (CUDA-graph captured, single GPU): m= 1: unfused=6.20us fused=6.16us fused+pdl=4.12us save/step=1.9us/120.3us m= 8: unfused=6.17us fused=6.16us fused+pdl=4.15us save/step=0.4us/117.0us m= 16: unfused=6.17us fused=4.12us fused+pdl=4.12us save/step=118.8us/118.8us m= 32: unfused=6.17us fused=4.12us fused+pdl=4.12us save/step=118.6us/118.5us m= 64: unfused=6.17us fused=4.14us fused+pdl=4.15us save/step=117.4us/117.1us m=128: unfused=6.17us fused=4.23us fused+pdl=4.22us save/step=112.7us/113.4us m=256: unfused=8.21us fused=6.17us fused+pdl=6.17us save/step=118.4us/118.5us m=512: unfused=10.26us fused=10.26us fused+pdl=10.26us save/step=0.1us/0.3us Eager (no graph capture) at all m: 1.56-1.67x speedup, ~10us saved/call (launch overhead amortization dominates outside CUDA graphs). Production decode batch range per rank with DP=TP=8 is m=[1, batch/8]. For peak decode (batch=128) m=16/rank — the fused+PDL path saves ~2us × 58 layers = ~118us/step on the post_attention_layernorm path alone. At the baseline 25.9 ms TPOT this is a 0.46% improvement; the real win compounds with the iter2 wire (residual variant) on the shared dense path and with the cast_x iter4 secondary that will fuse intermediate quantize calls. m=512 has no graph-capture win (kernel becomes compute-bound at high m with idle threads on the wide pass); for prefill-shape m=512 the caller's env flag is left off via the per-call surface check (SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM is decode-only in deploy). Recurring regression guards (verified): * No torch.mm(..., out_dtype=...) introduced (Dynamo meta_mm incompatibility from 34a927b). * compilation.piecewise_context_manager.get_token_to_kv_pool not touched (the upstream-removed path). * No CPU sync ops inside cuda graph capture (the BF16 stash is a pure device tensor mutation). Files: * python/sglang/jit_kernel/csrc/gemm/nvfp4/nvfp4_fused_rmsnorm_quant_kernels.cuh — new `cvt_fused_rmsnorm_only_to_fp4_linear` template kernel and its `invokeFusedRMSNormOnlyFP4QuantLinear` launcher + `fused_rmsnorm_only_to_fp4_linear_sm100a_sm120a` host entry. * python/sglang/jit_kernel/nvfp4.py — register the new entry in `_jit_nvfp4_fused_rmsnorm_quant_module`, add `_fused_rmsnorm_only_to_fp4_linear_custom_op` torch custom op + Python `fused_rmsnorm_only_to_fp4_linear` wrapper. * python/sglang/srt/layers/layernorm.py — add `RMSNorm.forward_with_fused_nvfp4_quant_no_residual`. * python/sglang/srt/layers/communicator.py — `CommunicateContext.next_consumer_fp4_global_scale` per-call scratch field; `LayerCommunicator.prepare_mlp` kwarg `next_consumer_fp4_global_scale` with finally-cleared scope; line 1042 fused-quant short-circuit with env + helper + dtype guards. * python/sglang/srt/models/deepseek_v2.py — surface the MoE expert input_global_scale to prepare_mlp (the w13_input_scale_quant_slice cached on the layer at load-time, [:1] view). * test/srt/test_nvfp4_fused_rmsnorm_only_quant.py — correctness + bench parametrized over m={1,8,16,32,64,128,256,512}. Iter+1 vector targets (if needed): * Secondary: also wire the residual variant (forward_with_fused_nvfp4_quant) at the L1024 fused_add_rmsnorm path inside the same _gather_hidden_states_and_residual function (the use_layer_norm_before_gather=True path, attn_tp_size==1). That path currently writes BF16 hidden_states for the dp_gather_partial that follows; needs kernel extension to also emit BF16 alongside FP4. Architectural complexity: medium. * Tertiary: extend the kernel cast_x_before_out_mul=True path (Gemma/glm4-style HF parity); currently bench shows the kernel works on both, but no deploy uses cast_x for DSv3.2. * Iter5: investigate flashinfer's bmm_Bfloat16_E2m1E2m1_* CUBIN launcher patch to fully eliminate the fp4_quantize step (deferred multi-day stretch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…ast_x_before_out_mul Mirrors the parametrization from sgl-project#15 iter4 commit 62fff01 (tertiary test on the residual variant) but applied to the iter4 PRIMARY residual-less variant landed in 3fc574b. Production deploys (DSv3.2-REAP) use the Llama-style kCastXBeforeOutMul=false path. Gemma/glm4-style HF-parity uses kCastXBeforeOutMul=true; the template kernel already supports both via its `cast_x_before_out_mul` arg, but the test only covered the false path until now. This commit parametrizes test_fused_no_residual_matches_unfused over both modes, and updates _unfused_reference to compute the HF-parity math in plain torch (flashinfer's rmsnorm only exposes Llama-style). Bit-exact results at all 16 (m, cast_x) combinations: m=1 cast_x=False: y max-err=0.000000 FP4 100% SF 100% m=1 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=8 cast_x=False: y max-err=0.000000 FP4 100% SF 100% m=8 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=16 cast_x=False: y max-err=0.003906 FP4 100% SF 100% m=16 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=32 cast_x=False: y max-err=0.007812 FP4 100% SF 100% m=32 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=64 cast_x=False: y max-err=0.007812 FP4 100% SF 100% m=64 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=128 cast_x=False: y max-err=0.007812 FP4 100% SF 100% m=128 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=256 cast_x=False: y max-err=0.007812 FP4 100% SF 100% m=256 cast_x=True : y max-err=0.000000 FP4 100% SF 100% m=512 cast_x=False: y max-err=0.015625 FP4 100% SF 100% m=512 cast_x=True : y max-err=0.000000 FP4 100% SF 100% Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…6 emit (L1024 wire, honest negative graph result)
Adds a second fused kernel variant for the `use_layer_norm_before_gather=True`
branch of CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual
(communicator.py L1024) — the path that fires under DSv3.2-REAP's
DP=TP=8 + attn_tp_size=1 deploy config.
Unlike iter2's residual variant (which omits the BF16 post-norm output
to save HBM bandwidth), this variant ALSO emits the BF16 hidden_states
because the immediately-following dp_gather_partial requires it for
the cross-DP allgather. The FP4 stash this kernel computes is wasted
in current deploy (local-rows FP4 doesn't survive the gather) and is
preserved as a no-op stash for the iter5 FP4-allgather plumbing.
Kernel: `cvt_fused_rmsnorm_to_fp4_and_bf16_linear` template
launcher: `invokeFusedRMSNormFP4AndBf16QuantLinear`
public entry: `fused_rmsnorm_to_fp4_and_bf16_linear_sm100a_sm120a`
Python wrapper: `fused_rmsnorm_to_fp4_and_bf16_linear(x, residual, w, gs)`
→ `(y_bf16, fp4, sf)` (residual is mutated in-place to `x + residual`)
Helper: `RMSNorm.forward_with_fused_nvfp4_quant_and_bf16_out`
Wire: communicator.py L1024 `use_layer_norm_before_gather` branch,
gated by SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM env + helper availability
+ next_consumer_fp4_global_scale set + dtype check. Same env flag as
the iter4 PRIMARY L1042 wire — both wires fire together under the
same deploy config knob.
Correctness (test/srt/test_nvfp4_fused_rmsnorm_and_bf16.py, 16/16 PASS):
cast_x=False (production / Llama-style) at m in {1,8,16,32,64,128,256,512}:
100% bit-exact FP4 + SF + residual + y (max y-err = bf16 ULP)
cast_x=True (HF-parity / Gemma-style) at all m:
98.9-99.0% FP4 match, 98.5-98.6% SF match — small rounding-order
delta vs the plain-torch reference; production uses cast_x=False.
Bench at hidden=7168, 58 layers/step on B200:
Eager (no graph): 1.05-1.10x speedup, ~1-2us/call (launch amort)
CUDA-graph captured:
m= 1: unfused=4.13us fused=4.13us fused+pdl=4.13us step=0us
m= 8: unfused=4.13us fused=4.14us fused+pdl=4.13us step=0us
m= 16: unfused=4.13us fused=4.13us fused+pdl=4.13us step=0us
m= 32: unfused=4.13us fused=4.13us fused+pdl=4.13us step=0us
m= 64: unfused=6.17us fused=6.17us fused+pdl=6.17us step=0us
m=128: unfused=6.17us fused=6.17us fused+pdl=6.17us step=0us
m=256: unfused=8.22us fused=8.22us fused+pdl=8.22us step=0us
m=512: unfused=11.41us fused=12.32us fused+pdl=12.32us step=-53us (REGRESSION)
HONEST NEGATIVE: at L1024 in CUDA-graph mode the BF16 emit penalty
cancels the launch amortization, so the deploy delta is 0 at decode
batches m<=256 and a slight regression at m=512. The iter5 FP4
allgather is needed to unlock the FP4 stash and recover the win on
this wire.
The wire still lands behind the same env flag so:
1. Validation / eager-mode runs get the 1.05-1.10x save.
2. The architecture is plumbed for iter5 (FP4 allgather) to drop in.
3. The correctness checks codify the kernel as a known-good baseline
for the eventual FP4-allgather plumbing.
iter5 vector to make this wire deliver in graph mode:
Allgather the (fp4, sf) tuple in parallel with the BF16 allgather,
pass through to the MoE expert apply_weights as the post-gather
stash. Requires a new collective op (FP4 byte allgather + SF
allgather) and rework of dp_gather_partial to call it.
Files:
* python/sglang/jit_kernel/csrc/gemm/nvfp4/nvfp4_fused_rmsnorm_quant_kernels.cuh
— new template kernel + launcher + host entry.
* python/sglang/jit_kernel/nvfp4.py
— register entry, add custom op + wrapper.
* python/sglang/srt/layers/layernorm.py
— add RMSNorm.forward_with_fused_nvfp4_quant_and_bf16_out.
* python/sglang/srt/layers/communicator.py
— wire at L1024 with env + helper + dtype guard.
* test/srt/test_nvfp4_fused_rmsnorm_and_bf16.py
— parametrized correctness + perf (cast_x x m).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…communicator.py L1024 Closes the iter4 SECONDARY honest-negative result (commit ba89b7f) by wiring the iter5 dp_gather_partial_fp4 collective (commit c263c7d's dp_attention.py / test_nvfp4_dp_gather_partial_fp4.py — vacuumed into the HIGGS iter7 sibling commit by accident but the diff is the iter5 collective infra) into CommunicateWithAllReduceAndLayerNormFn. _gather_hidden_states_and_residual's use_layer_norm_before_gather=True branch. Wire surface: immediately after `dp_gather_partial(hidden_states, local_hidden_states, forward_batch)` (L1073 in the iter4 SECONDARY commit's numbering), check for the iter4-SECONDARY-emitted local FP4 stash on local_hidden_states. If present, allocate global FP4 + SF buffers sized to `[M_global, n//2]` + `[M_global, (n//16)//4]` (int32 storage; viewed as fp8_e4m3fn for the consumer), call dp_gather_partial_fp4 to populate them, and reattach the stash on the new gathered BF16 tensor for the NVFP4 MoE expert apply_weights consumer (compressed_tensors_w4a4_nvfp4_moe.py L353 hook). Deploy gate: `_ITER5_FP4_ALLGATHER_M_GLOBAL_MAX = 128`. The iter5 grouped collective is +0.7us/layer SAVED at the deploy decode peak (m_global=128 @ DP=TP=8, batch=128) but regresses by ~7-10us/layer past m_global=256. The hard gate matches the deploy decode peak — the NVFP4 MoE wire only fires for m_global <= 128. Past that, the local stash is discarded and the downstream MoE expert falls back to its own fp4_quantize on the gathered BF16 (the iter4 SECONDARY honest- negative behavior). Bench at hidden=7168, B200, NCCL_NVLS_ENABLE=0, dp=2 graph-captured pynccl (the deploy-relevant measurement): m_global=128 (deploy decode peak): iter4 SECONDARY (BF16 emit + post-gather fp4_quantize): 23.82us iter5 PRIMARY (BF16 + FP4 + SF grouped, no consumer quant): 23.44us delta per layer: +0.38us SAVED Cumulative sgl-project#15 deploy projection over 58 MoE layers/step: iter1-3 (sgl-native + fused-RMSNorm + wide-SMEM): foundation iter4 PRIMARY (L1042 residual-less wire): +118us/step iter4 SECONDARY (L1024 residual wire): 0us/step (honest-neg) iter5 PRIMARY (L1024 FP4 allgather): +22us/step ----------------------------------------------------------------- cumulative sgl-project#15 deploy save: +140us/step TPOT projection: 25.9 ms baseline -> 25.76 ms TPOT (0.54%). Cumulative ai-blaise main delta at the production REAP grid stays on the +118us iter4 PRIMARY backbone; iter5 PRIMARY adds the small L1024 incremental win. Honest scope: the iter5 wire is small relative to iter4 PRIMARY (118us vs 22us) because the L1024 path's FP4 stash collective and the saved fp4_quantize at the consumer roughly cancel out — the ~+0.4us/layer win comes purely from pynccl ncclGroupStart/End fusing the two new allgathers into one launch. Per-call the wire adds FP4 (3584 bytes/token) + SF (448 bytes/token) of allgather traffic on top of BF16 (14336 bytes/token), but removes a ~3.6us/layer fp4_quantize from the consumer. Production peak gets the difference; past m_global=128 the bytes overhead exceeds the saved cost and we drop the stash to keep the L1024 wire honest. Correctness: * Backed by the dp_gather_partial_fp4 unit test (in c263c7d), which validated 100% bit-exact FP4 + 100% bit-exact SF vs the BF16-allgather-then-quantize reference at m_local in {1, 8, 16, 32, 64, 128, 256, 512} x dp=2. * The wire-up only re-attaches the stash; no new arithmetic is introduced. Recurring regression guards: * No torch.mm(..., out_dtype=...) introduced. * No torch.cumsum over (dp_size,) added. * No CPU sync inside CUDA graph capture (all ops are device-side or pynccl-captured). * forward_batch.token_to_kv_pool not touched. * compressed_tensors_w4a4_nvfp4_moe.py consumer hook unchanged — the stash attribute name and tuple shape are identical to the iter4 PRIMARY L1042 wire. Files: * python/sglang/srt/layers/communicator.py — import dp_gather_partial_fp4; after the L1073 dp_gather_partial call, run the iter5 FP4 + SF collective and re-attach the global stash when m_global <= 128. Iter6 next vectors: * Extend dp_gather_partial_fp4 to support sum_len padding for the DSA-CP prefill path (currently raises NotImplementedError). Reuses the rank-offset memcpy_triton pattern from _dp_gather_via_all_reduce but with an allgather instead of allreduce (allreduce sums FP4 bytes which corrupts them). * Investigate fusing dp_gather_partial (BF16) + dp_gather_partial_fp4 into a single ncclGroupStart/End spanning all THREE allgathers. Bench projection: another ~+1us/layer at production peak by sharing the per-launch overhead with the BF16 collective. * Investigate flashinfer bmm_Bfloat16_E2m1E2m1_* CUBIN launcher patch to eliminate the iter4 PRIMARY fp4_quantize call entirely (deferred multi-day stretch; see iter4 PRIMARY commit body for target API surface). Honest negatives kept as opt-in checkpoints: * Past m_global=128, the iter5 wire is a clear loss (-9.75us/layer at m_global=256). The hard gate `_ITER5_FP4_ALLGATHER_M_GLOBAL_MAX` makes the wire a strict subset of the iter4 SECONDARY env-flag deploy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…Group fusion plan) Recon notes for the iter6 PRIMARY vector: fuse the BF16 + FP4 + SF allgathers (currently 2 launches: BF16 separate, FP4+SF grouped via iter5 ab38f95) into a single ncclGroupStart/End block (1 launch). Plan landed: new dp_gather_partial_bf16_fp4_fused in dp_attention.py that wraps all 3 in one pynccl_comm group_start/end pair; communicator.py L1024 wire refactored to call it when the iter4 SECONDARY FP4 stash is present and m_global <= 128. Bench will reuse test_nvfp4_dp_gather_partial_fp4.py infrastructure: the _iter5_grouped helper already groups all 3, so iter6 = lifting that into the production wire. The bench delta is the production launch saved. Per iter5 projection: ~+1us/layer at production peak. Honest-negative fallback ready if bench shows <0.3us/layer. IKP not applicable to this vector (host-side NCCL launch fusion, not intra-kernel instruction profiling). Noted in recon for iter6 TERTIARY flashinfer bmm. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
…P4+SF) into one ncclGroupStart/End Iter5 PRIMARY (ab38f95) grouped FP4 + SF allgathers under a single ncclGroupStart/End but left the BF16 hidden_states allgather as a separate launch (via reg_all_gather_into_tensor -> pynccl_comm.all_gather without group wrapping). Captured graph saw TWO NCCL launches per NVFP4 MoE layer. Iter6 PRIMARY collapses to ONE launch. New collective dp_gather_partial_bf16_fp4_fused in dp_attention.py wraps all 3 allgathers in one pynccl_comm.group_start()/group_end() pair, bypassing the reg_all_gather_into_tensor custom op wrapper for the BF16 leg so the three calls can sit between the same group boundary. Wire surface: communicator.py L1024 CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual use_layer_norm_before_gather=True branch. Peek the iter4-SECONDARY local FP4 stash BEFORE the gather; if present and m_global <= 128 (the iter5 break-even gate, unchanged), call dp_gather_partial_bf16_fp4_fused. Otherwise fall back to the original dp_gather_partial BF16-only path (no FP4 stash to gather). Bench at hidden=7168, B200, NCCL_NVLS_ENABLE=0, dp=2 graph-captured pynccl, 2000 iters per variant, 3 trials. iter5 PROD column shows the two-launch pattern (BF16 separate + FP4+SF grouped); iter6 PROD column shows the single-launch pattern (all 3 grouped): m_local | iter5 PROD (us) | iter6 PROD (us) | iter5->iter6 delta --------+-----------------+-----------------+------------------- 1 | 25.07 | 14.76 | +10.31us SAVED 8 | 27.54 | 17.84 | +9.70us SAVED 16 | 29.40 | 19.18 | +10.22us SAVED 32 | 29.18 | 19.88 | +9.30us SAVED 64 | 33.94 | 23.49 | +10.45us SAVED 128 | 42.78 | 41.34 | +1.44us SAVED <- production decode peak 256 | 52.88 | 44.57 | +8.31us SAVED (gate-off: iter6 wire never fires here in production) 512 | 67.56 | 52.99 | +14.57us SAVED (gate-off: iter6 wire never fires here in production) Production decode peak (m_global=128, batch=128 @ DP=TP=8) saves +1.44us per NVFP4 MoE layer. The win is modest at peak because the BF16 leg alone (~27us) dominates the launch overhead at that size; the per-launch fixed cost being amortized is the smaller piece. At smaller decode batches (m_global <= 64) the win is +9-10us/layer because launch overhead is a much larger fraction of total cost. Cumulative sgl-project#15 deploy projection at peak (58 NVFP4 MoE layers/step): iter1-3 (sgl-native + fused-RMSNorm + wide-SMEM): foundation iter4 PRIMARY (L1042 residual-less wire): +118us/step iter4 SECONDARY (L1024 residual wire): 0us/step (honest-neg) iter5 PRIMARY (L1024 FP4 allgather, 2-launch): +22us/step iter6 PRIMARY (L1024 FP4 allgather, 1-launch): +83us/step ------------------------------------------------------------- cumulative sgl-project#15 deploy save at peak: +223us/step TPOT projection: 25.9 ms baseline -> 25.68 ms TPOT (0.86%). At smaller batches the iter6 win climbs to +9-10us/layer = +522 to +580us/step alone, which would push the iter6 SAVED column above iter4 PRIMARY. Cumulative ai-blaise main delta at the production REAP grid still stays on the +118us iter4 PRIMARY backbone with iter5 + iter6 stacking the L1024 incremental wins. Correctness: * test_nvfp4_dp_gather_partial_fp4.py correctness pass: 100% bit-exact FP4 + 100% bit-exact SF preserved across all m_local in {1,8,16,32,64,128,256,512} x dp=2. The fused collective only re-orders the NCCL launch grouping, not the arithmetic. * No new Dynamo or PCG-tracing surface introduced. * The bench shows zero regression at m_global=256/512 (where the iter5 production gate falls back to BF16-only anyway); the iter6 fused path is never invoked above the m_global=128 gate. Recurring regression guards (verified): * No torch.mm(..., out_dtype=...) introduced. * No torch.cumsum over (dp_size,) added. * No is_current_stream_capturing() call added (the only stream interaction is pynccl_comm.change_state which is graph-capture- safe per the iter5 commit). * No CPU sync inside CUDA graph capture (all 3 allgathers + the group_start/end pair are pynccl-captured). * forward_batch.token_to_kv_pool not touched. * No torch.matmul, .fill_(0), .copy_() on (1, hidden_size) BF16 shapes (the B200 SM_100 launch-config bug family). * compressed_tensors_w4a4_nvfp4_moe.py consumer hook unchanged -- the stash attribute name and tuple shape are identical to the iter4 PRIMARY L1042 wire. * fallback path uses the registered custom op wrapper (tp.all_gather_into_tensor) for torchcomms_ncclx + non-pynccl deploys, preserving Dynamo-safe graph tracing on those paths. Files: * python/sglang/srt/layers/dp_attention.py -- new dp_gather_partial_bf16_fp4_fused: pynccl group_start/end block spanning BF16 + FP4 + SF allgathers, with the same attn_tp_size==1 + max_len mode restrictions as the iter5 dp_gather_partial_fp4. Falls back to per-leg custom-op allgathers when pynccl unavailable. * python/sglang/srt/layers/communicator.py -- refactor L1074..L1149: peek stash BEFORE gather; pick fused 3-allgather (iter6) vs serial BF16 (pre-iter6) path. The allocation of _global_fp4 / _global_sf_storage moves above the gather call so the fused collective can target them directly. Drop-stash path consolidated into the else branch. * test/srt/test_nvfp4_dp_gather_partial_fp4.py -- bumped iters to 2000 for tighter noise; added (d) iter5 PROD and (e) iter6 PROD bench variants; relabeled the columns and added the iter5->iter6 + baseline->iter6 delta columns. The previous _iter5_grouped function (which already grouped all 3 in one call) is now exposed as _iter6_all_grouped to match the new production wire. IKP applicability: this iter6 vector is a host-side NCCL launch fusion. Per-instruction kernel profiling (IKP) does not apply -- the wins/losses are visible in the CUDA-graph captured microbench above, which collapses the per-launch overhead into the measured delta. IKP will be the right tool for iter7+ flashinfer bmm_Bfloat16_E2m1E2m1_* CUBIN reconnaissance (intra-kernel SASS attribution for fp4_quantize elimination). Iter7 next vectors: PRIMARY (architectural unlock; multi-day): flashinfer bmm_Bfloat16_E2m1E2m1_* CUBIN launcher patch to eliminate the iter4 PRIMARY fp4_quantize call entirely (~200-300 us/step ceiling per the iter4 PRIMARY commit projection). Start with IKP per-instruction profile of the existing fp4_quantize call inside the bmm path to confirm the latency target and identify which instructions are amenable to in-cubin fusion. Expected target: scaled_fp4_quant_linear (sgl-native) wins from iter1-3 are at ~40us/call (m_global=128); merging into the trtllm-gen bmm would drop that to zero per layer. SECONDARY: extend dp_gather_partial_bf16_fp4_fused (and the iter5 dp_gather_partial_fp4) to support sum_len padding for the DSA-CP prefill path (currently raises NotImplementedError). Reuses the rank-offset memcpy_triton pattern from _dp_gather_via_all_reduce but with an allgather instead of allreduce (allreduce sums FP4 bytes which corrupts them). Lower priority since deploy decode is the bottleneck. TERTIARY: investigate whether the BF16 leg of the fused gather can be replaced by an in-place reduce_scatter + cp_all_gather fast-path. The current fused path issues 3 allgathers but the BF16 leg is the largest single byte payload (14336 B/token vs 3584 B + 448 B for FP4 + SF). If the NVLS path could absorb the BF16 allgather under symmetric memory while the FP4+SF stay on the legacy ring path, we might reclaim another ~1-2us/layer at peak. Bench-only investigation; the production wire would stay on the iter6 fused path. Honest negatives kept as opt-in checkpoints: * At m_global=128 the iter5->iter6 win is +1.44us/layer median, +0.59us min, +2.62us max across 3 trials. The min is at the noise floor (typical pynccl bench noise ~0.5us at this iter count) but the median is above noise. The win is real but not dramatic at production peak. * At m_global < 128 the win is +9-10us/layer. The production deploy decode peak is m_global=128 (batch=128 @ DP=TP=8); the cumulative deploy save number uses the peak figure to stay conservative. Smaller batches benefit more. * The iter6 fused path is strictly within the iter5 m_global<=128 gate. Above the gate the wire falls back to dp_gather_partial BF16-only -- identical to the pre-iter5 behavior -- so the gate composition is conservative. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
… bmm bf16-act launcher patch plan) Iter6 PRIMARY (2940e32) closed out with the iter7 PRIMARY vector flagged: vendor + patch flashinfer's trtllm_fused_moe_kernel_launcher.cu to relax the dtype gate that rejects (Bfloat16 act + E2m1 weight). The upstream launcher accepts (Bfloat16, MxE2m1) but rejects (Bfloat16, E2m1) even though the cubin export ships kernels for BOTH tuples. Eliminating the host-side fp4_quantize call ceilings at ~200-300us/step (per the iter4 PRIMARY commit projection). This commit is the iter7 RECON CHECKPOINT (no runtime change). It writes the catalog + plan in notes/nvfp4_moe_iter7_recon.md to make the 4-stage scaffold reviewable before code lands. Recon findings (verified by direct inspection): 1. Cubin count: **58 Bmm_Bfloat16_E2m1E2m1_*.cubin** SM_100 variants ship in batched_gemm-b3c1646-c111d7c, exactly matching the iter6 commit projection. Tile coverage: - M=128 fixed. - N in {8, 16, 32, 64, 128, 256} - K in {256, 512} - Pipeline depths s in {3, 4, 5, 6, 9} - CGA in {1x1x1, 2x1x1, 1x1x{2,3,4}-splitK} - schedS (serial) and schPd (persistent) variants - All ship bA16_bB16 (NVFP4 E2m1 block-scale layout, sf_vec_size=16) and BF16 epilogue + biasM_bN_rgTma_clmp. Comparison: 397 MxE2m1 variants ship (bA32_bB32, sf_vec_size=32) for the currently-supported (Bfloat16, MxE2m1) path. The 58 E2m1 variants are the iter7 unlock target. 2. Production-shape match for DSv3.2-REAP (m_global=128, H=7168, intermediate=4096, top_k=8, num_experts=256): - GEMM1 (m, 4096): tile t128x128x128, t128x128x256, t128x64x512 all present in 58-set. Hot persistent variant t128x128x256_s6_et128x128_m256x128x64_cga2x1x1 ships. - GEMM2 (m, 7168): same tile family, cga2 cluster + splitK1-4 covers low-batch tails (m_global=1..64). 3. Blocking gate: trtllm_fused_moe_kernel_launcher.cu FP4BlockScaleLauncher::check_moe() at L1597-1616: if (mDtypeAct == btg::Dtype::E2m1) { TVM_FFI_ICHECK(mDtypeWeights == btg::Dtype::E2m1) ... } else if (mDtypeAct == btg::Dtype::Bfloat16 || ...) { TVM_FFI_ICHECK(mDtypeWeights == btg::Dtype::MxE2m1) << "Only MxE2m1 weights are supported by block scale MoE with Bfloat16, E4m3 or MxE4m3 activation"; } The gate REJECTS (Bfloat16 act + E2m1 weight). DSv3.2-REAP ships E2m1 weights with E4M3-bA16/bB16 group scales. The launcher mDtypeWeights branch (E2m1) requires mDtypeAct == E2m1, forcing the SGL deploy to pre-quantize via scaled_fp4_quant_linear before the call. The cubin for (BF16, E2m1) is sitting there, unreachable from the launcher gate. 4. Python-side mirror gate: flashinfer/fused_moe/core.py is_trtllm_moe_supported() at L122: if dtype_weights == DtypeTrtllmGen.E2m1 and dtype_act != DtypeTrtllmGen.E2m1: return False Mirrors the C++ gate. Both must relax for iter7. 5. Downstream path is clean: - trtllm_fused_moe_runner.cu L590: mGemm2(Gemm2::Runner(dtypeAct, dtypeWeights, btg::Dtype::Bfloat16, ...)) accepts both dtypes independently; no signature-level rejection. - sf_vec_size at L1670: auto const sf_vec_size = mDtypeWeights == btg::Dtype::MxE2m1 ? 32 : 16; already branches correctly for the E2m1 layout (16). No runner-side fix needed. Scaffold plan (4 stages, time-budgeted): Stage A: vendor launcher into SGL external_kernels with check_moe() + python gates relaxed. Keep flashinfer upstream untouched. Time: 2-4h. (this iter delivers vendor copy + patch DIFF) Stage B: bind a new TVM-FFI entry trtllm_fp4_block_scale_moe_bf16_act accepting BF16 hidden_states (no hidden_states_scale needed -- cubin SF-gen happens in epilogue). Time: 2-3h. Stage C: SGL wire in compressed_tensors_w4a4_nvfp4_moe.py L343, gated on SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE env. Skip the entire scaled_fp4_quant_linear / fp4_quantize block; pass x BF16 directly. Opt-in checkpoint pattern. Time: 1h. Stage D: standalone microbench test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py validates bit-exact end-to-end output and attributes per-layer us delta. Time: 1h. Stage E: end-to-end deploy bench validates the cumulative +200-300us/step save on the 58-layer DSv3.2-REAP MoE stack. Time: 1h. Total: 7-10h, well past the iter7 3-hour budget. This commit ships the recon catalog only; the next commit in this iter ships Stage A vendored scaffold + a checked-in launcher patch DIFF. Cumulative sgl-project#15 projection at peak (unchanged from iter6 close-out): iter1-3 foundation iter4 PRIMARY: +118us/step iter5 PRIMARY: +22us/step iter6 PRIMARY: +83us/step ------------------------------- cumulative: +223us/step (TPOT 25.9 -> 25.68 ms) If iter7 lands +200us/step on top: cumulative -> +423us/step at peak (TPOT 25.48 ms, 1.62% improvement). At smaller decode batches the win climbs proportionally to the relative fp4_quantize cost in the layer. Honest open questions: * The bmm epilogue may require a per-token scale tensor input even for BF16-act mode. To confirm in Stage A by inspecting KernelParams.h of the bmm export. * Output bit-exactness vs the current scaled_fp4_quant_linear path is the iter8 wire acceptance gate. The cubin in-epilogue SF-gen may differ from the host-side sgl-native quantize by +/-1 ULP at FP4 quantize boundaries; opt-in gate stays on until bench resolves. * Cubin selector indexes by (mDtypeAct, mDtypeWt) tuple; if the existing trtllmGen BatchedGemmInterface only maps (E2m1, E2m1) and (Bfloat16, MxE2m1) entries for FP4 MoE, the map needs extending in Stage A. (Bmm cubin path mirroring suggests it does map, but to be verified.) Recurring regression guards (this commit changes no runtime code): * forward_batch.token_to_kv_pool not touched. * No torch.mm(..., out_dtype=...). * No torch.cumsum. * No is_current_stream_capturing() in PCG-traced paths. * No torch.matmul on (1, hidden_size) BF16 (B200 SM_100 launch bug). * compressed_tensors_w4a4_nvfp4_moe.py untouched (Stage C wire only). Files: notes/nvfp4_moe_iter7_recon.md (189 lines; full catalog + plan) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets
added a commit
to BlaiseSI/optimization-playground
that referenced
this pull request
May 30, 2026
… BF16-act bmm launcher patch + recon Iter7 PRIMARY vector: vendor + patch flashinfer's trtllm_fused_moe_kernel_launcher.cu to relax the gate that rejects (Bfloat16 act + E2m1 weight) and unlock the 58 in-cubin Bmm_Bfloat16_E2m1E2m1_*.cubin SM_100 variants. Recon checkpoint landed in the prior commit (48806a9). This commit ships the **Stage A scaffold**: patch files, env-var declaration, vendored scaffold module, and microbench template. No runtime change yet. The full 4-stage plan in notes/nvfp4_moe_iter7_recon.md targets 7-10 hours total wall; iter7 budget is 3 hours. Stage A lands here; Stages B (vendored build glue), C (SGL wire), D (microbench), E (end-to-end deploy bench) land in iter8+ when the time budget opens. ## Patches ### 0001-relax-bf16-act-e2m1-weight.patch (FP4BlockScaleLauncher) +12 -3 lines on csrc/trtllm_fused_moe_kernel_launcher.cu Relaxes check_moe() to accept: * (E2m1 act, E2m1 weight) — current production path * (E2m1 act, MxE2m1 weight) — newly accepted (was rejected with the SAME error message that already permitted MxE2m1 with BF16; this is the symmetric branch) * (Bfloat16 act, MxE2m1 weight) — current production path * (Bfloat16 act, E2m1 weight) — NEWLY UNLOCKED for DSv3.2-REAP * (E4m3 / MxE4m3 act, MxE2m1 weight) — unchanged Verified clean apply against the pinned flashinfer source via `git apply --check`. ### 0002-python-mirror.patch (is_trtllm_moe_supported) +8 -1 lines on flashinfer/fused_moe/core.py Mirrors the C++ relaxation in the python dispatch guard. Without this, the launcher patch would still be reachable but the python side would return False before reaching dispatch. Verified clean apply. ## Env-var declaration SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE = EnvBool(False) Defaults False. Opt-in only. When True AND the vendored launcher exports the patched symbol (Stage B), the SGL deploy switches to the BF16-act fused path; otherwise falls through to the iter1-3 + iter4 PRIMARY production path. Matches the iter4 SECONDARY / iter5 SECONDARY opt-in checkpoint discipline. ## Scaffold module python/sglang/srt/external_kernels/flashinfer/__init__.py Exports HAS_BF16_ACT_BMM=False until Stage B. Smoke-tested: the import resolves and returns the False placeholder. Future iter8 wire reads this flag at startup to decide gate eligibility. ## Microbench template test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py Skeleton with the reference vs candidate path call sites elided (NotImplementedError) and the bench loop placeholder. The script runs today and prints the iter7 status message; iter8 Stage D fills in the reference + candidate bench bodies once the candidate symbol exists. ## Recon recap (verified findings) Cubin catalog (from prior commit 48806a9): * 58 Bmm_Bfloat16_E2m1E2m1_Fp32_bA16_bB16_*.cubin variants ship in batched_gemm-b3c1646-c111d7c (SM_100, BF16 epilogue, biasM_bN_ rgTma_clmp, dynamic-batch + cga 1x1x1 / 2x1x1 / 1x1x{2,3,4}-splitK). * Comparison: 397 MxE2m1 (bA32_bB32) variants ship for the currently-supported (BF16, MxE2m1) path. The 58 (bA16, bB16) E2m1 variants are the iter7 unlock target. * Tile coverage matches DSv3.2-REAP production shape: M=128, N in {8,16,32,64,128,256}, K in {256,512}, persistent+serial schedulers both present. Blocking gate (confirmed by direct inspection): * trtllm_fused_moe_kernel_launcher.cu L1610-1616: else if (mDtypeAct == Bfloat16 || E4m3 || MxE4m3): TVM_FFI_ICHECK(mDtypeWeights == MxE2m1) REJECTS (BF16, E2m1). Patch 0001 inserts the E2m1-weight branch. * fused_moe/core.py L122 mirrors the same rejection on the python side. Patch 0002 mirrors the relaxation. Downstream path clean (no runner-side change needed): * trtllm_fused_moe_runner.cu Gemm2::Runner ctor at L587-590 accepts (dtypeAct, dtypeWeights) independently. * launcher prepare_moe() L1670 already branches sf_vec_size correctly: sf_vec_size = (mDtypeWeights == MxE2m1) ? 32 : 16; The E2m1 layout (sf_vec_size=16) is the DSv3.2-REAP NVFP4 layout. * Entry function trtllm_fp4_block_scale_moe at L2145+ already determines mDtypeAct from hidden_states.dtype() (BF16 -> Bfloat16) and mDtypeWeights from weight_scale_vec_size (16 -> E2m1). The combo is achievable today AT THE ENTRY but rejected at check_moe(). ## Expected wins (unchanged from iter4 PRIMARY projection) Per-call: scaled_fp4_quant_linear elides ~40us at m_global=128. Per-step: 58 NVFP4 MoE layers * (40us - bmm-epilogue-SF-gen-cost). Net target: 200-300us/step (the bmm in-cubin path pays back some of the saved 40us for fused SF-gen in the epilogue). Cumulative sgl-project#15 deploy projection at peak after iter7 lands: iter1-3 foundation iter4 PRIMARY: +118us/step iter5 PRIMARY: +22us/step iter6 PRIMARY: +83us/step iter7 PRIMARY: +200..+300us/step (projected, scaffold-checkpoint) ------------------------------- cumulative: +423..+523us/step at peak TPOT: 25.9 ms -> 25.38..25.48 ms (1.62..2.01%) At smaller decode batches (m_global=64) the relative fp4_quantize cost climbs, pushing potential wins higher. ## Honest negatives / open questions 1. Stage B requires building the vendored .so via the same TVM-FFI toolchain flashinfer uses. flashinfer.jit.compile_aot is the entry; reproducing its include + flag stack from outside the wheel is the main Stage B risk. If reproducing the toolchain slips, the fallback is symlinking flashinfer's installed sources, applying patches in-place at SGL import time, and rebuilding via flashinfer's own compile pipeline (intrusive but feasible). 2. Bit-exactness vs the existing scaled_fp4_quant_linear path is the iter8 Stage D acceptance gate. The cubin in-epilogue SF-gen may differ from the host-side sgl-native quantize by +/-1 ULP at FP4 quantize boundaries. The opt-in env-var gate stays on until the bench resolves this. 3. The cubin selector behavior under (BF16 act, E2m1 weight) is unverified at runtime — Stage B is the first chance to exercise it. If the trtllmGen BatchedGemmInterface kernel-map only indexes (E2m1, E2m1) and (BF16, MxE2m1) entries for FP4 MoE and does not auto-discover the (BF16, E2m1) entry, the map needs extending (also in patches/ as iter7+1). ## Recurring regression guards (this commit changes no runtime code) * forward_batch.token_to_kv_pool not touched. * No torch.mm(..., out_dtype=...). * No torch.cumsum. * No is_current_stream_capturing() in PCG-traced paths. * No torch.matmul on (1, hidden_size) BF16 (B200 SM_100 launch bug). * compressed_tensors_w4a4_nvfp4_moe.py untouched (Stage C wire only). * environ.py change is purely additive (new var, EnvBool default False, no existing var renamed or removed). ## Files ADD: python/sglang/srt/external_kernels/__init__.py python/sglang/srt/external_kernels/README.md python/sglang/srt/external_kernels/flashinfer/__init__.py python/sglang/srt/external_kernels/flashinfer/csrc/ (empty, Stage B fills) python/sglang/srt/external_kernels/flashinfer/patches/0001-relax-bf16-act-e2m1-weight.patch python/sglang/srt/external_kernels/flashinfer/patches/0002-python-mirror.patch test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py MODIFY: python/sglang/srt/environ.py (+11 lines: SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE) Both patches verified clean against the pinned flashinfer source via `git apply --check` on a stripped-down repo seeded from the installed wheel. ## Iter7 budget honesty Stage A (this commit): ~3 hours wall — DONE, in budget Stage B (vendor build): 2-4 hours — iter8 PRIMARY Stage C (SGL wire): 1 hour — iter8 PRIMARY Stage D (microbench): 1 hour — iter8 PRIMARY Stage E (deploy bench): 1 hour — iter8 SECONDARY No runtime change in iter7. The iter6 PRIMARY (2940e32) NCCL launch fusion remains the production wire; cumulative +223us/step unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 2, 2026
…/forensic Round-15 audit found 16 items. All applied. Most consequential is replacing the round-14-part-2 'assert feasible' with a fatal() forensic-dump halt + new §10 two-fault-class invariant. MAJOR fixes: - #1 joint_decide infeasibility: was assert (round-14 part 2). Now fatal('joint_decide_infeasible', ...) writes structured JSON to <daemon-data>/forensic/joint_decide_infeasible_<ts>.json capturing event, state snapshot, candidates, DP inputs, traceback, then sys.exit(1). User mandate: deployment-bug-class faults expose themselves with full forensic context; load-class faults absorb. The fault classification is now an explicit §10 invariant. - #2 unavailability_cost signature fix: was transfer_time(add_tiers, u, state) — wrong signature (transfer_time is per σ→τ pair). Rewrote as sum over add_tiers of p_hat × P(serve fails) × reload. Caller passes state correctly. - #3 bw_free cold-start: pinned time_since_last_sample_s = +Inf at boot (trips the > LINK_IDLE_SECONDS branch → returns peak_bw_bps, the correct value for an unused link). - #4 multi-rank throughput aggregation: added per-field aggregation rule table to §6 — sum for byte fields, sum for link bandwidth (each rank has own slice), MEAN for throughput EMAs (same logical work observed N×), max for time_since_last_sample_s, identical for static config + n_bytes + subpool keys. - sgl-project#5 cross-rank subpool key consistency: explicit fatal() on disagreement. CLARITY: - sgl-project#6 PLAN.md drift: removed 'PUT updates cache file atomically' (cache was dropped in round-14), removed 'feasibility-fallback' (now fatal), 'samples_in_window' → 'time_since_last_sample_s'. - sgl-project#7 forecast_horizon: explicit branch for recent_event_rate=0 cold- start (returns heartbeat_s). - sgl-project#8 mean_unit_bytes empty-HBM guard: short-circuit returns ∞ to collapse top_k_pressure to K_MIN. - sgl-project#9 peak_bw_bps > 0 / h_max > 0 / prefill_bps > 0 as deployment-bug invariant (§10 Physical inputs); fatal() on violation. - sgl-project#10 APPLY_FAILED unknown batch_id: explicit silent drop (load-class, not deployment-bug); next state-fetch reconverges via always-fresh + idempotent invariants. - sgl-project#11 program state intermediates: explicit '4 states are exhaustive, intermediate states collapsed at dump time' in §5 schema comment. - sgl-project#12 'policy is not learned': softened 'in scope as an extension' → 'neither in scope nor ruled out'; consistent with paper which also dropped the PPO/DQN paragraph. - sgl-project#14 §12 S4 axis count: 7 → 9 (3 HBM + 3 DRAM-cap + 3 DISK-cap axes; the subtractive arithmetic in the original was wrong). - sgl-project#15 §12 scenario composition: explicit note that S1-S5 compose by subpool-key union; S2+S5 = {full, swa, draft}, S3+S5 = {full, mamba, draft}, etc. PLAN.md additions: - Daemon impl item 11: fatal(reason, **context) helper as the shared entry point for every deployment-bug-class halt; enumerates the call sites. Items not surfacing as a doc change: - sgl-project#13 forecast recompute overhead: noted in audit but the spec already says always-fresh covers this; no edit needed. - sgl-project#16 meta-invariant deployment-bug vs load-class: now covered by the new §10 'Two fault classes' invariant (above).
shivajid
referenced
this pull request
Jun 29, 2026
Added error log indicating that FlashInfer allreduce fusion requires specific NVIDIA GPUs.
5 tasks done
1 of 2 tasks
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #5