Skip to content

Rename image_url to image_file - #15

Merged
Ying1123 merged 1 commit into
mainfrom
fix-typo
Jan 16, 2024
Merged

Ying1123 merged 1 commit into
mainfrom
fix-typo

Conversation

@BabyChouSr

Copy link
Copy Markdown
Contributor

Addresses #5

@Ying1123
Ying1123 merged commit 5b27a1d into main Jan 16, 2024
@Ying1123
Ying1123 deleted the fix-typo branch January 16, 2024 23:41
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request Mar 28, 2025
pi314ever pushed a commit to pi314ever/sglang that referenced this pull request May 7, 2025
* support hpu

fix import error and improve contiguous pa

clean up

clean up

clean up

style

pre-commit

clean up

enable warmup

add moe layer

fix accuracy

disable radix cache automatically

remove sort

clean up

fix

fix model type

refine

style

style

add hpu test

add change

try directly install vllm-fork

clean up unnecessary changes

fix awq

address comments

fix acc

address comments

fix warmup

fix

cpu scheduler

fix style

fix device

remove block scales

refactor

refactor

fix

fix

remove change

fix attn mask

fix attn bias

add to device

fix style

address comments

address comments

Update python/sglang/srt/model_executor/forward_batch_info.py

Co-authored-by: JieXin Liang <Alcanderian@users.noreply.github.com>

Update python/sglang/srt/model_executor/model_runner.py

Co-authored-by: JieXin Liang <Alcanderian@users.noreply.github.com>

address comments

optimize allocator and rope

add profile

add heap based allocator

* fix hardcoded dtype

* fix style

* fix embedding model

* fix style
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
[Checkpoint] Fix the search procedure
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request May 28, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request Jun 4, 2025
pengxin99 pushed a commit to pengxin99/sglang that referenced this pull request Jun 19, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request Jul 8, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request Jul 10, 2025
yichiche pushed a commit to yichiche/sglang that referenced this pull request Jul 30, 2025
yichiche pushed a commit to yichiche/sglang that referenced this pull request Aug 7, 2025
yichiche pushed a commit to yichiche/sglang that referenced this pull request Aug 11, 2025
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 29, 2026
…project#15)

The compressed-tensors pydantic validator rejects num_bits=2 for
type=FLOAT (it only supports 4-bit nvfp4 and 8-bit fp8), so the
per-target weight_quant on a HIGGS checkpoint can't carry the
discriminator directly. Switch _is_higgs_2bit_moe to consult the
top-level ``quantization_config.format`` string first
(``higgs-dense-2bit-moe`` / ``higgs_dense_2bit_moe``) and fall back to
the per-args sniff for forward compatibility.

Also re-order the predicate so HIGGS detection runs before the
``weight_quant is None`` early-return — a stricter HIGGS checkpoint
may omit the per-target args entirely and rely solely on the format
string.

Verified on B200 decode pod: HIGGS format string -> True; NVFP4
format string -> False; underscore variant accepted; existing NVFP4
detection unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 29, 2026
…gl-project#15 iter2)

iter1 landed the HIGGS-2bit-MoE codec + CompressedTensors scheme as a
stepping-stone (commits 9fc19b3, 100bd91): on-disk storage savings
+ a dequant-and-NVFP4-quantize post-load path that's bit-identical to
the W4A4 NVFP4 trtllm path at runtime. The opt-in BF16-runtime fallback
(SGLANG_OPT_USE_HIGGS_MOE_2BIT_BF16_RUNTIME) was the path that
preserved the HIGGS 8x weight-memory savings, but it called the eager
PyTorch codec ``dequantize_higgs_moe_weights`` on every apply step.

Two real problems with that:

(1) Eager dequant cost at DeepSeek-V3.2-REAP per-rank shapes (E=32,
    H=7168, I=2048) is *96.5 ms per MoE layer per decode step* on B200
    — measured by sweeping the codec's per-call cost against the
    PYTHONPATH-mounted repo. Across 58 MoE layers that's 5.6 seconds of
    weight dequant *every decode step*, on top of the actual GEMMs.
    Per the brief and sibling subagent sgl-project#19's profiling, the expert
    weight HBM traffic is the binding constraint on the HIGGS path
    relative to NVFP4 — so the eager codec didn't just leave perf on
    the table, it actively made the BF16 runtime path unusable.

(2) The iter1 ``HiggsMoE2BitConfig`` validator rejected non-power-of-2
    ``in_dim`` outright. DeepSeek-V3.2's hidden_size is 7168 = 7 * 2^10,
    not a power of two, so the iter1 path could never load the actual
    target model. The FWHT only requires a *block size* to be a power
    of two; the iter1 codec already supported sub-row blocking but the
    post_init guard was over-restrictive.

This commit lands the iter2 perf work:

* ``python/sglang/jit_kernel/higgs_moe_2bit_dequant.py``: new Triton
  kernel ``_higgs_moe_dequant_unpack_kernel`` that reads HIGGS-packed
  weights ([E, out_dim, slot_bytes] uint8), decodes each 4-bit
  codebook index via the EDEN2-16 lookup, multiplies by the per-row /
  per-block FP16 scale, and emits a contiguous [E, out_dim, in_dim]
  BF16 tile. Output layout matches the eager codec exactly: per byte j
  the four positions 4j..4j+3 hold (ce_x, ce_y, co_x, co_y) for the
  even and odd pairs in that byte. Vectorized BF16 store via
  ``tl.join`` cascade (parity outer, axis inner) so each CTA emits a
  single [ROWS, BLOCK_K] write instead of four strided writes — that
  single change took the unpack kernel from 2.55 ms (w13 prod shape)
  to 0.65 ms (~3.9x). Heuristics tuned by sweep: ROWS_PER_PROGRAM=4,
  num_warps=4 is best across both w13 ([E,2I,H], block=1024) and w2
  ([E,H,I], block=2048) at DSv3.2 per-rank scale.

  The end-to-end fast dequant ``higgs_moe_2bit_dequant_fast``
  composes the Triton unpack with the existing fast-Hadamard CUDA
  kernel (``hadamard_transform`` from
  ``python/sglang/jit_kernel/hadamard.py``, Tri Dao's). The per-block
  inverse FWHT runs at orthonormal scale ``1/sqrt(block_size)`` on a
  ``[E*out_dim*nb, block_size]`` reshape; correctness round-trips to
  cos sim 0.9448 vs the original BF16 weight (bit-identical to the
  eager codec's 0.9448).

* ``higgs_moe_2bit_weights.py``: relax the in_dim power-of-2 check.
  Only the FWHT block size must be a power of two and divide in_dim.
  Power-of-2 in_dim's still default to single-block (unchanged
  behaviour); non-power-of-2 in_dim's require an explicit block_size.
  Verified: in_dim=2048 default block=2048 nb=1 still works;
  in_dim=7168 block=1024 nb=7 now works; in_dim=7168 default block
  still raises ("block_size must be a power of two").

* ``compressed_tensors_higgs_dense_2bit_moe.py``: route
  ``_dequant_to_bf16`` through the fast kernel on CUDA tensors; CPU
  fallback still uses the eager codec for parity. New
  ``_resolve_block_size`` picks the largest power-of-2 divisor of
  ``in_dim`` when ``SGLANG_OPT_HIGGS_MOE_2BIT_BLOCK_SIZE`` is unset,
  so DSv3.2-REAP shapes load cleanly without env knobs. Composition
  rule honored: this is the dispatch — there's no config-branched
  alternate. The BF16 runtime fallback already calls
  ``_dequant_to_bf16``, so updating that helper is enough to wire the
  fast path through ``_apply_bf16_runtime``.

* ``test/srt/test_higgs_moe_2bit_dequant_fast.py``: unit + scheme +
  bench coverage. Asserts ``higgs_moe_2bit_dequant_fast`` matches the
  eager ``dequantize_higgs_moe_weights`` within BF16 round-off on
  small toy, single-block (in_dim=2048), and sub-row-blocked
  (in_dim=7168 block=1024) shapes; asserts the scheme helper routes
  through the fast kernel; logs the per-call ms at DSv3.2 per-rank
  shapes for benchmark visibility.

Measured kernel-level speedup (B200, DSv3.2 per-rank, best of 10):

  shape (E, OUT, IN, block)              eager     iter2    speedup
  -------------------------------------  --------  -------  -------
  w13  (32, 4096, 7168, block=1024)       62.59 ms  2.21 ms  28.3x
  w2   (32, 7168, 2048, block=2048)       33.92 ms  1.02 ms  33.2x
  total per MoE layer                     96.52 ms  3.23 ms  29.9x
  x58 MoE layers per decode step         5598 ms    187 ms

Unpack-only (no FWHT, for arithmetic): w13 0.65 ms, w2 0.30 ms — at
2.8 / 3.4 TiB/s = 35% / 42% of B200 HBM peak. The remaining 1.57 ms
(w13) / 0.72 ms (w2) is the existing fast-Hadamard CUDA kernel which
itself runs at ~2.2 TiB/s on these shapes.

End-to-end TPOT delta vs the active FP8 trtllm-DSA deploy (4091 agg
tok/s, 25.9 ms TPOT):

* The default iter1 dispatch (post-load dequant -> NVFP4-quantize ->
  trtllm fp4 MoE) is unchanged by this commit. Runtime is still
  bit-identical to the W4A4 NVFP4 path, so default-deploy TPOT delta
  is zero.

* The BF16 runtime path (opt-in via the existing env var) is what
  changed. iter1 cost at production shape: ~5.6 sec / decode step for
  dequant alone -> totally unusable. iter2 cost: 187 ms / decode step
  for dequant. That's still ~7-8x the FP8 baseline TPOT, so the BF16
  runtime path is not yet production-deploy-ready — but it's now
  within striking distance of an A/B comparison, which iter1 wasn't.
  The remaining gap is the actual BF16 expert GEMM cost
  (``trtllm_bf16_moe``), which has different shape constraints than
  the NVFP4 kernel.

* The actual e2e perf win arrives in iter3, which the brief and
  sibling subagent sgl-project#19's profiling both flag as a CuTe-fused
  HIGGS-dequant + BMM kernel. That kernel would consume HIGGS-packed
  weights inline (skip the BF16 materialization), reading ~2.25x less
  weight memory per call than the NVFP4 path. With weight HBM as the
  binding constraint at decode, that's worth a measurable TPOT
  reduction relative to FP8 trtllm-DSA. iter2 unblocks iter3 by
  delivering a tested codec primitive that the CuTe kernel can use as
  an oracle.

What's left for iter3 (documented in scheme docstring + NOTE markers
at the call site):

(a) Fuse the FWHT into the same Triton (or CuTe) kernel as the
    unpack+scale step. That eliminates the 235 MiB BF16 round-trip
    between unpack and hadamard, projected ~2x further on dequant
    cost (~1.5 ms / layer). Triton's lack of bit-shuffle primitives
    for FWHT_1024 makes a CuTe-DSL implementation the natural fit
    (quack has a reference at width up to 32K).

(b) The actual perf-win kernel: HIGGS-2bit -> BF16 dequant fused
    with the per-expert BMM, mirroring W4A16 CuTe kernels. Inputs:
    HIGGS-packed [E, out_dim, slot] uint8, activation [tokens,
    hidden] bf16. Output: per-expert BMM result. Avoids
    materializing the BF16 working tile entirely — weight HBM read
    drops 2.25x vs NVFP4. The CuTe-DSL reference set (cutest, quack,
    cutlass cute, blackwell tutorial, CuTe DSL docs, CuTe paper) is
    mirrored at
    ``/home/spencergarnets/higgs_cute_rewrite_plan/references``.

Verification on the decode pod (PYTHONPATH-mounted override; pod is
under sgl-project#14 debug so we don't restart it):

  $ pytest -xvs test/srt/test_higgs_moe_2bit_dequant_fast.py
  6 passed, 2 warnings in 10.44s

  TestHiggsMoeDequantFast::test_dsv32_w13_shape_sub_row_blocking PASS
  TestHiggsMoeDequantFast::test_dsv32_w2_shape                    PASS
  TestHiggsMoeDequantFast::test_medium_pow2_single_block          PASS
  TestHiggsMoeDequantFast::test_small_pow2                        PASS
  TestHiggsMoeDequantFastViaScheme::
    test_scheme_helper_routes_through_fast_kernel                 PASS
  TestHiggsMoeDequantFastBench::test_bench_dsv32_per_rank         PASS
    [higgs_moe_dequant_fast] per layer (best of 10): 3.229 ms
    (x58 layers: 187.3 ms)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets pushed a commit to BlaiseSI/optimization-playground that referenced this pull request May 29, 2026
…+ cached slice

Adds SGLang JIT NVFP4 quantize with linear (non-swizzled) SF layout - drop-in for flashinfer.fp4_quantize used by trtllm_fp4_block_scale_moe on the DeepSeek-V3.2-REAP NVFP4 MoE deploy. Env-var SGLANG_USE_SGL_NVFP4_QUANT=1 enables; default off. Caches w13_input_scale_quant[:1] slice on the layer to skip a re-slice each decode step. Iter2 will fuse this kernel with upstream RMSNorm path when allreduce-fusion is not engaged.
SpencerGarnets pushed a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…r NVFP4 quantize) kernel

Adds a single-CTA-per-row kernel that collapses the upstream
`fused_add_rmsnorm` BF16 write+read of hidden_states with the
downstream `scaled_fp4_quant_linear` BF16 read. On B200 at production
hidden_size=7168 this saves the ~1.8 MB/layer of HBM traffic plus one
launch-overhead amortization per layer on the non-allreduce-fusion path
(small/medium batch decode, or when the closed flashinfer
allreduce-fusion is disabled).

Public API: `sglang.jit_kernel.nvfp4.fused_rmsnorm_scaled_fp4_quant_linear`
(also re-exported as `sglang.jit_kernel.norm.fused_add_rmsnorm_to_nvfp4_linear`
for callers already importing the fused_add_rmsnorm pair).

Correctness: 100% bit-exact residual + FP4 + SF byte match against the
unfused pair at m=1,8,16,32,64,128,256 for hidden=7168
(test/srt/test_nvfp4_fused_rmsnorm_quant.py).

Bench at hidden=7168, 58 layers/decode-step on B200 (CUDA-graph
captured, single GPU):
  m=  1: unfused=4.38us  fused=4.13us  fused+pdl=3.55us  saved=0.25us/layer  step=14.4us
  m=  8: unfused=4.43us  fused=4.14us  fused+pdl=3.57us  saved=0.29us/layer  step=16.8us
  m= 16: unfused=4.43us  fused=4.19us  fused+pdl=3.78us  saved=0.24us/layer  step=13.9us
  m= 32: unfused=4.47us  fused=4.19us  fused+pdl=3.82us  saved=0.27us/layer  step=15.7us
  m= 64: unfused=4.54us  fused=4.23us  fused+pdl=3.81us  saved=0.31us/layer  step=17.7us
  m=128: unfused=4.63us  fused=4.31us  fused+pdl=3.91us  saved=0.33us/layer  step=19.0us
  m=256: unfused=5.63us  fused=6.49us                                         (regression; compute-bound)

Iter2 secondary vector (PDL): `enable_pdl=True` plumbs through
`LaunchKernel.enable_pdl`. Bench shows additional 0.32-0.55 us/layer
saved on top of the fuse — kernel dispatches in the shadow of the
preceding global store (allreduce store or attention output write).
PDL gain is largest at low m where launch overhead dominates.

Iter2 stretch (NOT in this commit, documented in
nvfp4_fused_rmsnorm_quant_kernels.cuh tail): vendor a copy of
trtllm_fused_moe_kernel_launcher.cu, relax the MxE2m1 activation check,
rebuild via SGLang's JIT pipeline. That would unlock the 58
bmm_Bfloat16_E2m1E2m1_* in-CUBIN variants consuming BF16 activations
directly and eliminate the entire fp4_quantize step including this
fused kernel.

Foundation fixes (load-bearing):
  * `#pragma once` on nvfp4_quant.cuh, nvfp4_quant_kernels.cuh, and
    nvfp4_quant_linear_kernels.cuh. Without these the linear quant
    iter1 module fails to compile when the fused kernel pulls in
    `cvt_warp_fp16_to_fp4` via `#include nvfp4_quant_kernels.cuh`
    (double-definition errors). Confirmed iter1 `_jit_nvfp4_quant_linear_module()`
    was broken at HEAD pre-commit; now compiles clean.

m=256 regression: at that batch size the fused kernel's register
pressure (one CTA per row with deeper per-thread work) costs more than
the saved HBM traffic. Production target QPS keeps decode batch
mostly in m=1..128, where the fused path wins consistently. Iter3
vectors below address m>=256.

Deploy-delta projection (DP=TP=8, 58 MoE layers, B200 8 TB/s HBM):
  - Per-step TPOT win at m=1..128: 14-19 us (~0.3-0.4% at 5ms TPOT
    target, ~2.3-3.8% at 500us TPOT target)
  - With PDL: 18-49 us / step
  - Iter1 deploy delta was ~0 (foundation); this is the real perf win.

Iter3 vectors:
  - Cooperative SMEM staging for m>=256 to amortize register pressure.
  - Multi-stream pipelining (route the residual write to a separate
    stream from the FP4 store path).
  - Stretch goal (launcher patch): vendor + JIT-rebuild
    trtllm_fused_moe_kernel_launcher.cu, drop the MxE2m1 act check,
    invoke bmm_Bfloat16_E2m1E2m1_* directly with BF16 activations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets pushed a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
… apply_weights consumer + RMSNorm helper)

Secondary vector of iter3 — completes the kernel-side iter3 wide+SMEM
landed via 2f0dcfd (mistagged inside a DSA-indexer commit body but
functionally correct; the iter3 wide variant is at
nvfp4_fused_rmsnorm_quant_kernels.cuh:cvt_fused_rmsnorm_to_fp4_linear_wide).

Adds the activation surface so a deploy can flip the fused kernel on
without further C++ changes:

  * environ.py — registers SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM (EnvBool,
    default False). Comment block documents the three preconditions:
    compressed_tensors_w4a4_nvfp4_moe + flashinfer_trtllm runner,
    should_allreduce_fusion=False, non-cast_x_before_out_mul RMSNorm.

  * compressed_tensors_w4a4_nvfp4_moe.apply_weights — when env is set
    AND the incoming hidden_states carries
    `_sglang_pre_quantized_fp4 = (fp4, sf)`, consume the stash and skip
    the redundant fp4_quantize call. Defensive del-after-consume guards
    against moe_cp re-runs reusing the stale buffer.

  * RMSNorm.forward_with_fused_nvfp4_quant — wrapper that calls
    fused_rmsnorm_scaled_fp4_quant_linear and returns (fp4, sf, residual).
    Use it from a hook that knows the global_scale (the MoE layer's
    cached w13_input_scale_quant_slice). No call site is plumbed inside
    DeepSeekV2 yet — see Iter4 vectors below for the DP-aware hookup.

Why this is staged as activation-only:
  * The production deploy runs --dp 8 --tp 8 which hits the
    `context.attn_dp_size != 1` branch of
    `_gather_hidden_states_and_residual`. In that branch the local
    layernorm runs at line 995, then dp_gather_partial materializes a
    bf16 buffer at line 1008 before the MoE sees it. A naive fuse at
    line 995 would produce LOCAL (fp4, sf) but the gather is sized for
    bf16, so the trtllm consumer would see the wrong layout. The
    DP-aware fuse needs either a parallel fp4-aware gather or a
    post-gather quantize — that is iter4 work.

  * In the meantime, this commit lets a deploy verify the iter2/iter3
    kernel against its own dispatcher by setting the env var, attaching
    `_sglang_pre_quantized_fp4` from a unit-test fixture, and
    confirming apply_weights consumes the stash without regression.

Iter4 vectors (deferred):
  * DP-aware fuse in `_gather_hidden_states_and_residual` line 995:
    either (a) extend dp_gather_partial to gather fp4+sf instead of
    bf16, or (b) fuse layernorm + scaled_fp4_quant_linear inline AFTER
    the gather (line 1014, hidden_states already gathered).
  * Multi-stream pipelining of the residual write — schedule the
    p_res store on a side stream so the fp4 store can race ahead.
  * cast_x_before_out_mul correctness sweep — bench/test currently
    covers only Llama-style; add parameterized test for HF parity.
  * Stretch: vendor trtllm_fused_moe_kernel_launcher.cu, drop the
    MxE2m1 activation-dtype check, dispatch
    bmm_Bfloat16_E2m1E2m1_* directly with BF16 activations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets pushed a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…ches (m=384, 512)

Adds m=384 and m=512 to the existing
test_nvfp4_fused_rmsnorm_quant.py coverage so the iter3 wide+SMEM
variant (auto-selected at m>=kWideMinM=192) gets exercised at multiple
batch sizes, not just at the regression-driver m=256.

m=256 was already covered by iter2's DECODE_BATCHES; with iter3 it now
exercises the wide kernel. m=384 sits on the mid-batch slope where
register-pressure was iter2's regression mode. m=512 brings prefill
shapes into the same parametrize sweep so the m grid is contiguous
across the decode/prefill boundary.

Both correctness (test_fused_matches_unfused) and perf
(test_perf_vs_unfused) tests inherit the new ALL_BATCHES sweep. No
change to the unfused reference path — same fused_add_rmsnorm +
scaled_fp4_quant_linear pair used as the baseline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets pushed a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
… dispatch

Hot fix for the wide-variant dispatch helper added in the iter3 wide
SMEM-staged kernel. The launcher uses `std::getenv` and `std::atoi` to
read the SGLANG_NVFP4_FUSED_WIDE_MIN_M tuning override. Neither symbol
is brought in by the pre-existing CUDA-runtime includes, so add
<cstdlib> explicitly. Without this nvcc on some toolchains errors out
("std::getenv was not declared in this scope").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…trized test

iter3 tertiary vector held over. The fused (residual-add + RMSNorm +
linear NVFP4 quantize) kernel takes a `cast_x_before_out_mul` flag that
controls whether the post-norm accumulator is rounded to dtype before
multiplying by the RMSNorm weight (HF parity) or left in fp32 until the
weight multiply (Llama parity).

Iter2 / iter3 only exercised the False (Llama-style) path because the
DeepSeek-V3.2-REAP deploy uses Llama-style RMSNorm. But the kernel
emits a different specialization for cast_x=True (the round trip
through dtype changes which fp4 bucket a value falls into at
~50% of the boundary cases), so the True path needs its own correctness
coverage before another deploy can rely on it.

Parametrize test_fused_matches_unfused on cast_x in {False, True}. The
unfused reference now threads cast_x through to fused_add_rmsnorm so
the comparison kernel and the fused kernel see the same rounding mode.
Existing thresholds (residual max < 0.05, FP4 vals > 95% match, SF
bytes > 95% match) carry forward unchanged -- the cast_x=True path
should pass the same gates.

The if __name__ == "__main__" smoke runner picks up both flag values.

No production code touched; deploy preconditions still require
cast_x_before_out_mul=False on the post-attention RMSNorm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…nt kernel into post-attn layernorm path

Closes the iter3 deploy-wire stub. The iter3 commit (3a57f94) landed
the activation surface — env var + RMSNorm.forward_with_fused_nvfp4_quant
helper + apply_weights consumer hook in compressed_tensors_w4a4_nvfp4_moe
— but the RMSNorm helper was NEVER called from the actual forward path,
so deploy TPOT delta was 0. This commit plumbs the kernel into the real
call site.

Target call site: communicator.py
CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual,
the `not use_layer_norm_before_gather` branch where
`hidden_states = layernorm(hidden_states)` is invoked after the DP-attn
allgather + dp_scatter. The residual has already been folded into
hidden_states at attn_tp_rank==0 (line 997), so the layernorm here is a
pure RMSNorm — NO residual-add.

New residual-less variant of the fused kernel:
  `fused_rmsnorm_only_to_fp4_linear(x, weight, gs, eps, *, enable_pdl)`
  → `(y_bf16, fp4, sf)`

The kernel produces BOTH the post-norm BF16 hidden_states (consumed by
the gate router, shared_experts, and topk path) AND the FP4 + linear-SF
stash (consumed by the NVFP4 MoE expert apply_weights via the iter3
`_sglang_pre_quantized_fp4` hook in compressed_tensors_w4a4_nvfp4_moe.py).
Both consumers are downstream of the wired call site, with no
intervening allreduce / allgather / scatter that would invalidate the
stash — the gathered hidden_states flows directly into
DeepseekV2MoE.forward.

Wire surface:
  * RMSNorm.forward_with_fused_nvfp4_quant_no_residual(x, gs, *, enable_pdl)
    — helper that calls the new kernel; matches the existing
    forward_with_fused_nvfp4_quant helper (residual variant) signature
    minus the residual arg.
  * LayerCommunicator.prepare_mlp(..., next_consumer_fp4_global_scale=)
    — new optional kwarg; the deepseek_v2 caller passes
    self.mlp.experts.w13_input_scale_quant_slice when the next layer
    is an NVFP4 MoE expert. Stored on
    CommunicateContext.next_consumer_fp4_global_scale as per-call
    scratch; cleared in a finally block.
  * communicator.py line 1042 (the `hidden_states = layernorm(hidden_states)`
    call): when (env flag set) AND (next_consumer_fp4_global_scale set)
    AND (layernorm has the new helper) AND (dtype is fp16/bf16), the
    layernorm call is replaced with the fused-quant path. The stash is
    attached as `y._sglang_pre_quantized_fp4 = (fp4, sf)` and the
    consumer dels the attr after reading (iter3 commit).

Correctness (100% bit-exact vs flashinfer.rmsnorm + scaled_fp4_quant_linear pair):
  m=  1: y max-err=0.000000  FP4 match=100%  SF match=100%
  m=  8: y max-err=0.000000  FP4 match=100%  SF match=100%
  m= 16: y max-err=0.003906  FP4 match=100%  SF match=100%
  m= 32: y max-err=0.007812  FP4 match=100%  SF match=100%
  m= 64: y max-err=0.007812  FP4 match=100%  SF match=100%
  m=128: y max-err=0.007812  FP4 match=100%  SF match=100%
  m=256: y max-err=0.007812  FP4 match=100%  SF match=100%
  m=512: y max-err=0.015625  FP4 match=100%  SF match=100%

(y max-err = BF16 ULP differences from rsqrt rounding order; mean-err
column was 0.0 at every m.)

Bench at hidden=7168, 58 MoE layers/decode-step on B200 (CUDA-graph
captured, single GPU):
  m=  1: unfused=6.20us  fused=6.16us  fused+pdl=4.12us  save/step=1.9us/120.3us
  m=  8: unfused=6.17us  fused=6.16us  fused+pdl=4.15us  save/step=0.4us/117.0us
  m= 16: unfused=6.17us  fused=4.12us  fused+pdl=4.12us  save/step=118.8us/118.8us
  m= 32: unfused=6.17us  fused=4.12us  fused+pdl=4.12us  save/step=118.6us/118.5us
  m= 64: unfused=6.17us  fused=4.14us  fused+pdl=4.15us  save/step=117.4us/117.1us
  m=128: unfused=6.17us  fused=4.23us  fused+pdl=4.22us  save/step=112.7us/113.4us
  m=256: unfused=8.21us  fused=6.17us  fused+pdl=6.17us  save/step=118.4us/118.5us
  m=512: unfused=10.26us fused=10.26us fused+pdl=10.26us save/step=0.1us/0.3us

Eager (no graph capture) at all m: 1.56-1.67x speedup, ~10us saved/call
(launch overhead amortization dominates outside CUDA graphs).

Production decode batch range per rank with DP=TP=8 is m=[1, batch/8].
For peak decode (batch=128) m=16/rank — the fused+PDL path saves ~2us
× 58 layers = ~118us/step on the post_attention_layernorm path alone.
At the baseline 25.9 ms TPOT this is a 0.46% improvement; the real
win compounds with the iter2 wire (residual variant) on the shared
dense path and with the cast_x iter4 secondary that will fuse
intermediate quantize calls.

m=512 has no graph-capture win (kernel becomes compute-bound at high m
with idle threads on the wide pass); for prefill-shape m=512 the
caller's env flag is left off via the per-call surface check
(SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM is decode-only in deploy).

Recurring regression guards (verified):
  * No torch.mm(..., out_dtype=...) introduced (Dynamo meta_mm
    incompatibility from 34a927b).
  * compilation.piecewise_context_manager.get_token_to_kv_pool not
    touched (the upstream-removed path).
  * No CPU sync ops inside cuda graph capture (the BF16 stash is
    a pure device tensor mutation).

Files:
  * python/sglang/jit_kernel/csrc/gemm/nvfp4/nvfp4_fused_rmsnorm_quant_kernels.cuh
    — new `cvt_fused_rmsnorm_only_to_fp4_linear` template kernel and
    its `invokeFusedRMSNormOnlyFP4QuantLinear` launcher +
    `fused_rmsnorm_only_to_fp4_linear_sm100a_sm120a` host entry.
  * python/sglang/jit_kernel/nvfp4.py
    — register the new entry in `_jit_nvfp4_fused_rmsnorm_quant_module`,
    add `_fused_rmsnorm_only_to_fp4_linear_custom_op` torch custom op
    + Python `fused_rmsnorm_only_to_fp4_linear` wrapper.
  * python/sglang/srt/layers/layernorm.py
    — add `RMSNorm.forward_with_fused_nvfp4_quant_no_residual`.
  * python/sglang/srt/layers/communicator.py
    — `CommunicateContext.next_consumer_fp4_global_scale` per-call
    scratch field; `LayerCommunicator.prepare_mlp` kwarg
    `next_consumer_fp4_global_scale` with finally-cleared scope;
    line 1042 fused-quant short-circuit with env + helper + dtype
    guards.
  * python/sglang/srt/models/deepseek_v2.py
    — surface the MoE expert input_global_scale to prepare_mlp
    (the w13_input_scale_quant_slice cached on the layer at
    load-time, [:1] view).
  * test/srt/test_nvfp4_fused_rmsnorm_only_quant.py
    — correctness + bench parametrized over m={1,8,16,32,64,128,256,512}.

Iter+1 vector targets (if needed):
  * Secondary: also wire the residual variant
    (forward_with_fused_nvfp4_quant) at the L1024 fused_add_rmsnorm path
    inside the same _gather_hidden_states_and_residual function (the
    use_layer_norm_before_gather=True path, attn_tp_size==1). That path
    currently writes BF16 hidden_states for the dp_gather_partial that
    follows; needs kernel extension to also emit BF16 alongside FP4.
    Architectural complexity: medium.
  * Tertiary: extend the kernel cast_x_before_out_mul=True path
    (Gemma/glm4-style HF parity); currently bench shows the kernel
    works on both, but no deploy uses cast_x for DSv3.2.
  * Iter5: investigate flashinfer's bmm_Bfloat16_E2m1E2m1_* CUBIN
    launcher patch to fully eliminate the fp4_quantize step (deferred
    multi-day stretch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…ast_x_before_out_mul

Mirrors the parametrization from sgl-project#15 iter4 commit 62fff01 (tertiary
test on the residual variant) but applied to the iter4 PRIMARY
residual-less variant landed in 3fc574b.

Production deploys (DSv3.2-REAP) use the Llama-style kCastXBeforeOutMul=false
path. Gemma/glm4-style HF-parity uses kCastXBeforeOutMul=true; the
template kernel already supports both via its `cast_x_before_out_mul`
arg, but the test only covered the false path until now.

This commit parametrizes test_fused_no_residual_matches_unfused over
both modes, and updates _unfused_reference to compute the HF-parity
math in plain torch (flashinfer's rmsnorm only exposes Llama-style).

Bit-exact results at all 16 (m, cast_x) combinations:

  m=1   cast_x=False: y max-err=0.000000  FP4 100%  SF 100%
  m=1   cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=8   cast_x=False: y max-err=0.000000  FP4 100%  SF 100%
  m=8   cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=16  cast_x=False: y max-err=0.003906  FP4 100%  SF 100%
  m=16  cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=32  cast_x=False: y max-err=0.007812  FP4 100%  SF 100%
  m=32  cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=64  cast_x=False: y max-err=0.007812  FP4 100%  SF 100%
  m=64  cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=128 cast_x=False: y max-err=0.007812  FP4 100%  SF 100%
  m=128 cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=256 cast_x=False: y max-err=0.007812  FP4 100%  SF 100%
  m=256 cast_x=True : y max-err=0.000000  FP4 100%  SF 100%
  m=512 cast_x=False: y max-err=0.015625  FP4 100%  SF 100%
  m=512 cast_x=True : y max-err=0.000000  FP4 100%  SF 100%

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…6 emit (L1024 wire, honest negative graph result)

Adds a second fused kernel variant for the `use_layer_norm_before_gather=True`
branch of CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual
(communicator.py L1024) — the path that fires under DSv3.2-REAP's
DP=TP=8 + attn_tp_size=1 deploy config.

Unlike iter2's residual variant (which omits the BF16 post-norm output
to save HBM bandwidth), this variant ALSO emits the BF16 hidden_states
because the immediately-following dp_gather_partial requires it for
the cross-DP allgather. The FP4 stash this kernel computes is wasted
in current deploy (local-rows FP4 doesn't survive the gather) and is
preserved as a no-op stash for the iter5 FP4-allgather plumbing.

Kernel: `cvt_fused_rmsnorm_to_fp4_and_bf16_linear` template
launcher: `invokeFusedRMSNormFP4AndBf16QuantLinear`
public entry: `fused_rmsnorm_to_fp4_and_bf16_linear_sm100a_sm120a`

Python wrapper: `fused_rmsnorm_to_fp4_and_bf16_linear(x, residual, w, gs)`
  → `(y_bf16, fp4, sf)`  (residual is mutated in-place to `x + residual`)

Helper: `RMSNorm.forward_with_fused_nvfp4_quant_and_bf16_out`

Wire: communicator.py L1024 `use_layer_norm_before_gather` branch,
gated by SGLANG_USE_SGL_NVFP4_FUSED_RMSNORM env + helper availability
+ next_consumer_fp4_global_scale set + dtype check. Same env flag as
the iter4 PRIMARY L1042 wire — both wires fire together under the
same deploy config knob.

Correctness (test/srt/test_nvfp4_fused_rmsnorm_and_bf16.py, 16/16 PASS):
  cast_x=False (production / Llama-style) at m in {1,8,16,32,64,128,256,512}:
    100% bit-exact FP4 + SF + residual + y (max y-err = bf16 ULP)
  cast_x=True (HF-parity / Gemma-style) at all m:
    98.9-99.0% FP4 match, 98.5-98.6% SF match — small rounding-order
    delta vs the plain-torch reference; production uses cast_x=False.

Bench at hidden=7168, 58 layers/step on B200:
  Eager (no graph): 1.05-1.10x speedup, ~1-2us/call (launch amort)
  CUDA-graph captured:
    m=  1: unfused=4.13us  fused=4.13us  fused+pdl=4.13us  step=0us
    m=  8: unfused=4.13us  fused=4.14us  fused+pdl=4.13us  step=0us
    m= 16: unfused=4.13us  fused=4.13us  fused+pdl=4.13us  step=0us
    m= 32: unfused=4.13us  fused=4.13us  fused+pdl=4.13us  step=0us
    m= 64: unfused=6.17us  fused=6.17us  fused+pdl=6.17us  step=0us
    m=128: unfused=6.17us  fused=6.17us  fused+pdl=6.17us  step=0us
    m=256: unfused=8.22us  fused=8.22us  fused+pdl=8.22us  step=0us
    m=512: unfused=11.41us fused=12.32us fused+pdl=12.32us step=-53us (REGRESSION)

HONEST NEGATIVE: at L1024 in CUDA-graph mode the BF16 emit penalty
cancels the launch amortization, so the deploy delta is 0 at decode
batches m<=256 and a slight regression at m=512. The iter5 FP4
allgather is needed to unlock the FP4 stash and recover the win on
this wire.

The wire still lands behind the same env flag so:
  1. Validation / eager-mode runs get the 1.05-1.10x save.
  2. The architecture is plumbed for iter5 (FP4 allgather) to drop in.
  3. The correctness checks codify the kernel as a known-good baseline
     for the eventual FP4-allgather plumbing.

iter5 vector to make this wire deliver in graph mode:
  Allgather the (fp4, sf) tuple in parallel with the BF16 allgather,
  pass through to the MoE expert apply_weights as the post-gather
  stash. Requires a new collective op (FP4 byte allgather + SF
  allgather) and rework of dp_gather_partial to call it.

Files:
  * python/sglang/jit_kernel/csrc/gemm/nvfp4/nvfp4_fused_rmsnorm_quant_kernels.cuh
    — new template kernel + launcher + host entry.
  * python/sglang/jit_kernel/nvfp4.py
    — register entry, add custom op + wrapper.
  * python/sglang/srt/layers/layernorm.py
    — add RMSNorm.forward_with_fused_nvfp4_quant_and_bf16_out.
  * python/sglang/srt/layers/communicator.py
    — wire at L1024 with env + helper + dtype guard.
  * test/srt/test_nvfp4_fused_rmsnorm_and_bf16.py
    — parametrized correctness + perf (cast_x x m).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…communicator.py L1024

Closes the iter4 SECONDARY honest-negative result (commit ba89b7f) by
wiring the iter5 dp_gather_partial_fp4 collective (commit c263c7d's
dp_attention.py / test_nvfp4_dp_gather_partial_fp4.py — vacuumed into
the HIGGS iter7 sibling commit by accident but the diff is the iter5
collective infra) into CommunicateWithAllReduceAndLayerNormFn.
_gather_hidden_states_and_residual's use_layer_norm_before_gather=True
branch.

Wire surface: immediately after `dp_gather_partial(hidden_states,
local_hidden_states, forward_batch)` (L1073 in the iter4 SECONDARY
commit's numbering), check for the iter4-SECONDARY-emitted local FP4
stash on local_hidden_states. If present, allocate global FP4 + SF
buffers sized to `[M_global, n//2]` + `[M_global, (n//16)//4]` (int32
storage; viewed as fp8_e4m3fn for the consumer), call
dp_gather_partial_fp4 to populate them, and reattach the stash on the
new gathered BF16 tensor for the NVFP4 MoE expert apply_weights
consumer (compressed_tensors_w4a4_nvfp4_moe.py L353 hook).

Deploy gate: `_ITER5_FP4_ALLGATHER_M_GLOBAL_MAX = 128`. The iter5
grouped collective is +0.7us/layer SAVED at the deploy decode peak
(m_global=128 @ DP=TP=8, batch=128) but regresses by ~7-10us/layer
past m_global=256. The hard gate matches the deploy decode peak — the
NVFP4 MoE wire only fires for m_global <= 128. Past that, the local
stash is discarded and the downstream MoE expert falls back to its
own fp4_quantize on the gathered BF16 (the iter4 SECONDARY honest-
negative behavior).

Bench at hidden=7168, B200, NCCL_NVLS_ENABLE=0, dp=2 graph-captured
pynccl (the deploy-relevant measurement):

  m_global=128 (deploy decode peak):
    iter4 SECONDARY (BF16 emit + post-gather fp4_quantize): 23.82us
    iter5 PRIMARY  (BF16 + FP4 + SF grouped, no consumer quant): 23.44us
    delta per layer: +0.38us SAVED

  Cumulative sgl-project#15 deploy projection over 58 MoE layers/step:
    iter1-3 (sgl-native + fused-RMSNorm + wide-SMEM): foundation
    iter4 PRIMARY  (L1042 residual-less wire):  +118us/step
    iter4 SECONDARY (L1024 residual wire):       0us/step (honest-neg)
    iter5 PRIMARY  (L1024 FP4 allgather):       +22us/step
    -----------------------------------------------------------------
    cumulative sgl-project#15 deploy save:                +140us/step

  TPOT projection: 25.9 ms baseline -> 25.76 ms TPOT (0.54%).
  Cumulative ai-blaise main delta at the production REAP grid stays
  on the +118us iter4 PRIMARY backbone; iter5 PRIMARY adds the small
  L1024 incremental win.

Honest scope: the iter5 wire is small relative to iter4 PRIMARY
(118us vs 22us) because the L1024 path's FP4 stash collective and
the saved fp4_quantize at the consumer roughly cancel out — the
~+0.4us/layer win comes purely from pynccl ncclGroupStart/End
fusing the two new allgathers into one launch. Per-call the wire
adds FP4 (3584 bytes/token) + SF (448 bytes/token) of allgather
traffic on top of BF16 (14336 bytes/token), but removes a
~3.6us/layer fp4_quantize from the consumer. Production peak gets
the difference; past m_global=128 the bytes overhead exceeds the
saved cost and we drop the stash to keep the L1024 wire honest.

Correctness:
  * Backed by the dp_gather_partial_fp4 unit test (in c263c7d),
    which validated 100% bit-exact FP4 + 100% bit-exact SF vs the
    BF16-allgather-then-quantize reference at m_local in
    {1, 8, 16, 32, 64, 128, 256, 512} x dp=2.
  * The wire-up only re-attaches the stash; no new arithmetic is
    introduced.

Recurring regression guards:
  * No torch.mm(..., out_dtype=...) introduced.
  * No torch.cumsum over (dp_size,) added.
  * No CPU sync inside CUDA graph capture (all ops are device-side
    or pynccl-captured).
  * forward_batch.token_to_kv_pool not touched.
  * compressed_tensors_w4a4_nvfp4_moe.py consumer hook unchanged —
    the stash attribute name and tuple shape are identical to the
    iter4 PRIMARY L1042 wire.

Files:
  * python/sglang/srt/layers/communicator.py
    — import dp_gather_partial_fp4; after the L1073
       dp_gather_partial call, run the iter5 FP4 + SF collective and
       re-attach the global stash when m_global <= 128.

Iter6 next vectors:
  * Extend dp_gather_partial_fp4 to support sum_len padding for the
    DSA-CP prefill path (currently raises NotImplementedError).
    Reuses the rank-offset memcpy_triton pattern from
    _dp_gather_via_all_reduce but with an allgather instead of
    allreduce (allreduce sums FP4 bytes which corrupts them).
  * Investigate fusing dp_gather_partial (BF16) + dp_gather_partial_fp4
    into a single ncclGroupStart/End spanning all THREE allgathers.
    Bench projection: another ~+1us/layer at production peak by
    sharing the per-launch overhead with the BF16 collective.
  * Investigate flashinfer bmm_Bfloat16_E2m1E2m1_* CUBIN launcher
    patch to eliminate the iter4 PRIMARY fp4_quantize call entirely
    (deferred multi-day stretch; see iter4 PRIMARY commit body for
    target API surface).

Honest negatives kept as opt-in checkpoints:
  * Past m_global=128, the iter5 wire is a clear loss (-9.75us/layer
    at m_global=256). The hard gate `_ITER5_FP4_ALLGATHER_M_GLOBAL_MAX`
    makes the wire a strict subset of the iter4 SECONDARY env-flag
    deploy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…Group fusion plan)

Recon notes for the iter6 PRIMARY vector: fuse the BF16 + FP4 + SF
allgathers (currently 2 launches: BF16 separate, FP4+SF grouped via
iter5 ab38f95) into a single ncclGroupStart/End block (1 launch).

Plan landed: new dp_gather_partial_bf16_fp4_fused in dp_attention.py
that wraps all 3 in one pynccl_comm group_start/end pair; communicator.py
L1024 wire refactored to call it when the iter4 SECONDARY FP4 stash is
present and m_global <= 128.

Bench will reuse test_nvfp4_dp_gather_partial_fp4.py infrastructure:
the _iter5_grouped helper already groups all 3, so iter6 = lifting that
into the production wire. The bench delta is the production launch
saved.

Per iter5 projection: ~+1us/layer at production peak. Honest-negative
fallback ready if bench shows <0.3us/layer.

IKP not applicable to this vector (host-side NCCL launch fusion, not
intra-kernel instruction profiling). Noted in recon for iter6 TERTIARY
flashinfer bmm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
…P4+SF) into one ncclGroupStart/End

Iter5 PRIMARY (ab38f95) grouped FP4 + SF allgathers under a single
ncclGroupStart/End but left the BF16 hidden_states allgather as a
separate launch (via reg_all_gather_into_tensor -> pynccl_comm.all_gather
without group wrapping). Captured graph saw TWO NCCL launches per
NVFP4 MoE layer.

Iter6 PRIMARY collapses to ONE launch. New collective
dp_gather_partial_bf16_fp4_fused in dp_attention.py wraps all 3
allgathers in one pynccl_comm.group_start()/group_end() pair, bypassing
the reg_all_gather_into_tensor custom op wrapper for the BF16 leg so
the three calls can sit between the same group boundary.

Wire surface: communicator.py L1024
CommunicateWithAllReduceAndLayerNormFn._gather_hidden_states_and_residual
use_layer_norm_before_gather=True branch. Peek the iter4-SECONDARY
local FP4 stash BEFORE the gather; if present and m_global <= 128
(the iter5 break-even gate, unchanged), call
dp_gather_partial_bf16_fp4_fused. Otherwise fall back to the original
dp_gather_partial BF16-only path (no FP4 stash to gather).

Bench at hidden=7168, B200, NCCL_NVLS_ENABLE=0, dp=2 graph-captured
pynccl, 2000 iters per variant, 3 trials. iter5 PROD column shows the
two-launch pattern (BF16 separate + FP4+SF grouped); iter6 PROD column
shows the single-launch pattern (all 3 grouped):

  m_local | iter5 PROD (us) | iter6 PROD (us) | iter5->iter6 delta
  --------+-----------------+-----------------+-------------------
        1 |          25.07  |          14.76  | +10.31us SAVED
        8 |          27.54  |          17.84  |  +9.70us SAVED
       16 |          29.40  |          19.18  | +10.22us SAVED
       32 |          29.18  |          19.88  |  +9.30us SAVED
       64 |          33.94  |          23.49  | +10.45us SAVED
      128 |          42.78  |          41.34  |  +1.44us SAVED   <- production decode peak
      256 |          52.88  |          44.57  |  +8.31us SAVED   (gate-off: iter6 wire never fires here in production)
      512 |          67.56  |          52.99  | +14.57us SAVED   (gate-off: iter6 wire never fires here in production)

Production decode peak (m_global=128, batch=128 @ DP=TP=8) saves
+1.44us per NVFP4 MoE layer. The win is modest at peak because the
BF16 leg alone (~27us) dominates the launch overhead at that size;
the per-launch fixed cost being amortized is the smaller piece. At
smaller decode batches (m_global <= 64) the win is +9-10us/layer
because launch overhead is a much larger fraction of total cost.

Cumulative sgl-project#15 deploy projection at peak (58 NVFP4 MoE layers/step):
  iter1-3 (sgl-native + fused-RMSNorm + wide-SMEM): foundation
  iter4 PRIMARY  (L1042 residual-less wire):       +118us/step
  iter4 SECONDARY (L1024 residual wire):              0us/step (honest-neg)
  iter5 PRIMARY  (L1024 FP4 allgather, 2-launch):   +22us/step
  iter6 PRIMARY  (L1024 FP4 allgather, 1-launch):   +83us/step
  -------------------------------------------------------------
  cumulative sgl-project#15 deploy save at peak:              +223us/step

  TPOT projection: 25.9 ms baseline -> 25.68 ms TPOT (0.86%).

At smaller batches the iter6 win climbs to +9-10us/layer = +522 to
+580us/step alone, which would push the iter6 SAVED column above
iter4 PRIMARY. Cumulative ai-blaise main delta at the production
REAP grid still stays on the +118us iter4 PRIMARY backbone with
iter5 + iter6 stacking the L1024 incremental wins.

Correctness:
  * test_nvfp4_dp_gather_partial_fp4.py correctness pass: 100%
    bit-exact FP4 + 100% bit-exact SF preserved across all
    m_local in {1,8,16,32,64,128,256,512} x dp=2. The fused
    collective only re-orders the NCCL launch grouping, not the
    arithmetic.
  * No new Dynamo or PCG-tracing surface introduced.
  * The bench shows zero regression at m_global=256/512 (where the
    iter5 production gate falls back to BF16-only anyway); the iter6
    fused path is never invoked above the m_global=128 gate.

Recurring regression guards (verified):
  * No torch.mm(..., out_dtype=...) introduced.
  * No torch.cumsum over (dp_size,) added.
  * No is_current_stream_capturing() call added (the only stream
    interaction is pynccl_comm.change_state which is graph-capture-
    safe per the iter5 commit).
  * No CPU sync inside CUDA graph capture (all 3 allgathers + the
    group_start/end pair are pynccl-captured).
  * forward_batch.token_to_kv_pool not touched.
  * No torch.matmul, .fill_(0), .copy_() on (1, hidden_size) BF16
    shapes (the B200 SM_100 launch-config bug family).
  * compressed_tensors_w4a4_nvfp4_moe.py consumer hook unchanged --
    the stash attribute name and tuple shape are identical to the
    iter4 PRIMARY L1042 wire.
  * fallback path uses the registered custom op wrapper
    (tp.all_gather_into_tensor) for torchcomms_ncclx + non-pynccl
    deploys, preserving Dynamo-safe graph tracing on those paths.

Files:
  * python/sglang/srt/layers/dp_attention.py
    -- new dp_gather_partial_bf16_fp4_fused: pynccl group_start/end
       block spanning BF16 + FP4 + SF allgathers, with the same
       attn_tp_size==1 + max_len mode restrictions as the iter5
       dp_gather_partial_fp4. Falls back to per-leg custom-op
       allgathers when pynccl unavailable.

  * python/sglang/srt/layers/communicator.py
    -- refactor L1074..L1149: peek stash BEFORE gather; pick fused
       3-allgather (iter6) vs serial BF16 (pre-iter6) path. The
       allocation of _global_fp4 / _global_sf_storage moves above
       the gather call so the fused collective can target them
       directly. Drop-stash path consolidated into the else branch.

  * test/srt/test_nvfp4_dp_gather_partial_fp4.py
    -- bumped iters to 2000 for tighter noise; added (d) iter5 PROD
       and (e) iter6 PROD bench variants; relabeled the columns and
       added the iter5->iter6 + baseline->iter6 delta columns. The
       previous _iter5_grouped function (which already grouped all
       3 in one call) is now exposed as _iter6_all_grouped to match
       the new production wire.

IKP applicability: this iter6 vector is a host-side NCCL launch
fusion. Per-instruction kernel profiling (IKP) does not apply -- the
wins/losses are visible in the CUDA-graph captured microbench above,
which collapses the per-launch overhead into the measured delta.
IKP will be the right tool for iter7+ flashinfer
bmm_Bfloat16_E2m1E2m1_* CUBIN reconnaissance (intra-kernel SASS
attribution for fp4_quantize elimination).

Iter7 next vectors:

  PRIMARY (architectural unlock; multi-day): flashinfer
  bmm_Bfloat16_E2m1E2m1_* CUBIN launcher patch to eliminate the
  iter4 PRIMARY fp4_quantize call entirely (~200-300 us/step ceiling
  per the iter4 PRIMARY commit projection). Start with IKP
  per-instruction profile of the existing fp4_quantize call inside
  the bmm path to confirm the latency target and identify which
  instructions are amenable to in-cubin fusion. Expected target:
  scaled_fp4_quant_linear (sgl-native) wins from iter1-3 are at
  ~40us/call (m_global=128); merging into the trtllm-gen bmm would
  drop that to zero per layer.

  SECONDARY: extend dp_gather_partial_bf16_fp4_fused (and the iter5
  dp_gather_partial_fp4) to support sum_len padding for the DSA-CP
  prefill path (currently raises NotImplementedError). Reuses the
  rank-offset memcpy_triton pattern from _dp_gather_via_all_reduce
  but with an allgather instead of allreduce (allreduce sums FP4
  bytes which corrupts them). Lower priority since deploy decode
  is the bottleneck.

  TERTIARY: investigate whether the BF16 leg of the fused gather
  can be replaced by an in-place reduce_scatter + cp_all_gather
  fast-path. The current fused path issues 3 allgathers but the
  BF16 leg is the largest single byte payload (14336 B/token vs
  3584 B + 448 B for FP4 + SF). If the NVLS path could absorb the
  BF16 allgather under symmetric memory while the FP4+SF stay on
  the legacy ring path, we might reclaim another ~1-2us/layer at
  peak. Bench-only investigation; the production wire would stay
  on the iter6 fused path.

Honest negatives kept as opt-in checkpoints:

  * At m_global=128 the iter5->iter6 win is +1.44us/layer median,
    +0.59us min, +2.62us max across 3 trials. The min is at the
    noise floor (typical pynccl bench noise ~0.5us at this iter
    count) but the median is above noise. The win is real but
    not dramatic at production peak.

  * At m_global < 128 the win is +9-10us/layer. The production
    deploy decode peak is m_global=128 (batch=128 @ DP=TP=8); the
    cumulative deploy save number uses the peak figure to stay
    conservative. Smaller batches benefit more.

  * The iter6 fused path is strictly within the iter5 m_global<=128
    gate. Above the gate the wire falls back to dp_gather_partial
    BF16-only -- identical to the pre-iter5 behavior -- so the gate
    composition is conservative.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
… bmm bf16-act launcher patch plan)

Iter6 PRIMARY (2940e32) closed out with the iter7 PRIMARY vector
flagged: vendor + patch flashinfer's trtllm_fused_moe_kernel_launcher.cu
to relax the dtype gate that rejects (Bfloat16 act + E2m1 weight). The
upstream launcher accepts (Bfloat16, MxE2m1) but rejects (Bfloat16, E2m1)
even though the cubin export ships kernels for BOTH tuples. Eliminating
the host-side fp4_quantize call ceilings at ~200-300us/step (per the
iter4 PRIMARY commit projection).

This commit is the iter7 RECON CHECKPOINT (no runtime change). It writes
the catalog + plan in notes/nvfp4_moe_iter7_recon.md to make the
4-stage scaffold reviewable before code lands.

Recon findings (verified by direct inspection):

1. Cubin count: **58 Bmm_Bfloat16_E2m1E2m1_*.cubin** SM_100 variants
   ship in batched_gemm-b3c1646-c111d7c, exactly matching the iter6
   commit projection. Tile coverage:
     - M=128 fixed.
     - N in {8, 16, 32, 64, 128, 256}
     - K in {256, 512}
     - Pipeline depths s in {3, 4, 5, 6, 9}
     - CGA in {1x1x1, 2x1x1, 1x1x{2,3,4}-splitK}
     - schedS (serial) and schPd (persistent) variants
     - All ship bA16_bB16 (NVFP4 E2m1 block-scale layout, sf_vec_size=16)
       and BF16 epilogue + biasM_bN_rgTma_clmp.

   Comparison: 397 MxE2m1 variants ship (bA32_bB32, sf_vec_size=32) for
   the currently-supported (Bfloat16, MxE2m1) path. The 58 E2m1 variants
   are the iter7 unlock target.

2. Production-shape match for DSv3.2-REAP (m_global=128, H=7168,
   intermediate=4096, top_k=8, num_experts=256):
     - GEMM1 (m, 4096): tile t128x128x128, t128x128x256, t128x64x512 all
       present in 58-set. Hot persistent variant
       t128x128x256_s6_et128x128_m256x128x64_cga2x1x1 ships.
     - GEMM2 (m, 7168): same tile family, cga2 cluster + splitK1-4 covers
       low-batch tails (m_global=1..64).

3. Blocking gate: trtllm_fused_moe_kernel_launcher.cu
   FP4BlockScaleLauncher::check_moe() at L1597-1616:

       if (mDtypeAct == btg::Dtype::E2m1) {
         TVM_FFI_ICHECK(mDtypeWeights == btg::Dtype::E2m1) ...
       } else if (mDtypeAct == btg::Dtype::Bfloat16 || ...) {
         TVM_FFI_ICHECK(mDtypeWeights == btg::Dtype::MxE2m1)
             << "Only MxE2m1 weights are supported by block scale MoE
                 with Bfloat16, E4m3 or MxE4m3 activation";
       }

   The gate REJECTS (Bfloat16 act + E2m1 weight). DSv3.2-REAP ships E2m1
   weights with E4M3-bA16/bB16 group scales. The launcher mDtypeWeights
   branch (E2m1) requires mDtypeAct == E2m1, forcing the SGL deploy to
   pre-quantize via scaled_fp4_quant_linear before the call. The cubin
   for (BF16, E2m1) is sitting there, unreachable from the launcher gate.

4. Python-side mirror gate: flashinfer/fused_moe/core.py
   is_trtllm_moe_supported() at L122:
       if dtype_weights == DtypeTrtllmGen.E2m1 and dtype_act != DtypeTrtllmGen.E2m1:
           return False
   Mirrors the C++ gate. Both must relax for iter7.

5. Downstream path is clean:
     - trtllm_fused_moe_runner.cu L590:
         mGemm2(Gemm2::Runner(dtypeAct, dtypeWeights, btg::Dtype::Bfloat16, ...))
       accepts both dtypes independently; no signature-level rejection.
     - sf_vec_size at L1670:
         auto const sf_vec_size = mDtypeWeights == btg::Dtype::MxE2m1 ? 32 : 16;
       already branches correctly for the E2m1 layout (16). No runner-side
       fix needed.

Scaffold plan (4 stages, time-budgeted):

  Stage A: vendor launcher into SGL external_kernels with check_moe()
           + python gates relaxed. Keep flashinfer upstream untouched.
           Time: 2-4h. (this iter delivers vendor copy + patch DIFF)

  Stage B: bind a new TVM-FFI entry trtllm_fp4_block_scale_moe_bf16_act
           accepting BF16 hidden_states (no hidden_states_scale needed --
           cubin SF-gen happens in epilogue).
           Time: 2-3h.

  Stage C: SGL wire in compressed_tensors_w4a4_nvfp4_moe.py L343, gated
           on SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE env. Skip the entire
           scaled_fp4_quant_linear / fp4_quantize block; pass x BF16
           directly. Opt-in checkpoint pattern.
           Time: 1h.

  Stage D: standalone microbench
           test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py validates
           bit-exact end-to-end output and attributes per-layer us delta.
           Time: 1h.

  Stage E: end-to-end deploy bench validates the cumulative +200-300us/step
           save on the 58-layer DSv3.2-REAP MoE stack.
           Time: 1h.

  Total: 7-10h, well past the iter7 3-hour budget. This commit ships the
  recon catalog only; the next commit in this iter ships Stage A vendored
  scaffold + a checked-in launcher patch DIFF.

Cumulative sgl-project#15 projection at peak (unchanged from iter6 close-out):
  iter1-3 foundation
  iter4 PRIMARY:  +118us/step
  iter5 PRIMARY:   +22us/step
  iter6 PRIMARY:   +83us/step
  -------------------------------
  cumulative:    +223us/step (TPOT 25.9 -> 25.68 ms)

If iter7 lands +200us/step on top: cumulative -> +423us/step at peak
(TPOT 25.48 ms, 1.62% improvement). At smaller decode batches the win
climbs proportionally to the relative fp4_quantize cost in the layer.

Honest open questions:
  * The bmm epilogue may require a per-token scale tensor input even for
    BF16-act mode. To confirm in Stage A by inspecting KernelParams.h
    of the bmm export.
  * Output bit-exactness vs the current scaled_fp4_quant_linear path is
    the iter8 wire acceptance gate. The cubin in-epilogue SF-gen may
    differ from the host-side sgl-native quantize by +/-1 ULP at FP4
    quantize boundaries; opt-in gate stays on until bench resolves.
  * Cubin selector indexes by (mDtypeAct, mDtypeWt) tuple; if the
    existing trtllmGen BatchedGemmInterface only maps (E2m1, E2m1) and
    (Bfloat16, MxE2m1) entries for FP4 MoE, the map needs extending in
    Stage A. (Bmm cubin path mirroring suggests it does map, but to be
    verified.)

Recurring regression guards (this commit changes no runtime code):
  * forward_batch.token_to_kv_pool not touched.
  * No torch.mm(..., out_dtype=...).
  * No torch.cumsum.
  * No is_current_stream_capturing() in PCG-traced paths.
  * No torch.matmul on (1, hidden_size) BF16 (B200 SM_100 launch bug).
  * compressed_tensors_w4a4_nvfp4_moe.py untouched (Stage C wire only).

Files: notes/nvfp4_moe_iter7_recon.md  (189 lines; full catalog + plan)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SpencerGarnets added a commit to BlaiseSI/optimization-playground that referenced this pull request May 30, 2026
… BF16-act bmm launcher patch + recon

Iter7 PRIMARY vector: vendor + patch flashinfer's
trtllm_fused_moe_kernel_launcher.cu to relax the gate that rejects
(Bfloat16 act + E2m1 weight) and unlock the 58 in-cubin
Bmm_Bfloat16_E2m1E2m1_*.cubin SM_100 variants. Recon checkpoint
landed in the prior commit (48806a9). This commit ships the
**Stage A scaffold**: patch files, env-var declaration, vendored
scaffold module, and microbench template. No runtime change yet.

The full 4-stage plan in notes/nvfp4_moe_iter7_recon.md targets
7-10 hours total wall; iter7 budget is 3 hours. Stage A lands here;
Stages B (vendored build glue), C (SGL wire), D (microbench), E
(end-to-end deploy bench) land in iter8+ when the time budget
opens.

## Patches

### 0001-relax-bf16-act-e2m1-weight.patch (FP4BlockScaleLauncher)

  +12 -3 lines on csrc/trtllm_fused_moe_kernel_launcher.cu

Relaxes check_moe() to accept:
  * (E2m1 act, E2m1 weight) — current production path
  * (E2m1 act, MxE2m1 weight) — newly accepted (was rejected with the
    SAME error message that already permitted MxE2m1 with BF16; this
    is the symmetric branch)
  * (Bfloat16 act, MxE2m1 weight) — current production path
  * (Bfloat16 act, E2m1 weight) — NEWLY UNLOCKED for DSv3.2-REAP
  * (E4m3 / MxE4m3 act, MxE2m1 weight) — unchanged

Verified clean apply against the pinned flashinfer source via
`git apply --check`.

### 0002-python-mirror.patch (is_trtllm_moe_supported)

  +8 -1 lines on flashinfer/fused_moe/core.py

Mirrors the C++ relaxation in the python dispatch guard. Without
this, the launcher patch would still be reachable but the python
side would return False before reaching dispatch.

Verified clean apply.

## Env-var declaration

SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE = EnvBool(False)

Defaults False. Opt-in only. When True AND the vendored launcher
exports the patched symbol (Stage B), the SGL deploy switches to
the BF16-act fused path; otherwise falls through to the iter1-3 +
iter4 PRIMARY production path. Matches the iter4 SECONDARY /
iter5 SECONDARY opt-in checkpoint discipline.

## Scaffold module

python/sglang/srt/external_kernels/flashinfer/__init__.py

Exports HAS_BF16_ACT_BMM=False until Stage B. Smoke-tested:
the import resolves and returns the False placeholder. Future
iter8 wire reads this flag at startup to decide gate eligibility.

## Microbench template

test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py

Skeleton with the reference vs candidate path call sites elided
(NotImplementedError) and the bench loop placeholder. The script
runs today and prints the iter7 status message; iter8 Stage D
fills in the reference + candidate bench bodies once the
candidate symbol exists.

## Recon recap (verified findings)

Cubin catalog (from prior commit 48806a9):
  * 58 Bmm_Bfloat16_E2m1E2m1_Fp32_bA16_bB16_*.cubin variants ship in
    batched_gemm-b3c1646-c111d7c (SM_100, BF16 epilogue, biasM_bN_
    rgTma_clmp, dynamic-batch + cga 1x1x1 / 2x1x1 / 1x1x{2,3,4}-splitK).
  * Comparison: 397 MxE2m1 (bA32_bB32) variants ship for the
    currently-supported (BF16, MxE2m1) path. The 58 (bA16, bB16) E2m1
    variants are the iter7 unlock target.
  * Tile coverage matches DSv3.2-REAP production shape:
    M=128, N in {8,16,32,64,128,256}, K in {256,512}, persistent+serial
    schedulers both present.

Blocking gate (confirmed by direct inspection):
  * trtllm_fused_moe_kernel_launcher.cu L1610-1616:
      else if (mDtypeAct == Bfloat16 || E4m3 || MxE4m3):
        TVM_FFI_ICHECK(mDtypeWeights == MxE2m1)
    REJECTS (BF16, E2m1). Patch 0001 inserts the E2m1-weight branch.
  * fused_moe/core.py L122 mirrors the same rejection on the python
    side. Patch 0002 mirrors the relaxation.

Downstream path clean (no runner-side change needed):
  * trtllm_fused_moe_runner.cu Gemm2::Runner ctor at L587-590 accepts
    (dtypeAct, dtypeWeights) independently.
  * launcher prepare_moe() L1670 already branches sf_vec_size correctly:
      sf_vec_size = (mDtypeWeights == MxE2m1) ? 32 : 16;
    The E2m1 layout (sf_vec_size=16) is the DSv3.2-REAP NVFP4 layout.
  * Entry function trtllm_fp4_block_scale_moe at L2145+ already
    determines mDtypeAct from hidden_states.dtype() (BF16 -> Bfloat16)
    and mDtypeWeights from weight_scale_vec_size (16 -> E2m1). The
    combo is achievable today AT THE ENTRY but rejected at check_moe().

## Expected wins (unchanged from iter4 PRIMARY projection)

  Per-call: scaled_fp4_quant_linear elides ~40us at m_global=128.
  Per-step: 58 NVFP4 MoE layers * (40us - bmm-epilogue-SF-gen-cost).
  Net target: 200-300us/step (the bmm in-cubin path pays back some
  of the saved 40us for fused SF-gen in the epilogue).

  Cumulative sgl-project#15 deploy projection at peak after iter7 lands:
    iter1-3 foundation
    iter4 PRIMARY:  +118us/step
    iter5 PRIMARY:   +22us/step
    iter6 PRIMARY:   +83us/step
    iter7 PRIMARY:  +200..+300us/step (projected, scaffold-checkpoint)
    -------------------------------
    cumulative:     +423..+523us/step at peak
    TPOT:           25.9 ms -> 25.38..25.48 ms (1.62..2.01%)

  At smaller decode batches (m_global=64) the relative
  fp4_quantize cost climbs, pushing potential wins higher.

## Honest negatives / open questions

1. Stage B requires building the vendored .so via the same TVM-FFI
   toolchain flashinfer uses. flashinfer.jit.compile_aot is the
   entry; reproducing its include + flag stack from outside the
   wheel is the main Stage B risk. If reproducing the toolchain
   slips, the fallback is symlinking flashinfer's installed
   sources, applying patches in-place at SGL import time, and
   rebuilding via flashinfer's own compile pipeline (intrusive but
   feasible).

2. Bit-exactness vs the existing scaled_fp4_quant_linear path is
   the iter8 Stage D acceptance gate. The cubin in-epilogue SF-gen
   may differ from the host-side sgl-native quantize by +/-1 ULP
   at FP4 quantize boundaries. The opt-in env-var gate stays on
   until the bench resolves this.

3. The cubin selector behavior under (BF16 act, E2m1 weight) is
   unverified at runtime — Stage B is the first chance to exercise
   it. If the trtllmGen BatchedGemmInterface kernel-map only
   indexes (E2m1, E2m1) and (BF16, MxE2m1) entries for FP4 MoE and
   does not auto-discover the (BF16, E2m1) entry, the map needs
   extending (also in patches/ as iter7+1).

## Recurring regression guards (this commit changes no runtime code)

  * forward_batch.token_to_kv_pool not touched.
  * No torch.mm(..., out_dtype=...).
  * No torch.cumsum.
  * No is_current_stream_capturing() in PCG-traced paths.
  * No torch.matmul on (1, hidden_size) BF16 (B200 SM_100 launch bug).
  * compressed_tensors_w4a4_nvfp4_moe.py untouched (Stage C wire only).
  * environ.py change is purely additive (new var, EnvBool default
    False, no existing var renamed or removed).

## Files

ADD:
  python/sglang/srt/external_kernels/__init__.py
  python/sglang/srt/external_kernels/README.md
  python/sglang/srt/external_kernels/flashinfer/__init__.py
  python/sglang/srt/external_kernels/flashinfer/csrc/  (empty, Stage B fills)
  python/sglang/srt/external_kernels/flashinfer/patches/0001-relax-bf16-act-e2m1-weight.patch
  python/sglang/srt/external_kernels/flashinfer/patches/0002-python-mirror.patch
  test/srt/test_nvfp4_moe_bmm_bf16_act_bench.py

MODIFY:
  python/sglang/srt/environ.py  (+11 lines: SGLANG_USE_TRTLLM_BF16_ACT_FP4_MOE)

Both patches verified clean against the pinned flashinfer source
via `git apply --check` on a stripped-down repo seeded from the
installed wheel.

## Iter7 budget honesty

  Stage A (this commit):     ~3 hours wall — DONE, in budget
  Stage B (vendor build):    2-4 hours — iter8 PRIMARY
  Stage C (SGL wire):        1 hour    — iter8 PRIMARY
  Stage D (microbench):      1 hour    — iter8 PRIMARY
  Stage E (deploy bench):    1 hour    — iter8 SECONDARY

  No runtime change in iter7. The iter6 PRIMARY (2940e32) NCCL
  launch fusion remains the production wire; cumulative +223us/step
  unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 2, 2026
…/forensic

Round-15 audit found 16 items. All applied. Most consequential is
replacing the round-14-part-2 'assert feasible' with a fatal()
forensic-dump halt + new §10 two-fault-class invariant.

MAJOR fixes:
- #1 joint_decide infeasibility: was assert (round-14 part 2). Now
  fatal('joint_decide_infeasible', ...) writes structured JSON to
  <daemon-data>/forensic/joint_decide_infeasible_<ts>.json
  capturing event, state snapshot, candidates, DP inputs, traceback,
  then sys.exit(1). User mandate: deployment-bug-class faults
  expose themselves with full forensic context; load-class faults
  absorb. The fault classification is now an explicit §10 invariant.
- #2 unavailability_cost signature fix: was transfer_time(add_tiers,
  u, state) — wrong signature (transfer_time is per σ→τ pair).
  Rewrote as sum over add_tiers of p_hat × P(serve fails) × reload.
  Caller passes state correctly.
- #3 bw_free cold-start: pinned time_since_last_sample_s = +Inf at
  boot (trips the > LINK_IDLE_SECONDS branch → returns peak_bw_bps,
  the correct value for an unused link).
- #4 multi-rank throughput aggregation: added per-field aggregation
  rule table to §6 — sum for byte fields, sum for link bandwidth
  (each rank has own slice), MEAN for throughput EMAs (same logical
  work observed N×), max for time_since_last_sample_s, identical
  for static config + n_bytes + subpool keys.
- sgl-project#5 cross-rank subpool key consistency: explicit fatal() on
  disagreement.

CLARITY:
- sgl-project#6 PLAN.md drift: removed 'PUT updates cache file atomically' (cache
  was dropped in round-14), removed 'feasibility-fallback' (now
  fatal), 'samples_in_window' → 'time_since_last_sample_s'.
- sgl-project#7 forecast_horizon: explicit branch for recent_event_rate=0 cold-
  start (returns heartbeat_s).
- sgl-project#8 mean_unit_bytes empty-HBM guard: short-circuit returns ∞ to
  collapse top_k_pressure to K_MIN.
- sgl-project#9 peak_bw_bps > 0 / h_max > 0 / prefill_bps > 0 as deployment-bug
  invariant (§10 Physical inputs); fatal() on violation.
- sgl-project#10 APPLY_FAILED unknown batch_id: explicit silent drop (load-class,
  not deployment-bug); next state-fetch reconverges via always-fresh
  + idempotent invariants.
- sgl-project#11 program state intermediates: explicit '4 states are exhaustive,
  intermediate states collapsed at dump time' in §5 schema comment.
- sgl-project#12 'policy is not learned': softened 'in scope as an extension' →
  'neither in scope nor ruled out'; consistent with paper which
  also dropped the PPO/DQN paragraph.
- sgl-project#14 §12 S4 axis count: 7 → 9 (3 HBM + 3 DRAM-cap + 3 DISK-cap
  axes; the subtractive arithmetic in the original was wrong).
- sgl-project#15 §12 scenario composition: explicit note that S1-S5 compose
  by subpool-key union; S2+S5 = {full, swa, draft}, S3+S5 =
  {full, mamba, draft}, etc.

PLAN.md additions:
- Daemon impl item 11: fatal(reason, **context) helper as the
  shared entry point for every deployment-bug-class halt;
  enumerates the call sites.

Items not surfacing as a doc change:
- sgl-project#13 forecast recompute overhead: noted in audit but the spec
  already says always-fresh covers this; no edit needed.
- sgl-project#16 meta-invariant deployment-bug vs load-class: now covered by
  the new §10 'Two fault classes' invariant (above).
shivajid referenced this pull request Jun 29, 2026
Added error log indicating that FlashInfer allreduce fusion requires specific NVIDIA GPUs.
@JD-ETH JD-ETH mentioned this pull request Jul 15, 2026
1 of 2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants