Repository navigation
Conversation
GPT-OSS packs all experts into a single tensor per projection, so its weights take the fused fast path in FusedMoE.weight_loader and never exercise the per-expert loader. DeepSeek ships one tensor per expert, which walks that path and hits five separate problems -- each one only reachable after the previous is fixed: 1. Any Linear not on the ignore list fell through to `return None`, which the layer asserts against, and an attention module raised NotImplementedError outright. An MXFP4 checkpoint only serializes the experts; everything else is plain bf16, so default those to unquantized instead of failing the load. 2. `not expert_id` treated expert 0 as "all experts", sending its weights down the fused path. Test for None explicitly. 3. The DeepSeek name remap rewrites a trailing `.scale` to `.weight_scale_inv`, which is the fp8 block-scale parameter (those scales are reciprocals). MXFP4 scales are not, and their buffer is `weight_scale`, so the remapped name matched no parameter and every expert scale across all 43 MoE layers stayed at its zero-init -- reported only as a line in the "weights not initialized" list. Pick the suffix from the quantization method. 4. `Mxfp4MoEMethod` never set the `quant_method` weight attribute the generic scale loader dispatches on, so once the names matched, the first expert scale raised "quant method must be one of [...]". MXFP4 is a 32-element block scale: declare BLOCK. 5. Block scales stored as e8m0 were copied into the uint8 parameter buffer, which converts numerically rather than reinterpreting: a scale byte of 120 means 2^-7, so it lands as 0. Every block scale silently became zero, the experts produced zeros, and the model emitted fluent but unrelated text with no error anywhere. View the bits instead. The Marlin MoE path was also gated on SM90/SM120 even though Marlin itself targets SM80+ and its e2m1 decode is plain bit manipulation; gate it on the Marlin floor so Ampere can use it. Verified on 8x A800 (SM80, TP8) with DeepSeek-V4-Flash-0731 converted to MXFP4 with the expert bytes passed through unchanged: needle-in-a- haystack 12/12 (4K/32K/128K x depth 0.1/0.5/0.9), 50.2 tok/s single-stream decode at a 1M context ceiling. Before these fixes the same checkpoint scored 0/12.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
The draft model carries its own copy of the DeepSeek name remap, with the same hardcoded fp8 suffix. On an mxfp4 checkpoint that silently zeroes every draft expert scale -- which does not corrupt output, since the target model verifies each draft, but it wrecks the acceptance rate and most of the speedup with it. Measured on 8x A800 with DeepSeek-V4-Flash-0731 MXFP4, single stream at a 1M context ceiling: 58.1 -> 85.9 tok/s.
0e368ce to
290ccba
Compare
|
cc @mmangkad @BBuf — you're the two most frequent authors of The change I'd most like checked is in |
|
Closing this in favour of a much smaller change. I went back and checked the assumption this PR was built on — that an Ampere card has no path to DeepSeek's fp4 experts and therefore needs the checkpoint repacked into the generic MXFP4 layout. That assumption was wrong. SGLang already auto-detects Tested just now: the unmodified The one part of this PR that survives on its own merits is the e8m0 observation: copying a Thanks to whoever started reading this. Sorry for the noise. |
Motivation
MXFP4 support in SGLang has only ever been exercised by GPT-OSS checkpoints, which pack all experts into a single tensor per projection. Those take the fused fast path in
FusedMoE.weight_loaderand never touch the per-expert loading path.DeepSeek ships one tensor per expert. Loading such a checkpoint walks that path and hits six distinct problems in sequence — each only reachable after the previous one is fixed. Most of them fail loudly, but the last three do not: every expert block scale ends up zeroed at load time, the model starts, serves, and emits fluent text — it just stops following the prompt. Nothing in the logs reports a problem except one line in the routine "weights not initialized" list.
Modifications
NotImplementedErroron attentionnot expert_id→expert_id is None.scaleto.weight_scale_inv, which is the fp8 block-scale parameter (fp8 scales are reciprocals). MXFP4 scales are not, and their buffer isweight_scale, so the remapped name matched no parameter. Pick the suffix from the quantization methodquant method must be one of [...]on the first scaleMxfp4MoEMethodnever declared thequant_methodweight attribute the generic scale loader dispatches on. MXFP4 is a 32-element block scale: declareBLOCKAlso: the Marlin MoE path was gated on SM90/SM120, but Marlin targets SM80+ and its e2m1 decode is plain bit manipulation. Gated on the Marlin floor instead, so Ampere can use it.
GPT-OSS checkpoints should be unaffected: they enter through the fused
all-experts-at-once path, and the scale-name and dispatch changes live on the
per-expert path they never take. I have no GPT-OSS hardware to confirm that,
so it is reasoning about the code, not a measurement.
One caveat worth flagging: fix #1 turns a hard load failure into an
unquantized fallback. That is right for an MXFP4 checkpoint, where only the
experts are serialized in fp4 -- but it does mean a genuinely malformed
checkpoint now loads quietly instead of aborting.
Accuracy Tests
DeepSeek-V4-Flash-0731 on 8x A800 (SM80, TP=8), converted to MXFP4 with the expert bytes passed through unchanged. The official checkpoint already stores routed experts as e2m1 nibbles with e8m0 block scales, so the conversion is a lossless repack and any accuracy loss would come from the serving path alone.
Needle-in-a-haystack, 3 context lengths x 3 needle depths, 4 keyword checks
per cell (server launched with
--context-length 1000000; the longest contextactually exercised anywhere in this PR is 218K):
GSM8K, full 1319-question test set, greedy, thinking disabled, graded by
numeric match on the last number in the answer:
The 2.05-point gap is larger than sampling noise (~0.6 points at this n), so it
is reported as measured rather than claimed as parity. It is unlikely to come
from the weights: routed experts are byte-identical to the published
checkpoint and the FP8 layers are dequantised exactly. The more likely source
is prompt formatting — this checkpoint carries no chat template, so SGLang
falls back to a plain string format while the official endpoint applies
DeepSeek's own, and our answers do run notably longer (206 vs 129 tokens).
Worth chasing separately; it is not a property of these fixes.
Before these fixes the same checkpoint scored 0/9 on the same grid — every
cell missed, while the model answered fluently and nothing errored.
Speed Tests and Profiling
All on 8x A800-SXM4-80GB (NVLink, TP=8), Marlin MoE backend, server launched
with
--context-length 1000000.The tables below use different
--cuda-graph-max-bsvalues (16, 32 and 128),noted per table. Single-stream numbers are therefore not comparable across
tables — 93.9, 97.1 and 94 all mean "one request at a time" under three
different capture sizes.
Single stream, before vs after. These changes are load-time only, so the
first pair is run-to-run noise. The second pair is the draft model finally
producing usable drafts once its scales load:
Tuning the DSpark block size on top of that is a separate, orthogonal +9%
(85.9 -> 93.9 at gamma=3); it is not part of what these fixes buy.
DSpark block size (256 output tokens,
--cuda-graph-max-bs 32). Everydrafted token costs a full expert pass, so throughput peaks well before the
accepted length does. Accepted length rises with the block up to gamma=5 and
then drops at gamma=7 — that reversal is unexplained and worth a second look:
Prefill and prefix cache (gamma=3,
--cuda-graph-max-bs 16):Prefix caching is what makes long context usable here: re-sending a 218K-token
prompt costs 0.89 s instead of 63.7 s.
Concurrency scaling (gamma=3,
--cuda-graph-max-bs 128so the graphs coverthe whole range):
Aggregate is still rising at 128, so the practical ceiling is per-stream
latency rather than throughput. Note the capture size is a real trade-off: at
128 the graph pool leaves too little room for a 200K-token prefill and the
server OOMs mid-request. Size it to the concurrency actually served.
Reproducing
The checkpoint is a repack of
deepseek-ai/DeepSeek-V4-Flash-0731, produced bybenchmark/deepseek_v4/repack_mxfp4.pyin this PR (expert bytes copiedthrough unchanged, FP8 layers dequantised to BF16, four self-checks run during
conversion). Serving command:
gammain the block-size table is--speculative-dspark-block-size; the valuein parentheses is the resulting verify window (gamma + 1).
Checklist