Skip to content

Fix MXFP4 expert scales silently zeroing on per-expert checkpoints - #33541

Closed
Hakureirm wants to merge 2 commits into
sgl-project:mainfrom
Hakureirm:mxfp4-deepseek-per-expert-checkpoints
Closed

Hakureirm wants to merge 2 commits into
sgl-project:mainfrom
Hakureirm:mxfp4-deepseek-per-expert-checkpoints

Conversation

@Hakureirm

@Hakureirm Hakureirm commented Aug 4, 2026 •

Copy link
Copy Markdown

Serving profile after the fixes: prefill, prefix cache, concurrency

Motivation

MXFP4 support in SGLang has only ever been exercised by GPT-OSS checkpoints, which pack all experts into a single tensor per projection. Those take the fused fast path in FusedMoE.weight_loader and never touch the per-expert loading path.

DeepSeek ships one tensor per expert. Loading such a checkpoint walks that path and hits six distinct problems in sequence — each only reachable after the previous one is fixed. Most of them fail loudly, but the last three do not: every expert block scale ends up zeroed at load time, the model starts, serves, and emits fluent text — it just stops following the prompt. Nothing in the logs reports a problem except one line in the routine "weights not initialized" list.

Modifications

# Symptom Fix
1 Load aborts: assertion on a Linear, NotImplementedError on attention An MXFP4 checkpoint only serializes experts; default everything else to unquantized instead of failing the load
2 Expert 0's weights go down the fused path not expert_id → expert_id is None
3 Every expert scale silently unloaded (all 43 MoE layers) The DeepSeek name remap rewrites a trailing .scale to .weight_scale_inv, which is the fp8 block-scale parameter (fp8 scales are reciprocals). MXFP4 scales are not, and their buffer is weight_scale, so the remapped name matched no parameter. Pick the suffix from the quantization method
4 quant method must be one of [...] on the first scale Mxfp4MoEMethod never declared the quant_method weight attribute the generic scale loader dispatches on. MXFP4 is a 32-element block scale: declare BLOCK
5 Scales load, but every value becomes 0 e8m0 scales copied into a uint8 buffer convert numerically rather than reinterpreting — a scale byte of 120 means 2^-7, so it lands as 0. View the bits instead
6 Speculative decoding loses most of its speedup The DSpark draft model carries its own copy of the name remap with the same hardcoded fp8 suffix. Zeroed draft scales do not corrupt output — the target verifies every draft — they just wreck the acceptance rate

Also: the Marlin MoE path was gated on SM90/SM120, but Marlin targets SM80+ and its e2m1 decode is plain bit manipulation. Gated on the Marlin floor instead, so Ampere can use it.

GPT-OSS checkpoints should be unaffected: they enter through the fused
all-experts-at-once path, and the scale-name and dispatch changes live on the
per-expert path they never take. I have no GPT-OSS hardware to confirm that,
so it is reasoning about the code, not a measurement.

One caveat worth flagging: fix #1 turns a hard load failure into an
unquantized fallback. That is right for an MXFP4 checkpoint, where only the
experts are serialized in fp4 -- but it does mean a genuinely malformed
checkpoint now loads quietly instead of aborting.

Accuracy Tests

DeepSeek-V4-Flash-0731 on 8x A800 (SM80, TP=8), converted to MXFP4 with the expert bytes passed through unchanged. The official checkpoint already stores routed experts as e2m1 nibbles with e8m0 block scales, so the conversion is a lossless repack and any accuracy loss would come from the serving path alone.

Needle-in-a-haystack, 3 context lengths x 3 needle depths, 4 keyword checks
per cell (server launched with --context-length 1000000; the longest context
actually exercised anywhere in this PR is 218K):

context depth 0.1 depth 0.5 depth 0.9
4K 4/4 4/4 4/4
32K 4/4 4/4 4/4
128K 4/4 4/4 4/4

GSM8K, full 1319-question test set, greedy, thinking disabled, graded by
numeric match on the last number in the answer:

accuracy tokens/question
this PR, repacked MXFP4 on 8x A800 94.24% (1243/1319) 206
DeepSeek official API, same model, same questions, same grader 96.29% (1270/1319) 129

The 2.05-point gap is larger than sampling noise (~0.6 points at this n), so it
is reported as measured rather than claimed as parity. It is unlikely to come
from the weights: routed experts are byte-identical to the published
checkpoint and the FP8 layers are dequantised exactly. The more likely source
is prompt formatting — this checkpoint carries no chat template, so SGLang
falls back to a plain string format while the official endpoint applies
DeepSeek's own, and our answers do run notably longer (206 vs 129 tokens).
Worth chasing separately; it is not a property of these fixes.

Before these fixes the same checkpoint scored 0/9 on the same grid — every
cell missed, while the model answered fluently and nothing errored.

Speed Tests and Profiling

All on 8x A800-SXM4-80GB (NVLink, TP=8), Marlin MoE backend, server launched
with --context-length 1000000.

The tables below use different --cuda-graph-max-bs values (16, 32 and 128),
noted per table. Single-stream numbers are therefore not comparable across
tables
— 93.9, 97.1 and 94 all mean "one request at a time" under three
different capture sizes.

Single stream, before vs after. These changes are load-time only, so the
first pair is run-to-run noise. The second pair is the draft model finally
producing usable drafts once its scales load:

decode
no speculative, before and after ~50 tok/s, unchanged
before, DSpark speculative (default gamma=5) 58.1 tok/s
after, DSpark speculative (default gamma=5) 85.9 tok/s

Tuning the DSpark block size on top of that is a separate, orthogonal +9%
(85.9 -> 93.9 at gamma=3); it is not part of what these fixes buy.

DSpark block size (256 output tokens, --cuda-graph-max-bs 32). Every
drafted token costs a full expert pass, so throughput peaks well before the
accepted length does. Accepted length rises with the block up to gamma=5 and
then drops at gamma=7 — that reversal is unexplained and worth a second look:

gamma (draft tokens) 2 (3) 3 (4) 4 (5) 5 (6, default) 7 (8)
tok/s 86.4 93.9 89.6 85.9 86.0
accepted length 2.08 2.45 2.75 3.00 2.25

Prefill and prefix cache (gamma=3, --cuda-graph-max-bs 16):

context cold prefill prefill rate same prompt again decode after
16K 5.1 s 3.3K tok/s 0.29 s 97.1 tok/s
64K 23.5 s 2.9K tok/s 0.41 s 81.0 tok/s
128K 42.3 s 3.2K tok/s 0.70 s 57.9 tok/s
218K 63.7 s 3.6K tok/s 0.89 s 36.6 tok/s

Prefix caching is what makes long context usable here: re-sending a 218K-token
prompt costs 0.89 s instead of 63.7 s.

Concurrency scaling (gamma=3, --cuda-graph-max-bs 128 so the graphs cover
the whole range):

concurrency 1 8 16 32 128
aggregate tok/s 94 310 354 381 429
per-stream tok/s 94 40 23 12 3.5

Aggregate is still rising at 128, so the practical ceiling is per-stream
latency rather than throughput. Note the capture size is a real trade-off: at
128 the graph pool leaves too little room for a 200K-token prefill and the
server OOMs mid-request. Size it to the concurrency actually served.

Reproducing

The checkpoint is a repack of deepseek-ai/DeepSeek-V4-Flash-0731, produced by
benchmark/deepseek_v4/repack_mxfp4.py in this PR (expert bytes copied
through unchanged, FP8 layers dequantised to BF16, four self-checks run during
conversion). Serving command:

python3 -m sglang.launch_server \
  --model-path <repacked> --tp 8 \
  --context-length 1000000 --mem-fraction-static 0.78 \
  --attention-backend triton --moe-runner-backend marlin \
  --cuda-graph-max-bs 16 --chunked-prefill-size 16384 \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
  --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 \
  --trust-remote-code

gamma in the block-size table is --speculative-dspark-block-size; the value
in parentheses is the resulting verify window (gamma + 1).

Checklist

GPT-OSS packs all experts into a single tensor per projection, so its
weights take the fused fast path in FusedMoE.weight_loader and never
exercise the per-expert loader. DeepSeek ships one tensor per expert,
which walks that path and hits five separate problems -- each one only
reachable after the previous is fixed:

1. Any Linear not on the ignore list fell through to `return None`,
   which the layer asserts against, and an attention module raised
   NotImplementedError outright. An MXFP4 checkpoint only serializes
   the experts; everything else is plain bf16, so default those to
   unquantized instead of failing the load.

2. `not expert_id` treated expert 0 as "all experts", sending its
   weights down the fused path. Test for None explicitly.

3. The DeepSeek name remap rewrites a trailing `.scale` to
   `.weight_scale_inv`, which is the fp8 block-scale parameter (those
   scales are reciprocals). MXFP4 scales are not, and their buffer is
   `weight_scale`, so the remapped name matched no parameter and every
   expert scale across all 43 MoE layers stayed at its zero-init --
   reported only as a line in the "weights not initialized" list.
   Pick the suffix from the quantization method.

4. `Mxfp4MoEMethod` never set the `quant_method` weight attribute the
   generic scale loader dispatches on, so once the names matched, the
   first expert scale raised "quant method must be one of [...]".
   MXFP4 is a 32-element block scale: declare BLOCK.

5. Block scales stored as e8m0 were copied into the uint8 parameter
   buffer, which converts numerically rather than reinterpreting: a
   scale byte of 120 means 2^-7, so it lands as 0. Every block scale
   silently became zero, the experts produced zeros, and the model
   emitted fluent but unrelated text with no error anywhere. View the
   bits instead.

The Marlin MoE path was also gated on SM90/SM120 even though Marlin
itself targets SM80+ and its e2m1 decode is plain bit manipulation;
gate it on the Marlin floor so Ampere can use it.

Verified on 8x A800 (SM80, TP8) with DeepSeek-V4-Flash-0731 converted
to MXFP4 with the expert bytes passed through unchanged: needle-in-a-
haystack 12/12 (4K/32K/128K x depth 0.1/0.5/0.9), 50.2 tok/s
single-stream decode at a 1M context ceiling. Before these fixes the
same checkpoint scored 0/12.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@Hakureirm Hakureirm changed the title Load MXFP4 checkpoints that ship one tensor per expert Fix MXFP4 expert scales silently zeroing on per-expert checkpoints Aug 4, 2026
@Hakureirm
Hakureirm marked this pull request as draft August 4, 2026 11:09
@Hakureirm
Hakureirm marked this pull request as ready for review August 4, 2026 17:34
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

The draft model carries its own copy of the DeepSeek name remap, with
the same hardcoded fp8 suffix. On an mxfp4 checkpoint that silently
zeroes every draft expert scale -- which does not corrupt output, since
the target model verifies each draft, but it wrecks the acceptance rate
and most of the speedup with it.

Measured on 8x A800 with DeepSeek-V4-Flash-0731 MXFP4, single stream at
a 1M context ceiling: 58.1 -> 85.9 tok/s.
@Hakureirm
Hakureirm force-pushed the mxfp4-deepseek-per-expert-checkpoints branch from 0e368ce to 290ccba Compare August 4, 2026 18:14
@yhyang201 yhyang201 self-assigned this Aug 4, 2026
@Hakureirm

Copy link
Copy Markdown
Author

cc @mmangkad @BBuf — you're the two most frequent authors of mxfp4.py, and the two changes there (the quant_method weight attribute for the per-expert scale loader, and scoping it to the scale params rather than all of them) are in your area.

The change I'd most like checked is in deepseek_v4.py: the name remap rewrote a trailing .scale to .weight_scale_inv unconditionally. That is correct for fp8 block scales, which are reciprocals, but MXFP4 scales are not and their buffer is weight_scale — so on an mxfp4 checkpoint the remapped name matched no parameter and every expert scale silently stayed zero-initialised. I made the suffix depend on quant_config.get_name(). If the unconditional form was load-bearing for a case I haven't hit, I'd rather find out now than after this lands.

@Hakureirm

Copy link
Copy Markdown
Author

Closing this in favour of a much smaller change.

I went back and checked the assumption this PR was built on — that an Ampere card has no path to DeepSeek's fp4 experts and therefore needs the checkpoint repacked into the generic MXFP4 layout. That assumption was wrong. SGLang already auto-detects is_fp4_experts for DeepSeek-V4 and routes it to Mxfp4MarlinMoEMethod, which is a Marlin kernel and has no architectural reason to require SM90. The only thing standing in the way is a gate in mxfp4_marlin_moe.py.

Tested just now: the unmodified deepseek-ai/DeepSeek-V4-Flash-0731 checkpoint, with that single gate relaxed and nothing else changed, serves correctly on 8x A800 — needle 6/6, 51.9 tok/s, which matches what the repacked checkpoint did (50.2). So the repack script and the five loading fixes here are all solving a problem that does not need solving.

The one part of this PR that survives on its own merits is the e8m0 observation: copying a float8_e8m0fnu scale tensor into a uint8 buffer converts numerically rather than reinterpreting, so a scale byte of 120 (meaning 2^-7) lands as 0. That is a real trap for anyone wiring up a new MXFP4 loading path, but it is not reachable on any path in use today, so it does not need a patch.

Thanks to whoever started reading this. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants