Skip to content

Make DeepSeek-V4 serve on SM80: compile, dispatch, kernel, and the indexer's oversized intermediate - #33271

Closed
Hakureirm wants to merge 23 commits into
sgl-project:mainfrom
Hakureirm:sm80-dsv4-sparse-decode
Closed

Hakureirm wants to merge 23 commits into
sgl-project:mainfrom
Hakureirm:sm80-dsv4-sparse-decode

Conversation

@Hakureirm

@Hakureirm Hakureirm commented Aug 2, 2026 •

Copy link
Copy Markdown

Needs the run-ci label for any CI signal; I cannot add one.

Making DeepSeek-V4 serve on SM80 - and keeping the answers

Summary

DeepSeek-V4 cannot serve on SM80. Three independent blockers, all fixed here:

  1. Does not compile. dequantize_k_cache_paged and the Triton sparse decode kernel carry an fp8e4nv pointer in their signature; Triton on SM80 accepts only fp8e4b15 and fp8e5.
  2. Does not dispatch. Three sites select the FlashMLA CUDA kernels via is_sm120_supported(). Those kernels are arch-conditional builds for SM90a and the SM100 family (see kernels/aot/cmake/flashmla.cmake), so every other architecture walks into an import sgl_kernel.flash_mla that cannot succeed.
  3. Does not load a packed checkpoint. deepseek_v2.py classifies layers by matching quant_config.get_name() against {awq, awq_marlin, moe_wna16}, then reads .weight.dtype. An auto_round W4A16 checkpoint raises:
File ".../sglang/srt/models/deepseek_v2.py", line 783, in __init__
    and self.shared_experts.gate_up_proj.weight.dtype == torch.int8
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'. Did you mean: 'qweight'?

Past those, the Triton fallback is nominal rather than usable: it launched with grid=(B, H), so all 64 query heads re-read the same KV — MLA has one KV head, so that is a 64x amplification of the dominant traffic. The C4 indexer separately materialises a [B, S, num_heads] intermediate and scores each row once per head.

Modifications

area change
dispatch route by whether the FlashMLA kernels exist, not by SM120; the SM80 fallback's own backend choice and the indexer's torch routing now default by capability too
decode kernel one program per BLOCK_H heads, KV read once, reused via tl.dot; bit-arithmetic E4M3 with the scale folded into the exponent field; int32 loads
dequant 256-entry table so it compiles on SM80 — exact, not approximate
indexer chunk logits along the sequence axis; gather KV once when rows share a page table
loading classify packed layers by what was built, not by quantization name
dspark anchor the draft weight-name rename at the suffix: packed checkpoints store .scales, which the substring replace turned into .weight_scale_invs — dropped with a warning, leaving the draft on uninitialised weights and the speculative accept rate silently at zero
tests un-gate the portable cases from compute capability 12.0

An earlier revision of this branch also folded the indexer's head dimension out before the GEMM. That is withdrawn (reverted in c698ac13): the C4 indexer applies a per-head ReLU between the dot product and the weighted head sum, so the fold is not an identity — @efschu caught it in review below, and my own verification of it turns out to have compared two implementations that both omitted the ReLU. Everything quoted in this description is measured on the branch as it stands, with the fold gone. The exact-operator replacement — a fused kernel that keeps the per-head ReLU and never materialises the [B,S,H] intermediate — is follow-up work, gated on per-layer instrumentation of real checkpoints (see the thread).

Accuracy Tests

8xA800, DeepSeek-V4-Flash-0731 (auto_round W4A16), TP8. Branch checked out on its own, no out-of-tree patches.

The command below is the complete one, including the environment walls this PR does not fix — each is a separate pre-existing issue, and without them a reader cannot reproduce these results (one of them, the fused-projection one, fails silently). Two further walls used to belong in this list — the indexer's torch routing and the fallback's backend choice — but the branch now defaults both by capability on SM8x; the runs below predate that commit and were made with the equivalent env vars set, which selects the identical code paths:

export SGLANG_OPT_USE_TOPK_V2=0              # topk-v2 launches thread-block clusters (SM90+)
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=0      # deep_gemm invoked unconditionally though not importable here (#29738)
export SGLANG_OPT_FUSE_WQA_WKV=0             # fused path silently drops quantized wq_a/wkv (#33245)
export SGLANG_UVICORN_WORKER_HEALTHCHECK_TIMEOUT=14400  # 256K prefill exceeds the 10 s default

python3 -m sglang.launch_server \
  --model-path DeepSeek-V4-Flash-0731-W4A16 \
  --tp 8 --context-length 262144 --mem-fraction-static 0.70 \
  --attention-backend triton --cuda-graph-max-bs 8 \
  --chunked-prefill-size 8192 --trust-remote-code
$ python3 -m sglang.test.few_shot_gsm8k --num-questions 1319 --num-shots 5 --port 30000

Accuracy: 0.946
Invalid: 0.001
Latency: 708.7 s

Apples-to-apples against the official API — same 1319 GSM8K questions, same script, temperature=0, no output cap on either side, answer extraction identical:

  official deepseek-v4-flash API   1275/1319 = 96.7%
  this branch, local W4A16         1272/1319 = 96.4%

The 3-question gap is within run-to-run variance of the 8-way concurrent client; an earlier run of the same pair landed exactly equal.

Needle-in-a-haystack, randomised needles, depth 0.5, prefix cache flushed before each length:

(The ~2x wall-clock variance I reported earlier is explained: the second pass was hitting the prefix cache — the 99k tok/s "throughput" in that log was cache, not compute. Hence the flush per length here.)

  target    depth   actual prompt   hit    wall
   32768      0.5           29996   4/4     17.8 s
  131072      0.5          119852   4/4     98.1 s
  262144      0.5          239710   4/4    299.0 s
  524288      0.5          479372   4/4   1010.8 s
  1048576     0.5          958700   4/4   3710.2 s

The last two rows are at --context-length 1048576 --mem-fraction-static 0.55; 1M is this path's first completed run on this hardware, after the four allocation commits at the end of this branch.

Kernel-level (each row states what it is measured against):

  sparse decode        max diff 0, cosine 1.000000   vs an independently written
                                                     implementation of the operator
                                                     definition, qualified against the
                                                     upstream path first
  dequantize_k_cache   atol=0 rtol=0 (333 tokens)    vs the upstream torch reference
  indexer chunking /   bit-exact, max diff 0.0       vs the unchunked upstream torch
  shared-KV gather                                   path on the same inputs

Upstream test/registered/kernels/ops/attention/test_flash_mla_backends.py: 12 passed, 1 skipped. That file could not run on this hardware before this branch un-gated it from compute capability 12.0 — and it caught a real defect the first time it did.

SM86 / SM120, kernel and suite level (single consumer cards; neither fits the model, so serving claims remain A800-only):

  sm80   8xA800     12 passed / 1 skipped
  sm86   RTX 3090   12 passed / 1 skipped   135.7 s (cold Triton compile)
  sm120  RTX 5090   12 passed / 1 skipped    21.7 s

The consistent skip is the FlashInfer-vs-Triton case. On SM120 it skips not for the hardware reason its decorator suggests, but because flashinfer-python 0.6.12 — the version in both lmsysorg/sglang:latest and :dev-cu13 — ships no flashinfer.sparse_mla_sm120 module; as shipped, that case appears unrunnable anywhere (predates this branch). Real inputs captured on the A800 at the fp8_paged_mqa_logits entry and replayed on the 5090 land within bf16-materialisation distance of an fp64 oracle on both architectures. SM120 serving remains untested.

Speed Tests and Profiling

Sparse decode kernel: 320.14 ms -> 7.06 ms, warm-cache median of 20 runs.

Not quoting a speedup multiple: the baseline was taken cold-cache under a different protocol than the 7.06 ms, and dividing a warm number by a cold one overstates it. Re-measuring the baseline under the same protocol is outstanding.

Where the loss surface sits now, by context length: up to 32K the enablement is essentially free of the removed fold (the prefill gains came from the attention kernel, which stands); 128K–256K prefill grows with the indexer's quadratic term (98.1 s and 299.0 s wall for the needle runs above), and 512K–1M more so (1010.8 s and 3710.2 s). #33297 fuses that scoring and takes those two to 410.4 s and 1301.1 s.

Scope

Absorbs #33272, now closed — I had split the packed-layer classification out as a separate quantization concern, then found this branch cannot load a packed checkpoint without it. The indexer sequence-axis chunking overlaps in intent with #33259; whichever lands first is fine, it is here because without it the long-context results are not reproducible from this branch.

Checklist

  • pre-commit clean on all seven files
  • Tests — the existing suite is made to run on this hardware rather than a new one added
  • Documentation — no user-facing surface change
  • Accuracy and speed benchmarks
  • Code style

CI States

Latest PR Test (Base): ❌ Run #30887273929
Latest PR Test (Extra): ❌ Run #30887273580

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@Hakureirm

Copy link
Copy Markdown
Author

Adding the hardware evidence in one place, since this PR is a kernel change and the template asks for accuracy and speed sections — everything below was measured on 8xA800 (SM80) with DeepSeek-V4-Flash-0731, index_topk=512, 43 layers.

Accuracy

check result
sparse decode vs an independently written definition-based reference max diff 0, cosine 1.000000
dequantize_k_cache_paged vs the existing torch implementation, 333 tokens atol=0, rtol=0
test/registered/kernels/ops/attention/test_flash_mla_backends.py on A800 12 passed, 1 skipped
needle-in-a-haystack 4K / 32K / 128K, randomised needles 4/4 at every length

The suite in row 3 could not run on this hardware before the last commit of this PR — it was gated on compute capability 12.0. The first time it did run, it caught a real defect in an earlier revision of this branch.

Speed

before after
sparse decode kernel 320.14 ms 7.06 ms
server prefill throughput 107 tok/s 1085 tok/s
32K request, uncached 181.2 s 10.9 s

Every number is a warm-Triton-cache median of 20 runs. Cold-cache numbers are not conservative, they are wrong in both directions — autotune compiles while it times and selects a worse config.

Not tested by me: SM120. I have no such hardware. The FlashInfer path is untouched here and its test still requires SM120, so that case is unverified rather than verified-and-unchanged.

I cannot add the run-ci label myself, so CI has not run. Happy to rebase or split this if that helps review.

@Hakureirm

Copy link
Copy Markdown
Author

Correction to my own numbers above, found by re-auditing my measurement notes rather than by anyone pointing it out.

The 45.8x figure is not sound as stated, and neither is the 32K needle number.

  1. The 320.14 ms baseline was measured cold-cache. My own notes state that every number in the original round was taken on a first run, where Triton's autotune compiles while it times — which both inflates the measurement and makes autotune select a worse config. I later re-measured the optimised kernel warm (that is where 7.06 ms comes from) but never went back and re-measured the baseline the same way. Dividing a warm number by a cold one overstates the ratio by an unknown amount. I am re-measuring the original kernel warm and will post the honest ratio.

  2. The 32K request number is inconsistent in my own records — 10.9 s in one place, 17.1 s in another, with no reconciliation. I quoted the faster one. Treat it as unverified until I re-run it.

What is not affected, because these were measured warm and against a fixed reference from the start:

  • the accuracy table (max diff 0 / cosine 1.000000 against a definition-based reference; atol=0 rtol=0 for the dequant path; 12 passed / 1 skipped for the upstream suite on A800)
  • 7.06 ms for the kernel itself, warm-cache median of 20 runs
  • the direction and rough size of the win — the per-head kernel re-reads the same KV once per query head, and with 64 heads against MLA's single KV head that is a real 64x amplification of the dominant traffic

The correct framing until I re-measure is: the kernel is 7.06 ms warm; the previous implementation was substantially slower but I do not currently have a same-methodology number for it. I would rather say that than defend a ratio I cannot reproduce.

@Hakureirm

Copy link
Copy Markdown
Author

Progress update, since this PR has been sitting for the run-ci label and the picture has moved.

The kernel in this PR is now serving a real deployment, not just a microbenchmark. End-to-end numbers on the same 8xA800, measured with python -m sglang.benchmark.serving (the in-repo harness, not a script of mine), results written to disk:

scenario output throughput median TTFT mean TPOT
8K in / 1K out, concurrency 1 75.56 tok/s — 10.43 ms
shared-prefix, concurrency 4 (generated-shared-prefix) 122.79 tok/s 549 ms 28.92 ms

Long-context needle, same build: 256K and 512K both 4/4 (319.8 s and 1001.5 s wall clock, --chunked-prefill-size 8192). 512K previously OOMed; that is fixed by the indexer seq-axis chunking discussed in #33246, not by this PR, but it is the same deployment.

Two caveats I would rather state than have someone find:

  1. Speculative-decode accept rate is workload-dependent and I nearly reported a bad number. On --dataset-name random-ids the DSPARK accept rate reads 1.00 with accept length 6.00 — random token IDs make the output degenerate enough that the draft model is trivially right. On the shared-prefix workload it settles to 0.79 / 4.93, which is the number worth quoting. Anyone benchmarking speculative decoding on synthetic random tokens is measuring the dataset, not the model.

  2. I still owe an honest baseline ratio. As noted in my earlier correction, the 320.14 ms figure was taken cold-cache under a different protocol than the 7.06 ms it is divided by, so the "45.8x" is not reproducible as stated. I have not yet re-measured the original kernel under the warm-cache, fixed-config, median-of-20 protocol. Until I do, the defensible claim is the absolute number — 7.06 ms warm — not a ratio.

The accuracy results in this PR are unaffected by either caveat: max diff 0 / cosine 1.000000 against a definition-based reference, atol=0 rtol=0 for the dequant path, and 12 passed / 1 skipped on the upstream suite once it can run on this hardware.

@Hakureirm

Copy link
Copy Markdown
Author

@merrymercy @hnyls2002 @cctry — pinging Merge Oncall per the process in MAINTAINER.md, since this has no CI signal yet and I cannot add the run-ci label myself.

Short version of what is here: DSV4 currently cannot serve on SM80 at all — the Triton fallback and the K-cache dequant both fail to compile (fp8e4nv is not an accepted element type there), and three dispatch sites route by is_sm120_supported() rather than by whether the FlashMLA kernels exist. This PR fixes both and rewrites the Triton fallback so it is usable rather than nominal.

Verified on 8xA800: max diff 0 / cosine 1.000000 against a definition-based reference, atol=0 rtol=0 on the dequant path, needle 4/4 at 4K through 512K, and the upstream test file passing 12/1-skipped — that file could not run on this hardware before the last commit here, and caught a real defect the first time it did.

Happy to split it (the dispatch predicate, the kernel, and the test un-gating are independent) if that is easier to review.

@Hakureirm

Copy link
Copy Markdown
Author

Correcting my own ping above before anyone acts on it.

I wrote "needle 4/4 at 4K through 512K". That range is not attributable to this PR. This PR touches four files and none of them is indexer.py:

python/sglang/kernels/ops/attention/dsv4/dequant_k_cache.py
python/sglang/kernels/ops/attention/flash_mla_sm120_triton.py
python/sglang/srt/layers/attention/deepseek_v4_backend.py
test/registered/kernels/ops/attention/test_flash_mla_backends.py

The 256K and 512K runs were made on a build that also carried an indexer seq-axis chunking patch — the same defect as #33246 — which I deliberately kept out of this PR to avoid colliding with #33259. Without it, 512K OOMs in the indexer, and that has nothing to do with the kernel here.

What this PR alone is validated at:

check result
sparse decode vs a definition-based reference max diff 0, cosine 1.000000
dequantize_k_cache_paged vs the existing implementation, 333 tokens atol=0, rtol=0
upstream test_flash_mla_backends.py on A800 12 passed, 1 skipped
needle 4K / 32K / 128K 4/4

128K is the honest upper bound for this PR standalone. 256K and 512K are real results from the same deployment, but they depend on a fix that lives elsewhere, and I should not have folded them into a single range as though this branch produced them.

Everything else in the ping stands.

@Hakureirm Hakureirm changed the title Make DeepSeek-V4 run on SM80, and make the Triton sparse decode 45x faster Make DeepSeek-V4 serve on SM80: compile, dispatch, kernel, and the indexer's oversized intermediate Aug 2, 2026
@Hakureirm

Copy link
Copy Markdown
Author

Reproducibility update, and it changed the PR.

Everything I had reported here was produced on my machine's patch stack, not from this branch. I checked out the branch on its own and ran it — and it would not start:

File ".../sglang/srt/models/deepseek_v2.py", line 783, in __init__
    and self.shared_experts.gate_up_proj.weight.dtype == torch.int8
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'

That is #33272, which I had deliberately kept separate as a quantization concern. It is not separate: on a packed checkpoint — the only kind that fits the hardware this PR is about — nothing here can load without it. #33272 is now closed and the commit is folded in, which is why the file count went from 5 to 6.

Running the branch alone, with no out-of-tree patches:

target actual prompt tokens needle
32K 29,996 4/4
128K 119,852 4/4
256K 239,710 4/4

PYTHONPATH pointed at the branch checkout, verified from outside the container (sglang.__file__ resolves into it), no patch/*.py applied.

I am not quoting wall-clock from this run. Two passes over the same lengths differ by close to 2x (128K at 157 s and 275 s), which I have not explained yet — first-call Triton compilation is the obvious suspect but I have not confirmed it. The accuracy result is what this run establishes; the timings in the description above stand on their own separately-measured basis, and I will replace them with branch-only numbers once I can produce them without that variance.

One incidental finding while doing this: the branch would not accept --kv-cache-dtype fp8_e5m2, because overrides.py asserts against it for DSV4. I had been passing that flag all along. Dropping it changes nothing observable — which matches what the code suggests, that the flag does not reach DSV4's custom KV layout at all. So the right fix there is to stop passing it, not to relax the assertion, and I am not touching overrides.py in this PR.

@Hakureirm

Copy link
Copy Markdown
Author

@b8zhong — you reviewed and merged #33276 on this same DSV4 path today, so you may be the right person: this PR has no CI signal because it has no run-ci label, and I cannot add one.

It is rebased onto ec741e41 (your merge) as of a few minutes ago, no conflicts, pre-commit clean.

What it is: DeepSeek-V4 cannot serve on SM80 at all right now — two kernels fail to compile there (fp8e4nv is not an accepted Triton element type on SM80), three dispatch sites route by is_sm120_supported() rather than by whether the FlashMLA kernels exist, and a packed checkpoint cannot even load because deepseek_v2.py classifies layers by quantization name and then reads .weight on a layer that only has qweight. This fixes those and makes the Triton fallback usable rather than nominal.

Verified on 8xA800 with a real auto_round W4A16 checkpoint, running the branch on its own with no out-of-tree patches: needle 4/4 at 32K, 128K and 256K; sparse decode matches a definition-based reference at max diff 0 / cosine 1.000000; the upstream test_flash_mla_backends.py passes 12 / 1 skipped — that file could not run on this hardware before this PR, and caught a real defect in an earlier revision the first time it did.

Happy to split it if that helps review: the dispatch predicate, the decode kernel, the indexer changes, and the loader classification are independent commits.

@efschu

efschu commented Aug 2, 2026

Copy link
Copy Markdown

Before this merges, one thing needs to be on the table: the fold as derived is not an identity for this operator. The C4 indexer applies a per-head ReLU between the dot product and the weighted head sum (indexer.py:111 and :243 on main), and ReLU is not linear, so the head sum cannot move inside it — the derivation in the issue holds for the linear operator, not for this one. The PR's own diff shows the consequence: _folded_paged_logits is tl.dot → causal where → store with no ReLU anywhere, while the per-head fallback kept twenty lines below still applies F.relu — the fast path and the slow path compute different functions, and the fast path is default-on for every SM8x card.

On random tensors the divergence is large (I measure ~0.94 relative against the per-head reference, with ~50% of per-head products negative). Your 0.9967 top-k overlap and 4/4 needle results on a real checkpoint are genuine evidence the ReLU may be close to inert on trained weights — which would make the fold a good approximation, but one without an error bound, not the identity the derivation claims. Two suggestions: measure the clipped mass and folded-vs-real top-k overlap per layer on real data (I am running exactly that instrumentation on a V4-Flash checkpoint and will share the numbers), and until then ship it behind an explicit opt-in labelled as an approximation rather than as the default. The 37-99x is real and worth having — it just needs the honest label.

@efschu

efschu commented Aug 3, 2026

Copy link
Copy Markdown

The two rows in the |w| form, as promised. Same instrument as before, re-run with the absolute-value accumulators added; our setup only has the 32K point, so one context column rather than your three.

layer 32K o512 / frac_neg / clip|w|
40 0.559 / 0.524 / 0.531
42 0.783 / 0.479 / 0.390

Method note: clip|w| is Σ_h |w_h·min(s_h,0)| / Σ_h |w_h·s_h|, verified against the field semantics of clip_abs in the dataset you published, so the columns should be directly comparable. frac_neg and the top-512 overlap are native outputs of the same probe, not reconstructions. Layer 41 has no C4 indexer on this checkpoint (compress_ratio=128, where 40 and 42 are ratio-4), so it cannot be emitted at all.

Our layer-40 clip|w| of 0.531 sits close to your 0.491, and layer 42's 0.390 close to your 0.410 — which is the useful part: on the same records the signed form reads 0.239 and 0.498, so the two setups agree in the |w| form and disagree in the signed one. That is the behaviour you predicted.

efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
Includes:
- 460fd46 DSpark draft: anchor the .scale checkpoint rename at the suffix
- c015edc DSpark: the fused-KV-proj support probe must answer, not raise
- af9b734 docs: NOTE_490 (upstream PR sgl-project#33271 sweep) + rebase gates 6-9
@Hakureirm

Copy link
Copy Markdown
Author

Side by side, our 32K column against yours:

layer yours: o512 / frac_neg / clip|w| ours (32K): o512 / frac_neg / clip|w|
40 0.559 / 0.524 / 0.531 0.709 / 0.474 / 0.491
42 0.783 / 0.479 / 0.390 0.809 / 0.446 / 0.410

Layer 42 lands within a couple of hundredths on all three quantities across different checkpoints and hardware. Layer 40 is the worst layer in both setups, deeper in yours: your 32K point (0.559) is where our layer 40 only arrives at 256K (0.555, from 0.709 → 0.598 → 0.555 across the three lengths). So the layer-40 anomaly is a property of the trained model family, not of either measurement stack — and your checkpoint expresses it earlier in context length.

The signed-vs-|w| split on the same records (0.239 / 0.498 signed, converging in |w|) settles the definition question with independent data on both sides of it. That is the cleanest outcome the re-run could have produced.

Structure note that matches: our C4 set is the even layers 2–42 too — the odd layers carry no ratio-4 indexer on this checkpoint either, so your layer-41 gap is the same layout, not a divergence.

Your frac_neg values (0.524, 0.479) also land inside the 0.45–0.55 band we see everywhere, which was the open question about checkpoint dependence. One context point and two layers is not the full grid, but on those coordinates the two instruments now read the same physics.

…achable

apply_gptq_quant_layer delegates FusedMoE layers to MoeWNA16, which
re-runs the Marlin eligibility check itself and routes to the Marlin
MoE method when it passes. Two defects kept that delegation from ever
reaching Marlin:

- the config handed over lacked the desc_act key, and
  GPTQMarlinConfig.is_gptq_marlin_compatible treats a missing key as
  ineligible -- so the delegation always landed on the Triton runner;
- the use_marlin=False arm constructed GPTQMarlinMoEMethod from
  quant_args_marlin, a variable only defined in the marlin branch: a
  latent NameError, meaning the split had no working non-marlin arm and
  no test coverage on either arm.

Collapse the split into one delegation with a complete config: MoeWNA16
picks Marlin when eligible and the Triton runner when not. auto-round
has no act-order concept, so desc_act is always False. Observable
before this change: a GPTQ-packed auto-round MoE deployment logs
'Using default MoE kernel config' on every rank while its dense layers
run on Marlin.
@Hakureirm
Hakureirm force-pushed the sm80-dsv4-sparse-decode branch from 7913291 to 00cdd4b Compare August 3, 2026 17:28
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…utation, sgl-project#471/sgl-project#33407/sgl-project#33312/sgl-project#33098/sgl-project#33271 ports, sgl-project#481 PP fixes

Eight commits, largest bundle this session:

- sgl-project#479 (two commits): the GGUF-MoE offload coverage test and its refusal
  message named MXFP4/type 39 as THE example of a ggml type with no MoE
  kernel -- sgl-project#398 made that false (type 39 has a full native kernel set on
  a wheel that carries it). Traced the ACTIVE DSV4-Flash MXFP4 path end to
  end and confirmed it is native, not an untraced fallback; the refusal
  message now names the layer's own declared types instead of a stale
  hardcoded example, so it cannot go stale again the same way.
  docs/dev/NOTE_479_mxfp4_active_driver_path.md has the trace.
- sgl-project#471: ports upstream sglang#32320 -- mask the SM120 SWA page-split to
  touched pages only, instead of the full page range.
  docs/dev/TICKET_471_masked_page_split.md.
- sgl-project#33407 port: SM120 sparse-MLA buckets the topk width instead of using
  it unbucketed, falling back to Triton outside the bucketed range.
- sgl-project#33312 port: DSpark draft head resolves and loads fused shared experts.
- sgl-project#33098 port: DSpark draft fills the EP token-accounting fields.
- sgl-project#481: three pipeline-parallel defects from the sgl-project#445 window fixed in
  server_args.py/resident_fraction.py/expert_stats.py.
- sgl-project#33271 port: auto-round GPTQ MoE completes the MoeWNA16 delegation.

FEATURE_CATALOG.md §8 conflict-risk as flagged, but no actual conflict --
this branch's base (a5eff26) is now several merges behind (sgl-project#520, sgl-project#523,
design-453 landed since), yet none of those touched §8, so git's 3-way
merge combined everything automatically. All 17 sections and all 22
previously-preserved additive paragraphs verified present.

**Explicit verification requested and confirmed**: the two "ancient
pre-existing" failures in tests/moe_offload/test_gguf_moe_offload.py
(test_materialize_declines_uncovered_ggml_type,
test_guard_still_refuses_an_unstaged_gguf_layer) that this session
bisected back to before this session even began -- sgl-project#479's fix to
expert_offload.py and the type-39 assumption in gguf.py directly
addresses their root cause. Full file run: 20 passed, 0 failed, both
confirmed GREEN now.

One test-isolation quirk found and run to ground, not a code bug: the
three flash_mla test files this branch adds/touches
(test_flash_mla_backends.py, test_flash_mla_page_split_mask_471.py,
test_flash_mla_sm120_topk_buckets.py) interfere with each other when run
in the same pytest process -- a shared Triton-decorated function object
resolves to JITFunction instead of the expected InterpretedFunction once
another file in the same process has already imported flash_mla_sm120.py
under real (non-interpret) conditions. Verified this is NOT a merge
regression: reproduced identically on the unmerged branch tip in
isolation (fresh archive, no other code involved), and every one of the
30 tests across the three files passes cleanly when its file is run
alone. Branch-inherent test-ordering fragility, not in scope for this
merge to fix.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- tests/moe_offload/test_gguf_moe_offload.py alone: 20 passed (was 18
  passed/2 failed before this merge, per this session's own bisection).
- The eight new/touched test files, run individually where the isolation
  quirk above requires it: all pass in full (dspark 442: 18; auto_round:
  17; dspark shared-expert 33312: 21; gguf mxfp4 479: 18; pp defects 481:
  26; flash_mla backends/topk-buckets/page-split-mask: 10/25/10 each
  alone).
- test/registered/unit/quantization/ + unit/models/ + spec/dspark/: 28
  failed both before and after (diffed by test ID, not count -- the
  pre-merge archive showed 35 because 7 of its failures were an artifact
  of an incomplete git-archive extraction missing sgl-kernel/, confirmed
  by re-checking those 7 IDs against the working tree directly). The
  remaining 28 are dspark_kernel_parity's CUDA-only subtests plus
  test_gguf_dequant_scratch_budget.py, both pre-named as pre-existing.
- test/registered/unit/layers/ + tests/moe_offload/: 3 failed (the
  pre-named test_deterministic_fp8_gemm.py x2 +
  test_flashinfer_workspace_zeroing.py x1), 1168 passed -- confirms
  test_gguf_moe_offload.py's contribution to this count is now 0 failures.
- test/registered/unit/distributed/ (full, server_args.py touched): 24
  failed, byte-identical established baseline.
- ruff check --select=F401,F821,UP037: 15 findings, all in
  expert_offload.py, all confirmed pre-existing (same set already
  verified at cc5a92e in the earlier sgl-project#494 merge) and outside this
  branch's two diff hunks (2112-2138, 2616-2648) -- auto_round.py and
  every other touched file clean.
- codespell: clean on all 21 touched/added files.
… assembly

Invalid lanes (padding indices, ragged tail tiles) are clamped to slot 0
for address safety, but their KV/scale/rope bytes were loaded unmasked.
If slot 0 was never written, garbage bytes can decode to inf/NaN, which
p=0 cannot cancel in the PV dot (0 * inf = NaN) -- silent output
corruption. Mask all three loads by lane validity (zero bytes decode to
exactly 0.0 through the subnormal branch; scale 127 is the neutral 2^0).

_e4m3_scaled also assembled out-of-range exponents from garbage scale
bytes: below 1 the sign bit gets garbage, at 255+ the value becomes
inf/NaN, and the subnormal branch produced wrong magnitudes for sc <= 9.
Clamp both branches; exact for all reachable inputs since the true value
under/overflows in those regimes anyway.
@Hakureirm

Copy link
Copy Markdown
Author

Pushed cd1a849db8: an internal adversarial review pass caught two latent correctness issues in the rewritten sparse decode kernel — invalid lanes (padding indices / ragged tail tiles) had their KV/scale/rope loads unmasked after being clamped to slot 0, and _e4m3_scaled could assemble out-of-range exponents from garbage scale bytes. Both are silent-NaN paths reachable when slot 0 of the KV pool was never written (fresh torch.empty allocation). Fixed by masking all three loads by lane validity and clamping the exponent assembly in both branches.

Verified on 8x A800: registered kernel tests 25 passed / 1 skipped, needle-in-haystack 12/12 (4K/32K/128K x 3 depths), no throughput regression.

Hakureirm added a commit to Hakureirm/sglang that referenced this pull request Aug 4, 2026
The torch route materialises a [B, S, num_heads] fp32 score tensor and
reads it back twice; at long context that traffic, not FLOPs, is the
indexer's cost. The fused Triton kernel keeps the whole chain -- gather,
E4M3 bit decode, per-head dot, ReLU, weighting, head sum, per-token
scale -- in registers. The operator is unchanged, per-head ReLU included.

Decode keeps the torch variant: it runs under CUDA graph capture, where
Triton autotuning cannot, and its batches are too small to benefit.

Needle wall clock on 8xA800 (W4A16, cache flushed per length, 4/4 hits
at every length): 32K 16.7 -> 14.2 s, 128K 99.7 -> 56.3 s, 256K
313.2 -> 141.4 s. Kernel-level against the torch path on real shapes:
5.3-7.2x. Against an fp64 oracle the kernel lands ~1e4x closer than the
torch path (fp32 accumulation, no bf16 materialisation); its top-512
matches the oracle exactly on every tested row.

The test compares both implementations against the fp64 oracle rather
than only against each other, and requires the criterion to catch a
deliberately broken (ReLU-dropped) implementation -- the two defences
that were missing when a wrong derivation of this operator passed
verification in sgl-project#33271's history.
Hakureirm added a commit to Hakureirm/sglang that referenced this pull request Aug 4, 2026
SGLANG_DSV4_INDEXER_FOLD_APPROX=1 routes prefill-sized indexer calls to
the folded linear operator: q_eff = sum_h w_h q_h, one [B, D] @ [D, N]
GEMM over the shared KV. This is the approximation withdrawn from sgl-project#33271
when its derivation was shown not to commute with the per-head ReLU --
returning here in the shape review asked for then: explicitly lossy,
default off, with the measured per-layer cost grid quoted at the
implementation. Non-shared page tables and decode fall back to the exact
fused path. The test pins the fold to its own spec (the linear operator)
and the fallback bitwise to the exact path; closeness to the real
operator is deliberately not asserted -- the divergence is the
documented cost.
@Hakureirm

Copy link
Copy Markdown
Author

Closing this. A maintainer has said upstream has no plan to support SM80, which is a fair call to make — this PR exists only for that hardware, so it should not sit in the queue taking up review attention.

For anyone who lands here later with the same problem: most of what this PR does turns out to be unnecessary. DeepSeek-V4 on Ampere works with the stock checkpoint once a single architecture gate is relaxed; the details are in the closing comment on #33541. The one change here that is not about SM80 is the sparse decode kernel bounding its tile loop by the padded top-k width rather than the real entry count — that is a plain waste on any architecture, though it only becomes visible when the padding dominates.

Thanks for the CI cycles.

@Hakureirm Hakureirm closed this Aug 4, 2026
ehuaa added a commit to ehuaa/sglang that referenced this pull request Aug 10, 2026
…on path

The FlashMLA sparse CUDA kernels are arch-conditional builds -- the
target list in kernels/aot/cmake/flashmla.cmake is sm_90a, sm_100f and
(with CUDA 13) sm_103a -- so on SM80 both entry points abort:

  sparse_decode_fwd   RuntimeError: Unsupported architecture for sparse
                      decode fwd
  sparse_prefill_fwd  RuntimeError: only supported on SM90a and SM100f

'a'/'f' binaries do not carry forward across compute capabilities, so
the guard is an exact-match list of the architectures that *do* have
the kernel; anything else falls back rather than dispatching into a
kernel that was never built for it. That polarity is what keeps a
future architecture from crashing instead of degrading.

Take the Triton sparse decode kernel from PR sgl-project#33271 (sgl-project/sglang),
which is written for this cache layout: it reads the paged FP8 KV
directly, four E4M3 lanes per int32 load, and decodes them with bit
arithmetic that folds the E8M0 scale into the exponent -- no separate
dequant pass and no fp8e4nv in the signature. One program covers
BLOCK_H heads so each KV tile is read once instead of once per head.

Prefill takes the same kernel: on a fallback architecture the sparse
prefill branch is skipped, so both phases share the decode path. The
FlashInfer default is also repointed, since it is CUTLASS built for
SM120 only and every architecture arriving here does so precisely
because it has no CUDA sparse-decode kernel.

With this the model serves on 8x A100 with CUDA graphs on.
ehuaa added a commit to ehuaa/sglang that referenced this pull request Aug 10, 2026
`_apply_attn_sink` and `_merge_partial_attn` did the sink normalisation
and the LSE-weighted merge as plain torch expressions over `out.float()`.
That promotes the entire bf16 output to fp32 as a real tensor: two
output-sized transients for the sink, four for the merge. At 8K tokens x
64 heads x 512 the sink alone tries to allocate 896 MiB, and a tp8
batched run dies with

  torch.OutOfMemoryError: Tried to allocate 896.00 MiB
    at _apply_attn_sink: (out.float() * w.unsqueeze(-1))

A single request never reaches that size, so this only appears under
batch. The weights themselves genuinely need fp32 (logaddexp, exp), and
that part is unchanged; what was wasteful is materialising the output in
fp32 just to scale it elementwise.

Two small Triton kernels keep the fp32 value in registers instead.
Measured on A100 against the torch form, 1K-16K tokens:

  identical results (the same single rounding, max|delta| = 0 vs fp32)
  240x less peak memory (4104 -> 17 MiB at 16K tokens)
  5.5x faster (7.40 -> 1.35 ms)

An in-place bf16 multiply also removes the transient, but rounds the
weight to bf16 before applying it, which moves relative error from
2.8e-3 to 4.4e-3 -- an unnecessary trade when the fused form keeps fp32.

Note this code predates PR sgl-project#33271 and is untouched by it: the same OOM
is reachable on SM120 at batch, so the fix is not Ampere-specific.
ehuaa added a commit to ehuaa/sglang that referenced this pull request Aug 26, 2026
…on path

The FlashMLA sparse CUDA kernels are arch-conditional builds -- the
target list in kernels/aot/cmake/flashmla.cmake is sm_90a, sm_100f and
(with CUDA 13) sm_103a -- so on SM80 both entry points abort:

  sparse_decode_fwd   RuntimeError: Unsupported architecture for sparse
                      decode fwd
  sparse_prefill_fwd  RuntimeError: only supported on SM90a and SM100f

'a'/'f' binaries do not carry forward across compute capabilities, so
the guard is an exact-match list of the architectures that *do* have
the kernel; anything else falls back rather than dispatching into a
kernel that was never built for it. That polarity is what keeps a
future architecture from crashing instead of degrading.

Take the Triton sparse decode kernel from PR sgl-project#33271 (sgl-project/sglang),
which is written for this cache layout: it reads the paged FP8 KV
directly, four E4M3 lanes per int32 load, and decodes them with bit
arithmetic that folds the E8M0 scale into the exponent -- no separate
dequant pass and no fp8e4nv in the signature. One program covers
BLOCK_H heads so each KV tile is read once instead of once per head.

Prefill takes the same kernel: on a fallback architecture the sparse
prefill branch is skipped, so both phases share the decode path. The
FlashInfer default is also repointed, since it is CUTLASS built for
SM120 only and every architecture arriving here does so precisely
because it has no CUDA sparse-decode kernel.

With this the model serves on 8x A100 with CUDA graphs on.

(cherry picked from commit 4e9ad4c)
ehuaa added a commit to ehuaa/sglang that referenced this pull request Aug 26, 2026
`_apply_attn_sink` and `_merge_partial_attn` did the sink normalisation
and the LSE-weighted merge as plain torch expressions over `out.float()`.
That promotes the entire bf16 output to fp32 as a real tensor: two
output-sized transients for the sink, four for the merge. At 8K tokens x
64 heads x 512 the sink alone tries to allocate 896 MiB, and a tp8
batched run dies with

  torch.OutOfMemoryError: Tried to allocate 896.00 MiB
    at _apply_attn_sink: (out.float() * w.unsqueeze(-1))

A single request never reaches that size, so this only appears under
batch. The weights themselves genuinely need fp32 (logaddexp, exp), and
that part is unchanged; what was wasteful is materialising the output in
fp32 just to scale it elementwise.

Two small Triton kernels keep the fp32 value in registers instead.
Measured on A100 against the torch form, 1K-16K tokens:

  identical results (the same single rounding, max|delta| = 0 vs fp32)
  240x less peak memory (4104 -> 17 MiB at 16K tokens)
  5.5x faster (7.40 -> 1.35 ms)

An in-place bf16 multiply also removes the transient, but rounds the
weight to bf16 before applying it, which moves relative error from
2.8e-3 to 4.4e-3 -- an unnecessary trade when the fused form keeps fp32.

Note this code predates PR sgl-project#33271 and is untouched by it: the same OOM
is reachable on SM120 at batch, so the fix is not Ampere-specific.

(cherry picked from commit 97ad98d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants