Repository navigation
Make DeepSeek-V4 serve on SM80: compile, dispatch, kernel, and the indexer's oversized intermediate - #33271
Make DeepSeek-V4 serve on SM80: compile, dispatch, kernel, and the indexer's oversized intermediate#33271Hakureirm wants to merge 23 commits into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Adding the hardware evidence in one place, since this PR is a kernel change and the template asks for accuracy and speed sections — everything below was measured on 8xA800 (SM80) with DeepSeek-V4-Flash-0731, Accuracy
The suite in row 3 could not run on this hardware before the last commit of this PR — it was gated on compute capability 12.0. The first time it did run, it caught a real defect in an earlier revision of this branch. Speed
Every number is a warm-Triton-cache median of 20 runs. Cold-cache numbers are not conservative, they are wrong in both directions — autotune compiles while it times and selects a worse config. Not tested by me: SM120. I have no such hardware. The FlashInfer path is untouched here and its test still requires SM120, so that case is unverified rather than verified-and-unchanged. I cannot add the |
|
Correction to my own numbers above, found by re-auditing my measurement notes rather than by anyone pointing it out. The 45.8x figure is not sound as stated, and neither is the 32K needle number.
What is not affected, because these were measured warm and against a fixed reference from the start:
The correct framing until I re-measure is: the kernel is 7.06 ms warm; the previous implementation was substantially slower but I do not currently have a same-methodology number for it. I would rather say that than defend a ratio I cannot reproduce. |
|
Progress update, since this PR has been sitting for the The kernel in this PR is now serving a real deployment, not just a microbenchmark. End-to-end numbers on the same 8xA800, measured with
Long-context needle, same build: 256K and 512K both 4/4 (319.8 s and 1001.5 s wall clock, Two caveats I would rather state than have someone find:
The accuracy results in this PR are unaffected by either caveat: max diff 0 / cosine 1.000000 against a definition-based reference, atol=0 rtol=0 for the dequant path, and 12 passed / 1 skipped on the upstream suite once it can run on this hardware. |
|
@merrymercy @hnyls2002 @cctry — pinging Merge Oncall per the process in MAINTAINER.md, since this has no CI signal yet and I cannot add the Short version of what is here: DSV4 currently cannot serve on SM80 at all — the Triton fallback and the K-cache dequant both fail to compile ( Verified on 8xA800: max diff 0 / cosine 1.000000 against a definition-based reference, atol=0 rtol=0 on the dequant path, needle 4/4 at 4K through 512K, and the upstream test file passing 12/1-skipped — that file could not run on this hardware before the last commit here, and caught a real defect the first time it did. Happy to split it (the dispatch predicate, the kernel, and the test un-gating are independent) if that is easier to review. |
|
Correcting my own ping above before anyone acts on it. I wrote "needle 4/4 at 4K through 512K". That range is not attributable to this PR. This PR touches four files and none of them is The 256K and 512K runs were made on a build that also carried an indexer seq-axis chunking patch — the same defect as #33246 — which I deliberately kept out of this PR to avoid colliding with #33259. Without it, 512K OOMs in the indexer, and that has nothing to do with the kernel here. What this PR alone is validated at:
128K is the honest upper bound for this PR standalone. 256K and 512K are real results from the same deployment, but they depend on a fix that lives elsewhere, and I should not have folded them into a single range as though this branch produced them. Everything else in the ping stands. |
|
Reproducibility update, and it changed the PR. Everything I had reported here was produced on my machine's patch stack, not from this branch. I checked out the branch on its own and ran it — and it would not start: That is #33272, which I had deliberately kept separate as a quantization concern. It is not separate: on a packed checkpoint — the only kind that fits the hardware this PR is about — nothing here can load without it. #33272 is now closed and the commit is folded in, which is why the file count went from 5 to 6. Running the branch alone, with no out-of-tree patches:
I am not quoting wall-clock from this run. Two passes over the same lengths differ by close to 2x (128K at 157 s and 275 s), which I have not explained yet — first-call Triton compilation is the obvious suspect but I have not confirmed it. The accuracy result is what this run establishes; the timings in the description above stand on their own separately-measured basis, and I will replace them with branch-only numbers once I can produce them without that variance. One incidental finding while doing this: the branch would not accept |
9c13c04 to
1101d33
Compare
|
@b8zhong — you reviewed and merged #33276 on this same DSV4 path today, so you may be the right person: this PR has no CI signal because it has no It is rebased onto What it is: DeepSeek-V4 cannot serve on SM80 at all right now — two kernels fail to compile there ( Verified on 8xA800 with a real auto_round W4A16 checkpoint, running the branch on its own with no out-of-tree patches: needle 4/4 at 32K, 128K and 256K; sparse decode matches a definition-based reference at max diff 0 / cosine 1.000000; the upstream Happy to split it if that helps review: the dispatch predicate, the decode kernel, the indexer changes, and the loader classification are independent commits. |
|
Before this merges, one thing needs to be on the table: the fold as derived is not an identity for this operator. The C4 indexer applies a per-head ReLU between the dot product and the weighted head sum ( On random tensors the divergence is large (I measure ~0.94 relative against the per-head reference, with ~50% of per-head products negative). Your 0.9967 top-k overlap and 4/4 needle results on a real checkpoint are genuine evidence the ReLU may be close to inert on trained weights — which would make the fold a good approximation, but one without an error bound, not the identity the derivation claims. Two suggestions: measure the clipped mass and folded-vs-real top-k overlap per layer on real data (I am running exactly that instrumentation on a V4-Flash checkpoint and will share the numbers), and until then ship it behind an explicit opt-in labelled as an approximation rather than as the default. The 37-99x is real and worth having — it just needs the honest label. |
|
The two rows in the |w| form, as promised. Same instrument as before, re-run with the absolute-value accumulators added; our setup only has the 32K point, so one context column rather than your three.
Method note: Our layer-40 |
Includes: - 460fd46 DSpark draft: anchor the .scale checkpoint rename at the suffix - c015edc DSpark: the fused-KV-proj support probe must answer, not raise - af9b734 docs: NOTE_490 (upstream PR sgl-project#33271 sweep) + rebase gates 6-9
|
Side by side, our 32K column against yours:
Layer 42 lands within a couple of hundredths on all three quantities across different checkpoints and hardware. Layer 40 is the worst layer in both setups, deeper in yours: your 32K point (0.559) is where our layer 40 only arrives at 256K (0.555, from 0.709 → 0.598 → 0.555 across the three lengths). So the layer-40 anomaly is a property of the trained model family, not of either measurement stack — and your checkpoint expresses it earlier in context length. The signed-vs-|w| split on the same records (0.239 / 0.498 signed, converging in |w|) settles the definition question with independent data on both sides of it. That is the cleanest outcome the re-run could have produced. Structure note that matches: our C4 set is the even layers 2–42 too — the odd layers carry no ratio-4 indexer on this checkpoint either, so your layer-41 gap is the same layout, not a divergence. Your frac_neg values (0.524, 0.479) also land inside the 0.45–0.55 band we see everywhere, which was the open question about checkpoint dependence. One context point and two layers is not the full grid, but on those coordinates the two instruments now read the same physics. |
1f0d916 to
7913291
Compare
…achable apply_gptq_quant_layer delegates FusedMoE layers to MoeWNA16, which re-runs the Marlin eligibility check itself and routes to the Marlin MoE method when it passes. Two defects kept that delegation from ever reaching Marlin: - the config handed over lacked the desc_act key, and GPTQMarlinConfig.is_gptq_marlin_compatible treats a missing key as ineligible -- so the delegation always landed on the Triton runner; - the use_marlin=False arm constructed GPTQMarlinMoEMethod from quant_args_marlin, a variable only defined in the marlin branch: a latent NameError, meaning the split had no working non-marlin arm and no test coverage on either arm. Collapse the split into one delegation with a complete config: MoeWNA16 picks Marlin when eligible and the Triton runner when not. auto-round has no act-order concept, so desc_act is always False. Observable before this change: a GPTQ-packed auto-round MoE deployment logs 'Using default MoE kernel config' on every rank while its dense layers run on Marlin.
7913291 to
00cdd4b
Compare
…utation, sgl-project#471/sgl-project#33407/sgl-project#33312/sgl-project#33098/sgl-project#33271 ports, sgl-project#481 PP fixes Eight commits, largest bundle this session: - sgl-project#479 (two commits): the GGUF-MoE offload coverage test and its refusal message named MXFP4/type 39 as THE example of a ggml type with no MoE kernel -- sgl-project#398 made that false (type 39 has a full native kernel set on a wheel that carries it). Traced the ACTIVE DSV4-Flash MXFP4 path end to end and confirmed it is native, not an untraced fallback; the refusal message now names the layer's own declared types instead of a stale hardcoded example, so it cannot go stale again the same way. docs/dev/NOTE_479_mxfp4_active_driver_path.md has the trace. - sgl-project#471: ports upstream sglang#32320 -- mask the SM120 SWA page-split to touched pages only, instead of the full page range. docs/dev/TICKET_471_masked_page_split.md. - sgl-project#33407 port: SM120 sparse-MLA buckets the topk width instead of using it unbucketed, falling back to Triton outside the bucketed range. - sgl-project#33312 port: DSpark draft head resolves and loads fused shared experts. - sgl-project#33098 port: DSpark draft fills the EP token-accounting fields. - sgl-project#481: three pipeline-parallel defects from the sgl-project#445 window fixed in server_args.py/resident_fraction.py/expert_stats.py. - sgl-project#33271 port: auto-round GPTQ MoE completes the MoeWNA16 delegation. FEATURE_CATALOG.md §8 conflict-risk as flagged, but no actual conflict -- this branch's base (a5eff26) is now several merges behind (sgl-project#520, sgl-project#523, design-453 landed since), yet none of those touched §8, so git's 3-way merge combined everything automatically. All 17 sections and all 22 previously-preserved additive paragraphs verified present. **Explicit verification requested and confirmed**: the two "ancient pre-existing" failures in tests/moe_offload/test_gguf_moe_offload.py (test_materialize_declines_uncovered_ggml_type, test_guard_still_refuses_an_unstaged_gguf_layer) that this session bisected back to before this session even began -- sgl-project#479's fix to expert_offload.py and the type-39 assumption in gguf.py directly addresses their root cause. Full file run: 20 passed, 0 failed, both confirmed GREEN now. One test-isolation quirk found and run to ground, not a code bug: the three flash_mla test files this branch adds/touches (test_flash_mla_backends.py, test_flash_mla_page_split_mask_471.py, test_flash_mla_sm120_topk_buckets.py) interfere with each other when run in the same pytest process -- a shared Triton-decorated function object resolves to JITFunction instead of the expected InterpretedFunction once another file in the same process has already imported flash_mla_sm120.py under real (non-interpret) conditions. Verified this is NOT a merge regression: reproduced identically on the unmerged branch tip in isolation (fresh archive, no other code involved), and every one of the 30 tests across the three files passes cleanly when its file is run alone. Branch-inherent test-ordering fragility, not in scope for this merge to fix. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - tests/moe_offload/test_gguf_moe_offload.py alone: 20 passed (was 18 passed/2 failed before this merge, per this session's own bisection). - The eight new/touched test files, run individually where the isolation quirk above requires it: all pass in full (dspark 442: 18; auto_round: 17; dspark shared-expert 33312: 21; gguf mxfp4 479: 18; pp defects 481: 26; flash_mla backends/topk-buckets/page-split-mask: 10/25/10 each alone). - test/registered/unit/quantization/ + unit/models/ + spec/dspark/: 28 failed both before and after (diffed by test ID, not count -- the pre-merge archive showed 35 because 7 of its failures were an artifact of an incomplete git-archive extraction missing sgl-kernel/, confirmed by re-checking those 7 IDs against the working tree directly). The remaining 28 are dspark_kernel_parity's CUDA-only subtests plus test_gguf_dequant_scratch_budget.py, both pre-named as pre-existing. - test/registered/unit/layers/ + tests/moe_offload/: 3 failed (the pre-named test_deterministic_fp8_gemm.py x2 + test_flashinfer_workspace_zeroing.py x1), 1168 passed -- confirms test_gguf_moe_offload.py's contribution to this count is now 0 failures. - test/registered/unit/distributed/ (full, server_args.py touched): 24 failed, byte-identical established baseline. - ruff check --select=F401,F821,UP037: 15 findings, all in expert_offload.py, all confirmed pre-existing (same set already verified at cc5a92e in the earlier sgl-project#494 merge) and outside this branch's two diff hunks (2112-2138, 2616-2648) -- auto_round.py and every other touched file clean. - codespell: clean on all 21 touched/added files.
… assembly Invalid lanes (padding indices, ragged tail tiles) are clamped to slot 0 for address safety, but their KV/scale/rope bytes were loaded unmasked. If slot 0 was never written, garbage bytes can decode to inf/NaN, which p=0 cannot cancel in the PV dot (0 * inf = NaN) -- silent output corruption. Mask all three loads by lane validity (zero bytes decode to exactly 0.0 through the subnormal branch; scale 127 is the neutral 2^0). _e4m3_scaled also assembled out-of-range exponents from garbage scale bytes: below 1 the sign bit gets garbage, at 255+ the value becomes inf/NaN, and the subnormal branch produced wrong magnitudes for sc <= 9. Clamp both branches; exact for all reachable inputs since the true value under/overflows in those regimes anyway.
|
Pushed Verified on 8x A800: registered kernel tests 25 passed / 1 skipped, needle-in-haystack 12/12 (4K/32K/128K x 3 depths), no throughput regression. |
The torch route materialises a [B, S, num_heads] fp32 score tensor and reads it back twice; at long context that traffic, not FLOPs, is the indexer's cost. The fused Triton kernel keeps the whole chain -- gather, E4M3 bit decode, per-head dot, ReLU, weighting, head sum, per-token scale -- in registers. The operator is unchanged, per-head ReLU included. Decode keeps the torch variant: it runs under CUDA graph capture, where Triton autotuning cannot, and its batches are too small to benefit. Needle wall clock on 8xA800 (W4A16, cache flushed per length, 4/4 hits at every length): 32K 16.7 -> 14.2 s, 128K 99.7 -> 56.3 s, 256K 313.2 -> 141.4 s. Kernel-level against the torch path on real shapes: 5.3-7.2x. Against an fp64 oracle the kernel lands ~1e4x closer than the torch path (fp32 accumulation, no bf16 materialisation); its top-512 matches the oracle exactly on every tested row. The test compares both implementations against the fp64 oracle rather than only against each other, and requires the criterion to catch a deliberately broken (ReLU-dropped) implementation -- the two defences that were missing when a wrong derivation of this operator passed verification in sgl-project#33271's history.
SGLANG_DSV4_INDEXER_FOLD_APPROX=1 routes prefill-sized indexer calls to the folded linear operator: q_eff = sum_h w_h q_h, one [B, D] @ [D, N] GEMM over the shared KV. This is the approximation withdrawn from sgl-project#33271 when its derivation was shown not to commute with the per-head ReLU -- returning here in the shape review asked for then: explicitly lossy, default off, with the measured per-layer cost grid quoted at the implementation. Non-shared page tables and decode fall back to the exact fused path. The test pins the fold to its own spec (the linear operator) and the fallback bitwise to the exact path; closeness to the real operator is deliberately not asserted -- the divergence is the documented cost.
|
Closing this. A maintainer has said upstream has no plan to support SM80, which is a fair call to make — this PR exists only for that hardware, so it should not sit in the queue taking up review attention. For anyone who lands here later with the same problem: most of what this PR does turns out to be unnecessary. DeepSeek-V4 on Ampere works with the stock checkpoint once a single architecture gate is relaxed; the details are in the closing comment on #33541. The one change here that is not about SM80 is the sparse decode kernel bounding its tile loop by the padded top-k width rather than the real entry count — that is a plain waste on any architecture, though it only becomes visible when the padding dominates. Thanks for the CI cycles. |
…on path
The FlashMLA sparse CUDA kernels are arch-conditional builds -- the
target list in kernels/aot/cmake/flashmla.cmake is sm_90a, sm_100f and
(with CUDA 13) sm_103a -- so on SM80 both entry points abort:
sparse_decode_fwd RuntimeError: Unsupported architecture for sparse
decode fwd
sparse_prefill_fwd RuntimeError: only supported on SM90a and SM100f
'a'/'f' binaries do not carry forward across compute capabilities, so
the guard is an exact-match list of the architectures that *do* have
the kernel; anything else falls back rather than dispatching into a
kernel that was never built for it. That polarity is what keeps a
future architecture from crashing instead of degrading.
Take the Triton sparse decode kernel from PR sgl-project#33271 (sgl-project/sglang),
which is written for this cache layout: it reads the paged FP8 KV
directly, four E4M3 lanes per int32 load, and decodes them with bit
arithmetic that folds the E8M0 scale into the exponent -- no separate
dequant pass and no fp8e4nv in the signature. One program covers
BLOCK_H heads so each KV tile is read once instead of once per head.
Prefill takes the same kernel: on a fallback architecture the sparse
prefill branch is skipped, so both phases share the decode path. The
FlashInfer default is also repointed, since it is CUTLASS built for
SM120 only and every architecture arriving here does so precisely
because it has no CUDA sparse-decode kernel.
With this the model serves on 8x A100 with CUDA graphs on.
`_apply_attn_sink` and `_merge_partial_attn` did the sink normalisation
and the LSE-weighted merge as plain torch expressions over `out.float()`.
That promotes the entire bf16 output to fp32 as a real tensor: two
output-sized transients for the sink, four for the merge. At 8K tokens x
64 heads x 512 the sink alone tries to allocate 896 MiB, and a tp8
batched run dies with
torch.OutOfMemoryError: Tried to allocate 896.00 MiB
at _apply_attn_sink: (out.float() * w.unsqueeze(-1))
A single request never reaches that size, so this only appears under
batch. The weights themselves genuinely need fp32 (logaddexp, exp), and
that part is unchanged; what was wasteful is materialising the output in
fp32 just to scale it elementwise.
Two small Triton kernels keep the fp32 value in registers instead.
Measured on A100 against the torch form, 1K-16K tokens:
identical results (the same single rounding, max|delta| = 0 vs fp32)
240x less peak memory (4104 -> 17 MiB at 16K tokens)
5.5x faster (7.40 -> 1.35 ms)
An in-place bf16 multiply also removes the transient, but rounds the
weight to bf16 before applying it, which moves relative error from
2.8e-3 to 4.4e-3 -- an unnecessary trade when the fused form keeps fp32.
Note this code predates PR sgl-project#33271 and is untouched by it: the same OOM
is reachable on SM120 at batch, so the fix is not Ampere-specific.
…on path
The FlashMLA sparse CUDA kernels are arch-conditional builds -- the
target list in kernels/aot/cmake/flashmla.cmake is sm_90a, sm_100f and
(with CUDA 13) sm_103a -- so on SM80 both entry points abort:
sparse_decode_fwd RuntimeError: Unsupported architecture for sparse
decode fwd
sparse_prefill_fwd RuntimeError: only supported on SM90a and SM100f
'a'/'f' binaries do not carry forward across compute capabilities, so
the guard is an exact-match list of the architectures that *do* have
the kernel; anything else falls back rather than dispatching into a
kernel that was never built for it. That polarity is what keeps a
future architecture from crashing instead of degrading.
Take the Triton sparse decode kernel from PR sgl-project#33271 (sgl-project/sglang),
which is written for this cache layout: it reads the paged FP8 KV
directly, four E4M3 lanes per int32 load, and decodes them with bit
arithmetic that folds the E8M0 scale into the exponent -- no separate
dequant pass and no fp8e4nv in the signature. One program covers
BLOCK_H heads so each KV tile is read once instead of once per head.
Prefill takes the same kernel: on a fallback architecture the sparse
prefill branch is skipped, so both phases share the decode path. The
FlashInfer default is also repointed, since it is CUTLASS built for
SM120 only and every architecture arriving here does so precisely
because it has no CUDA sparse-decode kernel.
With this the model serves on 8x A100 with CUDA graphs on.
(cherry picked from commit 4e9ad4c)
`_apply_attn_sink` and `_merge_partial_attn` did the sink normalisation
and the LSE-weighted merge as plain torch expressions over `out.float()`.
That promotes the entire bf16 output to fp32 as a real tensor: two
output-sized transients for the sink, four for the merge. At 8K tokens x
64 heads x 512 the sink alone tries to allocate 896 MiB, and a tp8
batched run dies with
torch.OutOfMemoryError: Tried to allocate 896.00 MiB
at _apply_attn_sink: (out.float() * w.unsqueeze(-1))
A single request never reaches that size, so this only appears under
batch. The weights themselves genuinely need fp32 (logaddexp, exp), and
that part is unchanged; what was wasteful is materialising the output in
fp32 just to scale it elementwise.
Two small Triton kernels keep the fp32 value in registers instead.
Measured on A100 against the torch form, 1K-16K tokens:
identical results (the same single rounding, max|delta| = 0 vs fp32)
240x less peak memory (4104 -> 17 MiB at 16K tokens)
5.5x faster (7.40 -> 1.35 ms)
An in-place bf16 multiply also removes the transient, but rounds the
weight to bf16 before applying it, which moves relative error from
2.8e-3 to 4.4e-3 -- an unnecessary trade when the fused form keeps fp32.
Note this code predates PR sgl-project#33271 and is untouched by it: the same OOM
is reachable on SM120 at batch, so the fix is not Ampere-specific.
(cherry picked from commit 97ad98d)
Summary
DeepSeek-V4 cannot serve on SM80. Three independent blockers, all fixed here:
dequantize_k_cache_pagedand the Triton sparse decode kernel carry anfp8e4nvpointer in their signature; Triton on SM80 accepts onlyfp8e4b15andfp8e5.is_sm120_supported(). Those kernels are arch-conditional builds for SM90a and the SM100 family (seekernels/aot/cmake/flashmla.cmake), so every other architecture walks into animport sgl_kernel.flash_mlathat cannot succeed.deepseek_v2.pyclassifies layers by matchingquant_config.get_name()against{awq, awq_marlin, moe_wna16}, then reads.weight.dtype. An auto_round W4A16 checkpoint raises:Past those, the Triton fallback is nominal rather than usable: it launched with
grid=(B, H), so all 64 query heads re-read the same KV — MLA has one KV head, so that is a 64x amplification of the dominant traffic. The C4 indexer separately materialises a[B, S, num_heads]intermediate and scores each row once per head.Modifications
BLOCK_Hheads, KV read once, reused viatl.dot; bit-arithmetic E4M3 with the scale folded into the exponent field; int32 loads.scales, which the substring replace turned into.weight_scale_invs— dropped with a warning, leaving the draft on uninitialised weights and the speculative accept rate silently at zeroAn earlier revision of this branch also folded the indexer's head dimension out before the GEMM. That is withdrawn (reverted in
c698ac13): the C4 indexer applies a per-head ReLU between the dot product and the weighted head sum, so the fold is not an identity — @efschu caught it in review below, and my own verification of it turns out to have compared two implementations that both omitted the ReLU. Everything quoted in this description is measured on the branch as it stands, with the fold gone. The exact-operator replacement — a fused kernel that keeps the per-head ReLU and never materialises the[B,S,H]intermediate — is follow-up work, gated on per-layer instrumentation of real checkpoints (see the thread).Accuracy Tests
8xA800, DeepSeek-V4-Flash-0731 (auto_round W4A16), TP8. Branch checked out on its own, no out-of-tree patches.
The command below is the complete one, including the environment walls this PR does not fix — each is a separate pre-existing issue, and without them a reader cannot reproduce these results (one of them, the fused-projection one, fails silently). Two further walls used to belong in this list — the indexer's torch routing and the fallback's backend choice — but the branch now defaults both by capability on SM8x; the runs below predate that commit and were made with the equivalent env vars set, which selects the identical code paths:
Apples-to-apples against the official API — same 1319 GSM8K questions, same script,
temperature=0, no output cap on either side, answer extraction identical:The 3-question gap is within run-to-run variance of the 8-way concurrent client; an earlier run of the same pair landed exactly equal.
Needle-in-a-haystack, randomised needles, depth 0.5, prefix cache flushed before each length:
(The ~2x wall-clock variance I reported earlier is explained: the second pass was hitting the prefix cache — the 99k tok/s "throughput" in that log was cache, not compute. Hence the flush per length here.)
The last two rows are at
--context-length 1048576 --mem-fraction-static 0.55; 1M is this path's first completed run on this hardware, after the four allocation commits at the end of this branch.Kernel-level (each row states what it is measured against):
Upstream
test/registered/kernels/ops/attention/test_flash_mla_backends.py: 12 passed, 1 skipped. That file could not run on this hardware before this branch un-gated it from compute capability 12.0 — and it caught a real defect the first time it did.SM86 / SM120, kernel and suite level (single consumer cards; neither fits the model, so serving claims remain A800-only):
The consistent skip is the FlashInfer-vs-Triton case. On SM120 it skips not for the hardware reason its decorator suggests, but because
flashinfer-python0.6.12 — the version in bothlmsysorg/sglang:latestand:dev-cu13— ships noflashinfer.sparse_mla_sm120module; as shipped, that case appears unrunnable anywhere (predates this branch). Real inputs captured on the A800 at thefp8_paged_mqa_logitsentry and replayed on the 5090 land within bf16-materialisation distance of an fp64 oracle on both architectures. SM120 serving remains untested.Speed Tests and Profiling
Sparse decode kernel: 320.14 ms -> 7.06 ms, warm-cache median of 20 runs.
Not quoting a speedup multiple: the baseline was taken cold-cache under a different protocol than the 7.06 ms, and dividing a warm number by a cold one overstates it. Re-measuring the baseline under the same protocol is outstanding.
Where the loss surface sits now, by context length: up to 32K the enablement is essentially free of the removed fold (the prefill gains came from the attention kernel, which stands); 128K–256K prefill grows with the indexer's quadratic term (98.1 s and 299.0 s wall for the needle runs above), and 512K–1M more so (1010.8 s and 3710.2 s). #33297 fuses that scoring and takes those two to 410.4 s and 1301.1 s.
Scope
Absorbs #33272, now closed — I had split the packed-layer classification out as a separate quantization concern, then found this branch cannot load a packed checkpoint without it. The indexer sequence-axis chunking overlaps in intent with #33259; whichever lands first is fine, it is here because without it the long-context results are not reproducible from this branch.
Checklist
CI States
Latest PR Test (Base): ❌ Run #30887273929
Latest PR Test (Extra): ❌ Run #30887273580