Skip to content

[Perf] [NVIDIA] dflash: Split SWA MLA value columns on GB300 - #46471

Draft
maocheng23 wants to merge 9 commits into
sgl-project:mainfrom
maocheng23:maocheng/dflash-swa-mla-value-split
Draft

maocheng23 wants to merge 9 commits into
sgl-project:mainfrom
maocheng23:maocheng/dflash-swa-mla-value-split

Conversation

@maocheng23

@maocheng23 maocheng23 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Causal DFlash MLA drafts can expose too little parallel work during short-block SWA attention. Split the 512 output value columns across CTAs while retaining the native sequential softmax recurrence.

Stacked on #44676 at 2c95c09bfa9cb13206f005154d4e9c75ad1c5e28. This PR targets main because the dependency branch lives in the author's fork. Merge #44676 first, then rebase this PR. The incremental diff is limited to five files.

Modifications

  • Add a Triton value-column split kernel for BF16 absorbed MLA (QK=576, V=512), causal window 4096 and draft block 8. Each CTA repeats QK/softmax for its value columns; prefix/current-token loops retain native M16/N64 tiles, BF16 probability conversion and FP32 accumulation. No split-KV reduction or scratch buffer is needed.
  • Add SGLANG_OPT_DFLASH_SWA_MLA_VALUE_SPLIT=1, off by default. Dispatch is restricted to GB300 (SM103), DFlash draft MLA on Triton, DCP1, and these measured (local heads, batch) -> value partitions: (12,1) -> 8, (24,1) -> 4, (12,4) -> 2. Other shapes and unsupported attention options retain native dispatch, including deterministic mode, custom masks, sinks and scaling.
  • Add CPU dispatch/fallback coverage and GB300 tests for native equality, an independent FP32 reference, ragged/tile/window boundaries, strides, causal masking and CUDA graph metadata changes. Register the GPU suite on the existing base-c / 2-gpu-gb300 lane.

Enable before server startup:

export SGLANG_OPT_DFLASH_SWA_MLA_VALUE_SPLIT=1
# Use with the existing DFlash MLA launch arguments:
# --speculative-draft-attention-backend triton
# --speculative-draft-kv-cache-dtype bf16

The draft export must already describe causal sliding attention with window 4096 and block 8; the environment flag does not change model semantics.

Accuracy Tests

Current stacked revision f87521086: 83 CPU tests and 2,091 subtests passed, including all 55 new routing/fallback cases plus the parent MLA, hook and geometry suites. CUDA remained uninitialized. Changed-file pre-commit, git diff --check, and actual CI registry collection pass. Direct CI-style python file.py -f invocation also passes: 55 routing cases; the GPU test entry point passes its metadata case and skips 30 numerical cases when GPUs are hidden (not counted as GPU validation).

Historical GPU validation was run on the implementation before this rebase, based on #44676 at ae826bdd56d84088052dcb709d161f046947879e, using GB300, Torch 2.14.1+cu130, Triton 3.8.0, Transformers 5.19.0, FlashInfer 0.7.2 and sglang-kernel 0.4.10:

  • 31 tests in the GPU file passed: 30 GB300 numerical/replay cases plus one argument-validation case. Both native and optimized outputs were checked against independent FP32 math, and against each other with bitwise equality.
  • The attention matrix passed 15 timed shapes and 27 boundary executions.
  • One real-weight five-layer TP4 draft-backbone fixture produced bitwise-equal native/optimized outputs on all four ranks in eager and graph modes. A native/native control also passed.

The Triton JIT function and its launch arguments are unchanged by the rebase/metadata cleanup. The dependency's latest revision changes DFlash allocator/KV-write behavior, so these historical GPU results are not presented as a new GPU validation of this revision. Draft until the current stacked revision has a fresh GB300 run. This does not establish cross-TP or cross-batch invariance.

CPU reproduction (the non-existent device index hides all GPUs on the test host):

SGLANG_USE_CPU_ENGINE=1 CUDA_VISIBLE_DEVICES=999 python -m pytest -q \
  test/registered/unit/layers/attention/test_dflash_swa_mla_routing.py \
  test/registered/unit/spec/test_dflash_mla.py \
  test/registered/unit/spec/test_oot_dflash_hooks.py \
  test/registered/unit/layers/attention/test_mla_decode_geometry.py

GB300 reproduction:

python -m pytest -q test/registered/kernels/ops/attention/test_dflash_swa_mla.py

Speed Tests and Profiling

The following measurements use the historical GPU-validated revision above. Fixed real-weight five-layer draft backbone, TP4 (24 local heads), BF16, batch 1, prefix 4095, block 8, window 4096:

Execution Native ms Value split ms Speedup
CUDA graph 3.679992 3.076570 1.196x
Eager 9.110528 8.888128 1.025x

Median of three repetitions; each repetition times 20 forwards and uses the maximum rank mean CUDA latency. The graph result is 16.4% lower latency. Timing includes the five draft layers, current-block cache writes and TP communication. It excludes target execution, context preparation, embedding lookup, logits/selection and scheduling. This is one backbone fixture, not end-to-end serving tokens/s, acceptance length or answer accuracy.

Single-GPU attention shapes at prefix 4095:

Local heads Batch Partitions Native us Value split us Speedup
12 1 8 530.454 358.986 1.478x
24 1 4 535.472 373.219 1.435x
12 4 2 542.589 400.409 1.355x

The 12-head rows are attention shapes, not a distributed TP8 serving benchmark. Both paths use the existing SWA prefix truncation.

Checklist

  • Changed-file pre-commit and diff checks.
  • CPU and GPU tests added; GB300 CI lane selected.
  • Activation and supported shapes documented above and in the kernel interface.
  • Historical numerical and timing results reported with their scope and revision.
  • Fresh GB300 validation on this stacked revision before marking ready.

CI States

Latest PR Test (Base): ❌ Run #40235600475
Latest PR Test (Extra): ❌ Run #40235600339
Latest PR Test (AMD ROCm 10): ❌ Run #40235600608

maocheng23 and others added 9 commits October 22, 2026 02:00
…ntion

Rebase of the PR's 11 commits (through 983a911) onto main as one commit.
main deleted tokenspeed_mla_backend.py (sgl-project#44803), so the PR's two edits to
that file are dropped here; the remaining tokenspeed_mla references are
replaced in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…peed_mla

main removed the tokenspeed_mla backend (sgl-project#44803), so MLA drafts now run on
the default MLA draft backend: flashinfer on CUDA, triton on ROCm and XPU.

- Drop tokenspeed_mla from DRAFT_ATTENTION_BACKEND_CHOICES and the DFlash
  and draft-worker rules that kept it off GQA drafts. A target's MLA-only
  backend (trtllm_mla, cutedsl_mla) is outside the draft choices and falls
  back as before.
- An MLA draft leaving trtllm_mha takes the platform fallback. main now
  sends non-MLA drafts to fa4 on SM100, which the MLA draft layer is not
  validated on.
- Test triton and flashinfer only in the GPU backend test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-ups:

- FlashInfer MLA reads the mask from the layers' attn_type instead of a
  DFlash flag on ModelConfig: per layer for the ragged run, and once from
  the model's attention layers for the paged plans, which fix the mask per
  batch. Mixed ENCODER_ONLY and causal layers are rejected. The backend no
  longer knows about DFlash, and the test kits drop is_dflash_mla_draft.
- The attention mode is parsed once into DFlashDraftConfig.is_mla.
  DFlashDraftModel picks the layer attention class from it and passes the
  class to its decoder layers; ModelConfig keeps the flag local.
- kv_b_proj is absorbed in DFlashDraftModel.post_load_weights, which
  load_weights calls and which the dummy, sharded-state and remote loaders
  call directly; the kv_b_proj weight-loader hook is gone. The fused a_proj
  loader also accepts an already-fused weight, like stacked linear loaders.
- Sliding MLA draft layers check for triton in the layer, as DFlashAttention
  does for attention sinks, instead of in the FlashInfer backend.
- The DFlash layer causality helpers move back to dflash.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_stacked_ctx_kv_params now returns None for drafts without fused
context-KV support (MLA), so the SimpleNamespace models in this test
raised AttributeError in every case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The launch hook re-derived the backend a DSpark draft inherits and loaded
the draft config to keep MLA drafts off trtllm_mha. DSpark already chooses
a layout-specific backend when it builds the draft (dsv4 for DeepSeek-V4
drafts), so draft_attention_backend_override now returns flashinfer for an
MLA draft whose inherited backend, from resolve_draft_worker_attention_backend,
is trtllm_mha. The draft config is read only on that path, and the launch
hook keeps only the DFlash rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop docstrings on private helpers and methods whose names say what they
do, and cut multi-line comments that restated the next lines down to the
constraint each one carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One rule, _keep_mla_draft_off_trtllm_mha, keeps MLA drafts off trtllm_mha
  for both DFlash and DSpark. DSpark publishes its result at launch again
  instead of overriding it in the worker, so the KV planners, which read the
  published draft backend, see the backend the draft runs. It reuses the
  draft config _handle_dspark already reads for gamma, which now carries
  is_mla; dspark_worker_v2.py returns to main.
- FlashInfer MLA's chunked-prefix MHA runner takes its masks from the layer
  (per call) and the backend's planned causality (per plan), like the
  ragged and paged paths.
- DFlashDraftModel builds one BumpAllocator per forward for its MLA layers,
  sized as in DeepseekV2Model, instead of one per layer per forward.
- The MLA prefix-valid write masks rows past commit_lens like the MHA pool:
  CUDA/HIP run the MHA pool's tiled kernel unchanged, with the latent row
  passed as both K and V, and other platforms keep only the valid rows. It
  no longer relies on the MLA write kernels skipping loc 0, which not every
  MLA write path does. A test checks nothing past commit_lens is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant