Skip to content

[feat][DCP] Switch DCP decode to persistent PA + make DCP compatible with MTP - #1746

Merged
valarLip merged 5 commits into
ROCm:mainfrom
yitingw1:atom/dcp_persistent_mtp
Aug 3, 2026
Merged

valarLip merged 5 commits into
ROCm:mainfrom
yitingw1:atom/dcp_persistent_mtp

Conversation

@yitingw1

Copy link
Copy Markdown
Contributor

1. Motivation

  • gfx950 changes to persistent PA, while gfx942 still use non-persistent PA (lack lse=1 persistent kernel)
  • gfx950: make MLA DCP compatible with MTP with BF16. (FP8 is waiting for aiter kernel). gfx942: can not use DCP and MTP simultaneously.

2. Usage

Persistent mode is automatic on gfx950 — no new flag. MTP+DCP: combine existing flags.

# Native
python -m atom.entrypoints.openai_server --model DeepSeek-R1 \
  --kv_cache_dtype bf16 -tp 8 -dcp 8 --method mtp --num-speculative-tokens 1  # or 2 / 3

# vLLM plugin
vllm serve DeepSeek-R1 -tp 8 -dcp 8 --kv-cache-dtype bfloat16 \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

fp8 + DCP + MTP raises NotImplementedError instead of silently miscomputing.

3. Design

Flow — MTP draft loop under DCP

 i == 0:  slot_mapping = slot_mapping[: len(input_ids)]     (FIX #1: grid == q rows)
             │
             ▼
 i >= 1:  kv_indptr += cu_seqlens_q   (GLOBAL, not DCP-aware)
          FIX #2 (dcp>1):
            local_ctx  = get_dcp_local_seq_lens(context_lens, W, rank)
            kv_indptr  = cumsum(local_ctx)        ← LOCAL shard
            kv_indices = regenerate from local kv_indptr
            worker buffers = full rebuild
            slot_mapping[t] = raw_slot[t] if (ctx-1)%W==rank else -1

4. Test

MTP+DCP: native + plugin, -tp8 -dcp8, bf16, --method mtp --num-speculative-tokens {1,2,3}; sustained concurrent load to force bs>1 cudagraph decode, then full gsm8k (no --limit), confirm 0 crashes.

MTP+DCP, native, full gsm8k (1319 questions, bf16, cudagraph):

num_speculative_tokens Crashes flexible strict
1 0 0.9484 0.9462
2 0 0.9515 0.9522
3 0 0.9500 0.9484

MTP+DCP, vLLM plugin (limit=100, not yet full):

num_speculative_tokens Crashes flexible strict
1 0 0.97 0.97
2 0 0.94 0.93
3 0 0.95 0.95

Submission Checklist

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1746 --add-label <label>

@zufayu
zufayu requested a review from jiayyu July 31, 2026 01:18
@valarLip
valarLip merged commit f2f5e0b into ROCm:main Aug 3, 2026
52 of 57 checks passed
whx-sjtu added a commit that referenced this pull request Aug 4, 2026
Resolves against #1746 (DCP persistent PA):

- layer_mla.py: take main's removal of the DCP persistent-mode guard; our
  side only reworded its comment, and the guard is gone because DCP now runs
  persistent on gfx950.
- metadata.py: size the work descriptors with main's persistent_num_heads
  inside our per-query-length worst-case loop, so the DCP head scaling and the
  spec-decode capacity fix both hold.
- metadata.py: run main's DCP-on-gfx942 persistent opt-out before our capacity
  assertion, so the assertion cannot fire on a path that just disabled
  persistent metadata.

Co-authored-by: Cursor <cursoragent@cursor.com>
whx-sjtu added a commit that referenced this pull request Aug 4, 2026
The rationale claimed no aiter decode kernel returns a usable LSE, and pinned
that on the persistent kernel leaving final_lse at FLT_MAX. That is backwards.
Offline comparison against a torch reference showed the persistent kernel's LSE
is correct across batch sizes, and #1746 has since switched DCP decode onto it
precisely because gfx950 emits a per-token LSE for the cross-rank merge. The
kernel that cannot be used is the split-KV one, whose stage2 reduce leaves
final_lse untouched for a request whose valid split count collapses.

So causal drafting is a scope decision -- one drafting semantics shared with
ATOM's native DSpark, no merge to build -- not a kernel limitation, and
non-causal is a buildable follow-up. Say that instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
whx-sjtu added a commit that referenced this pull request Aug 4, 2026
The rationale claimed no aiter decode kernel returns a usable LSE, and pinned
that on the persistent kernel leaving final_lse at FLT_MAX. That is backwards.
Offline comparison against a torch reference showed the persistent kernel's LSE
is correct across batch sizes, and #1746 has since switched DCP decode onto it
precisely because gfx950 emits a per-token LSE for the cross-rank merge. The
kernel that cannot be used is the split-KV one, whose stage2 reduce leaves
final_lse untouched for a request whose valid split count collapses.

So causal drafting is a scope decision -- one drafting semantics shared with
ATOM's native DSpark, no merge to build -- not a kernel limitation, and
non-causal is a buildable follow-up. Say that instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
whx-sjtu added a commit that referenced this pull request Aug 5, 2026
Numbers now come from a run against the rebased tree rather than the one taken
before #1746 (DCP persistent PA) was integrated: 0.9545 flexible / 0.9538
strict, acceptance 3.777. All inside the error bar of the earlier 0.9522, as
expected for a change that only re-sizes work descriptors.

Also note the TimeoutError trap. A run whose requests hit the local-completions
client timeout read 0.9424 while its acceptance figures stayed normal -- the
retries re-generate under a different batch composition, and greedy chain of
thought is sensitive enough to flip on that. Worth checking before anyone
reports a regression from these numbers.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants