Skip to content

[PCP] Fewer V4 prefill gathers and fix PCP+TBO accuracy - #2789

Merged
valarLip merged 23 commits into
mainfrom
yitingw1/pcp_opt
Nov 14, 2026
Merged

valarLip merged 23 commits into
mainfrom
yitingw1/pcp_opt

Conversation

@yitingw1

@yitingw1 yitingw1 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Requires ROCm/aiter#7149 (6-rank groups).

Size. This PR is 29 files / ~1.9K lines, above the usual split threshold. Most of the growth after the first version answers two review rounds: the paged-prefill metadata rework, grouped gathers, DSpark local quantization and the private TBO mark primitive touch the same builder and gather paths, so splitting by comment would leave intermediate states that fail the PCP+TBO accuracy check (gsm8k 0.68 on main). Tests account for about half of the diff.

Motivation

At small prefill batches PCP on DeepSeek-V4-Pro is host-bound: a step at -tp 1 -pcp 8 launches ~3,200 GPU ops, and the CPU, not the GPU, sets its length (~190 ms of host time per step, with the GPU idle for up to 126 ms of it). Every attention layer issued 3–5 separate torch.distributed all-gathers, and the PCP reindex blocked on a .item() per ragged field. DSpark gathered three full aux hidden states per token and then projected every row on every rank. Validating this exposed a PCP+TBO accuracy bug on main.

Change Kind Effect
Group the attention-side PCP gathers Perf 397 → 245 collectives per prefill step; TTFT −9.2 % (ISL 2K) / −10.3 % (ISL 8K) at -pcp 8, concurrency 1
Per-rank paged-prefill metadata Perf no reindex kernels, no .item(); TTFT −10 % at ISL 16K
DSpark context KV from each rank's own rows Perf gathers 640 B (fp8) / 1 KiB (bf16) per token per stage instead of ~42 KiB/token; projection and quantization 1/pcp per rank
Hold TBO comm-stream outputs for the compute stream Correctness PCP+TBO gsm8k 0.68 → 0.965

Technical Details

1. Grouped gathers

pcp_allgather_rerange_many gathers several tensors in one all_gatherv (aiter's custom all-gather: one launch, one barrier; PyNCCL group past its size limit) and restores token order per tensor. Each layer now issues one gather for the two compressor inputs (Compressor.project in _launch_compressors, gathered, then Compressor.forward(projected)) and one for the extend K with its positions:

Attention-side collectives per layer (fp8 KV) main this PR
CSA (30 layers): main + indexer compressor, k_packed, k_rope, positions 5 2
HCA (31 layers): compressor, k_packed, k_rope, positions 4 2

MoE's gather / reduce-scatter (2 per layer) are unchanged. Positions are gathered rather than read off the forward context because an MTP draft pass runs at the target's positions + 1. gather_rows (now in atom/distributed/comm_utils.py, shared with MoE's DP / PCP / comm-fused gathers) carries one-byte dtypes as uint8 and everything else as is: aiter's grouped all_gatherv copies bytes, and only its PyNCCL fallback lacks fp8 / e8m0 (MoE used to dress int32 ids as fp32, a constraint of the per-tensor custom all-gather it no longer calls). At group sizes aiter's custom all-gatherv has no kernel for (6), this PR requires ROCm/aiter#7149, which makes aiter decline them and fall back to NCCL.

2. Per-rank paged-prefill metadata

main built whole-batch paged-prefill metadata and reindexed it per rank (pcp_reindex_ragged, sized by an .item() per ragged field). Each rank now stages its own indptrs from the builder's numpy arrays through the CpuGpuBuffer publication group and launches the index kernel only over the tokens it owns; the host mirror, pad / reindex kernels, upload_numpy and the csa_indices.fill_(-1) are gone. Under PCP+TBO each request group stages its own ub{i}_ rows the same way. V4-Pro -tp 1 -pcp 8, ISL 16K, mbt 4096: TTFT −9.8 % (conc 1) / −11.1 % (conc 8); with --enable-tbo prefill −11.0 % / −8.5 % (includes the grouped gathers of §1).

3. DSpark context KV

The projection from aux hidden states to window KV (main_proj + norm, then per stage wqkv_a + norm + RoPE + fake-quant) is row by row, so each rank now projects only its own rows at their own positions, gathers the projected KV, and writes as before. DSparkLayer.write_context_kv is split into context_kv_rows + write_context_kv_rows. The V4 DSpark draft overrides write_split_context_kv: it projects and quantizes its own rows into the fp8 2buff layout and sends every layer's rows in one grouped gather; other flavors keep the base write_split_context_kv, which gathers the aux states. pcp_round_robin_split now returns unit strides: .contiguous() left a one-row shard (a 2-token chunk at pcp 8) at row stride 8, which aiter's cached-positions RoPE asserts on.

4. PCP+TBO: comm-stream outputs read on the compute stream

Under TBO a PCP gather runs on the comm stream, so its output is allocated there, while the compute stream reads it. Once the forward frees it, the caching allocator hands that memory straight back to the comm stream, and the comm stream waits only on its own ubatch's compute (ubatching.py, one shared comm and compute stream): the other ubatch's next gather of the same layer can overwrite the extend K before this ubatch's attention has read it.

-pcp 8 --enable-tbo, gsm8k 200 q gsm8k
main (two nodes) 0.69 / 0.675
this PR, extend-K gathers on main's path / K only / positions only 0.71 / 0.70 / 0.975
… K + positions on main's path + record_stream 0.965
this PR (grouped gathers alone: 0.96) 0.965; final commit 0.97

tbo_on_comm(fn, *args) now runs every TBO comm phase whose output the caching allocator owns: yield, fn on the comm stream, switch back, and record_stream the TBO context's compute stream on every tensor fn returns (nested lists / tuples included). That is the extend K and positions, the compressor inputs, FusedMoE's PCP and DP routed-rows gathers and reduce-scatters, and pure TP's overlapped tbo_all_reduce (-tp 8 --enable-tbo prefill: gsm8k 0.96). Mori keeps its hand-written phases: dispatch / combine return views of each ubatch's own mori buffers, which the allocator never hands out. The grouped gathers passed main's accuracy only by timing (0.96 without the hold), so gsm8k cannot pin the call sites; a test fails instead if any of these writes a comm phase out by hand. Under a graph the V4 compressors read their PCP-gathered inputs on side streams, so those are held for the side streams the same way.

Test Plan

pytest tests/test_pcp_ops.py tests/test_tbo_on_comm.py tests/test_tbo_mark_used_on_compute.py \
  tests/model_ops/test_dp_gather_routed_rows.py \
  tests/test_dspark.py tests/attentions/deepseek_v41/test_tbo.py tests/model_ops/test_indexer_rows.py ...

# DeepSeek-V4-Pro(-DSpark), MI355X: -tp 1 -pcp 8, kv fp8, index cache fp4, prefix caching, mbt 512,
# --level 3 --cudagraph-mode FULL; gsm8k 5-shot. Perf: no spec, mbt 16384, random prompts, OSL 1,
# concurrency 1, two main/PR pairs (second in reverse order), warmed up 5 min.
python -m atom.benchmarks.benchmark_serving --dataset-name random --random-input-len 2048 \
  --random-output-len 1 --max-concurrency 1 --num-prompts 1500 --ignore-eos

Test Result

Unit tests: 700 passed, 12 skipped; ruff and black clean. Each new test fails with its change reverted (22 mutations: 17 on the trees before this review round, 5 on the final one). aiter's test_custom_allgatherv.py -t 6 passes with its fix and fails with the world-size gates forced open; -t 8 unchanged.

-tp 1 -pcp 8 unless noted Result
DSpark k=5, fp8 KV, 200 q gsm8k 0.97, acceptance 63.35 % (#2125's smoke: 63.35 %)
DSpark k=5, bf16 KV, 100 q 0.98, acceptance 61.66 %
MTP k=3, 200 q 0.975, acceptance 65.96 %
TBO, 200 q 0.965; final commit 0.97 (main 0.69 / 0.675)
-tp 4 -pcp 2, 20K-token prompts (PyNCCL fallback for fp8) + 200 q, final commit 4/4 served, 0.97
-tp 1 -pcp 6 with ROCm/aiter#7149 (no ATOM 6-rank special case), 100 q 0.97
-tp 8 --enable-tbo prefill, mbt 16384, TBO from 1024 tokens (64 of 80 prefill steps in a profiled probe ran TBO), 200 q 0.96
Prefill, concurrency 1 (two pairs) main this PR
Mean TTFT, ISL 2048 367.2 / 366.2 ms 335.0 / 331.3 ms −9.2 %
Mean TTFT, ISL 8192 389.8 / 388.6 ms 349.8 / 348.6 ms −10.3 %
Collectives per step (trace) 397 245
Host time per step (trace) 189–193 ms 163–168 ms

Known Limitations

  1. -pcp with speculative decoding and --enable-tbo stays refused at startup ([PCP] Fix sampled-id sync, DSpark aux gather and FP4 indexer padding #2125): PCP+TBO stripes each request group on its own, while DSpark's aux gather and context-KV projection, like the draft prefill split, read the batch as one stripe. Lifting that needs all three to follow the groups.
  2. Under --enable-tbo all the TP all-reduce and DP collectives now record_stream while decode graphs capture. Measured at -tp 8 --enable-tbo all (DSV4-Pro, FULL graphs, 8 of 10 buckets recorded split): 1952 holds per rank during capture, 135 MiB held in total, and the graph pool grows by 0.00 GB with or without the hold; KV blocks unchanged.
  3. The custom all-gather copies each input into its registered buffer (D2D memcpys per step 244 → 518); still a net win at the sizes measured.
  4. Performance was measured only at -tp 1 -pcp 8 (concurrency 1 and 8, with and without --enable-tbo prefill), where every attention gather takes the custom all-gather; -tp 4 -pcp 2 was checked for correctness only.
  5. EAGLE3 under PCP on a DSA target gathers the aux states twice (EAGLE3 strips them off the target's output, which DeepseekV2ForCausalLM has already gathered); pre-existing, for a separate fix.
  6. tests/test_fused_compress_ragged.py fails 4 cases on this image on main too; the PCP unit tests import aiter, so a CI runner without it skips them.

Submission Checklist

🤖 Generated with Claude Code

yitingw1 and others added 7 commits October 28, 2026 08:36
Each V4 attention layer under PCP prefill issued 3-5 separate all-gathers
(main compressor input, indexer compressor input, extend K -- two under fp8
KV -- and positions), each through torch.distributed. Now it issues two:

- `pcp_allgather_rerange_many` gathers several tensors in one grouped
  `all_gatherv` and restores token order per tensor. One-byte dtypes travel
  as uint8, since past aiter's custom all-gather limit the call falls back to
  PyNCCL, which has no fp8 dtype.
- The extend K is gathered together with the positions its SWA write
  addresses the ring by. Positions are gathered rather than read off the
  forward context: an MTP draft pass runs at the target's positions + 1.
- `_launch_compressors` projects both compressors' inputs and gathers them in
  one collective; `Compressor.forward(combined=)` takes the result.

V4-Pro (30 CSA + 31 HCA layers): 397 -> 245 collectives per prefill step.

`pcp_reindex_ragged` sized its output with a blocking `.item()`, four times
per prefill step (sixteen under PCP+TBO). The builder already computes the
four prefill indptrs in numpy; it now keeps them on the metadata
(`paged_prefill_indptr_cpu`) and each rank's local indptr is computed on the
host (`pcp_local_indptr`), in both the full-batch and the per-request-group
paths.

`pcp_round_robin_split` returns unit strides: `.contiguous()` leaves a
one-row shard at row stride pcp_size, which aiter's cached-positions RoPE
rejects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After a PCP prefill the aux hooks hold only this rank's round-robin rows.
`compute_draft_kv` all-gathered the three aux hidden states (~42 KiB a row)
and every rank then projected all rows into window KV, the same work on
every rank. The projection (main_proj + norm, then per stage wqkv_a + norm +
RoPE + fake-quant) works row by row, so project only the local rows at their
own positions, gather the projected KV rows (1 KiB a row a stage, one
grouped collective for all stages), and write them as before.

`DSparkLayer.write_context_kv` is split into `context_kv_rows` and
`write_context_kv_rows`. A draft opts in with `context_kv_by_rows`; only V4
DSpark does, and any other draft keeps gathering the aux states.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…unts

aiter's custom all-reduce, which `all_gatherv` tries first, runs at 6
ranks, but its all-gatherv kernel exists for 2, 4 and 8 only and throws
otherwise. The attention-side gathers went through torch.distributed before
the grouped gather, so PCP 6 worked; take one tensor at a time there again.

Also assert that the grouped inputs share a row count (a mismatch threw on
the custom path but passed on PyNCCL), and fix two review nits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The DSpark drafter branches on the KV cache dtype while tracing
(`dspark_fp8_planned`), but `Config.compute_hash` left it out -- the known
limitation #2566 deferred. A DSV4-Pro DSpark run with `--kv-cache-dtype
bf16` after an fp8 one in the same container hashed to the same directory,
loaded the fp8 drafter graph and died in warmup on every rank
(`IndexError: list index out of range` on the split graph's outputs).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…eads them

Under TBO a PCP gather runs on the comm stream, so its output is allocated
there, while the compute stream reads it. When the forward frees it, the
caching allocator hands the memory straight back to the comm stream, and the
comm stream waits only on its own ubatch's compute: the other ubatch's next
collective can overwrite it before this ubatch's kernels have read it.

On main this corrupts the gathered extend K. DSV4-Pro, pcp8 + `--enable-tbo`,
gsm8k (200 q): main 0.69 / 0.675 on two nodes. With only the extend-K gathers
on the main code path: 0.71 (K alone: 0.70, positions alone: 0.975); the same
plus `record_stream` on the gathered tensors: 0.965. The grouped gathers of
this branch have the same shape, and score 0.96 only by timing.

`tbo_mark_used_on_compute` records the compute stream on such tensors after
`tbo_switch_to_compute_sync`, for the attention-side gathers (extend K and
positions, the compressor inputs) and FusedMoE's PCP routed-rows gather and
reduce-scatter. The DP and mori TBO paths have the same shape and are left
alone here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The previous commit marked the comm-stream outputs by hand at four call
sites, and nothing would notice one going missing: this branch scores 0.96
under TBO with or without the marks. `tbo_on_comm(fn, *args)` makes the
comm phase one call -- yield, `fn` on the comm stream, switch back, hold
every tensor it returns -- so a phase cannot skip the hold, and a test
fails if V4 attention or `FusedMoE._run_experts` writes a phase out by hand.

`tbo_mark_used_on_compute` now takes the compute stream from the TBO
context instead of the current stream, so it no longer depends on being
called after the switch back. `pcp_aux_rows` documents that it describes the
base layer-hook capture; EAGLE3 strips aux off the target's output instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2789 --add-label <label>

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — I reviewed HEAD 9d1db12b2. The core changes check out: the grouped gather, host-side indptr, record_stream in tbo_on_comm, and per-row DSpark KV projection. I also confirmed aiter's custom_all_gatherv returns a freshly allocated output (no aliasing with the registered buffer), the GPU/CPU kv_indptr_* copies are written in one place (_build_paged_prefill_meta) and stay consistent, and dummy runs don't enter the new DSpark branch.

Six inline comments. The two most worth acting on:

  1. Under PCP + TBO, DSpark's context KV is reconstructed as one global stripe while the buffer is per-ubatch and padded per group, so draft KV lands on wrong rows (dspark_proposer.py:645).
  2. The comm-stream lifetime hazard this PR fixes via tbo_on_comm remains on the DP / TP-allreduce / MORI paths that still hand-write yield → collective → switch (ubatching.py:615).

Comment thread atom/spec_decode/dspark_proposer.py Resolved
Comment thread atom/utils/tbo/ubatching.py Resolved
Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/models/deepseek_v4/__init__.py Resolved
Comment thread tests/test_tbo_mark_used_on_compute.py Resolved
yitingw1 and others added 2 commits October 29, 2026 08:55
… gathers

Review follow-ups on #2789:
- tbo_all_reduce and FusedMoE's DP gather / reduce-scatter run through
  tbo_on_comm, like the PCP comm phases, so their comm-stream outputs are
  held for the compute stream; the hold walks nested lists and tuples.
  Mori is unchanged: it returns views of its own per-ubatch buffers.
- Under a graph the V4 compressors read their PCP-gathered inputs on side
  streams; hold them for those streams.
- gather_rows moves to pcp_utils and carries one-byte dtypes as uint8 and
  the rest as is (aiter's all_gatherv copies bytes; only its PyNCCL
  fallback lacks fp8 / e8m0). MoE's DP, PCP and comm-fused gathers and the
  PCP grouped gather share it.
- The allocator-reuse positive control skips under expandable_segments.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tbo_all_reduce imports it ahead of the TBO branch, and the non-GPU CI's
aiter has no aiter.dist.communication_op.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ROCm/aiter#7149 makes custom_all_gatherv decline group sizes it has no
kernel for, so all_gatherv falls back to NCCL there. The grouped gather
now takes the one-collective path at every PCP size, 6 included.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@yitingw1

Copy link
Copy Markdown
Contributor Author

@valar-cop review this pr

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed

any perf number for tp>1 / pp+pcp? and a gpu run of pcp+tbo and pcp+dspark?

ci green.

Comment thread atom/spec_decode/dspark_proposer.py Resolved
_apply_pcp_reindex now derives each rank's indptr from the host copy the
builder publishes (paged_prefill_indptr_cpu); the test builds its metadata
by hand, so it sets that copy beside kv_indptr_extend.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@yitingw1

yitingw1 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Re the review: perf was measured at -tp 1 -pcp 8 only (Known Limitations); -tp 4 -pcp 2 was checked for correctness (20K-token prompts, gsm8k 0.97), and PP+PCP was not run because there is no PP+PCP deployment yet; we will test it when one comes up. The GPU runs are in the Test Result table: PCP+TBO gsm8k 0.965 (0.97 on the final commit), and PCP+DSpark 0.97 with fp8 KV and 0.98 with bf16 KV. PCP+DSpark+TBO is refused at startup.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please address the pcp 6 validation and dspark collective..

please re-run ATOM Test for fd0893d. any pcp+graph perf number? gsm8k?

ci green.

Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/models/deepseek_v4/__init__.py Resolved
Comment thread atom/spec_decode/dspark_proposer.py Resolved
The file failed 20 of 22 cases on main with RUN_H2D_GPU_TESTS=1, all from
fixtures that fell behind the code:
- set_pcp patched pcp_round_robin_query_indices (removed in #2688); the
  builder reads get_pcp_rank.
- make_runner lacked forward_vars["batch_info"], config.max_num_seqs and
  the builder's csa_layers, max_model_len_idx, _local_heads and
  index_logits_workspace.
- The padded decode case modelled the bucket width through
  UBatchWrapper._decode_ub_running_bs; since #2692 the split comes from
  _prepare_ubatch_decode's split_bs, so the test passes the bucket.
- prepare_prefill was called with a total_tokens_num the fake batch lacks.

22/22 pass on this branch and on its merge base.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed, the pcp 6 fallback removal is risky..

any perf number at tp>1? and the graph-pool footprint from record_stream is still not measured..

ci green.

Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/models/deepseek_v4/__init__.py Resolved
Both PCP reindex paths (rank and PCP+TBO group) compute local offsets from
the host copy and reindex the GPU copy; a length mismatch would silently
reindex against a different query set. Clarify that after either reindex
both copies describe the same rank- or group-local queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitingw1

yitingw1 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Pushed fd912b7 (host/device indptr length asserts + docstring); replies inline. On the summary points:

  • Tests / CI: CI was green on 319ffe1; fd912b7 only adds two asserts and a docstring line. PCP + graph on fd912b7: -tp 1 -pcp 8 gsm8k 5-shot (200 q) 0.975 with DSpark k=5 (acceptance 63.7%), 0.96 with --enable-tbo prefill.
  • TP>1 perf: not measured on purpose. TP>1 combined with PCP>1 is slower than either alone and is not deployed, so we only keep it correct (-tp 4 -pcp 2 gsm8k 0.97).
  • Graph pool footprint of record_stream: measured at -tp 8 --enable-tbo all (DSV4-Pro, FULL graphs, 8 of 10 buckets recorded split), same tree with and without the hold. During capture each rank holds 1952 tensors (135 MiB in total); the graph pool grows by 0.00 GB in both arms and KV blocks are identical. Known Limitation 2 updated.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-read at c889ff9, by reading the code and running the 7 affected CPU test files (163 pass). Thanks for the rounds since 10-29. The tbo_on_comm sweep, the unified gather_rows, the side-stream record_stream, and the graph-pool measurement all read right. I'm not repeating what's been answered: the PCP+DSpark+TBO startup refusal, -pcp 6 on aiter#7149, and the indptr length asserts.

Performance (the three I'd most like addressed)

This PR's point is to stop every rank redoing the whole prefill. Three places still pay per step for work that's already known on the host, or that each rank repeats:

  1. pcp_reindex_ragged uploads with upload_numpy every forward, per field (pcp_utils.py:322). upload_numpy's own docstring says "A buffer uploaded every forward should be a CpuGpuBuffer". That's 4 pageable H2Ds per prefill step, and 12+ under PCP+TBO, plus the arange / searchsorted / gather kernels behind each.
  2. The PCP+TBO group path reindexes a contiguous range with the general machinery (deepseek_v4_attn.py:3333). A group owns [gts, gte). For a contiguous range, the indices are a view indices[cpu[gts]:cpu[gte]] and the indptr is cpu[gts:gte+1] - cpu[gts]: no kernel, no upload. Today each group pays 3–4 kernels and an upload per field.
  3. DSpark gathers bf16 KV rows and then every rank quantizes all n rows (dspark_proposer.py:658). The quantization is per row, so quantizing the local rows first and gathering the 2buff result cuts the wire (by my estimate about 640 B per row instead of 1 KiB) and removes the W-fold repeated quantization.

Items 1 and 2 share a root fix, which is the first inline comment on deepseek_v4_attn.py:406: build the rank- or group-local indptr in the builder's numpy pass and upload it once.

Structure

The rest is inline: the proposer reaching into the draft model, gather_rows' home, the two all-gather paths, Compressor's two modes, and the TBO comm-phase API.

Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/spec_decode/dspark_proposer.py Resolved
Comment thread atom/spec_decode/dspark_proposer.py Resolved
Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/distributed/pcp_utils.py Resolved
Comment thread atom/models/deepseek_v4/__init__.py Resolved
Comment thread atom/utils/tbo/ubatching.py Resolved
Comment thread tests/test_tbo_on_comm.py Resolved
@valar-cop

valar-cop commented Nov 2, 2026 •

Copy link
Copy Markdown

yitingw1 and others added 2 commits November 10, 2026 05:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inish split

Port the per-rank paged-prefill build onto main's stage_prefill /
finish_prefill structure (#3136):
- Under PCP, stage_prefill stages this rank's own rows (pcp_split) and
  finish_prefill reindexes them; pad query segments get length 0 and the
  per-step indptr/indices fill and reindex helpers are gone.
- Under PCP+TBO each group stages and finishes its own ub{i}_ inputs
  (_stage_pcp_group / _finish_pcp_group) with group-rebased cu_seqlens_q,
  block_tables and replay ranges.
- _pcp_split_rows takes scheduled and running token counts separately.
- Move shared comm helpers into atom/distributed/comm_utils.py.
- Tests: rank segments with running > scheduled, PP relay _aux_blocks in
  the DSpark drafter stubs.

gsm8k 5-shot 200q, V4-Pro -tp 1 -pcp 8: 0.965, +TBO prefill 0.970,
DSpark k=5 0.975 (acceptance 63.76%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yitingw1 and others added 4 commits November 10, 2026 11:54
…n merge

main's #3072 keys the compile cache on kv_cache_dtype as this branch's
222feeb did; the merge kept both, repeating the keyword in two test
fixtures (SyntaxError at collection) and the hash factor in Config.
Keep main's version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve dspark_proposer.compute_draft_kv against #3091's served_by_round
early return, and fix three tests this branch broke:
- Drafter._aux_blocks becomes the module-level aux_blocks(drafter), so
  SimpleNamespace stubs with only _aux_buffers/_relayed_aux keep working
  (test_pp_aux_relay).
- compute_draft_kv keys the non-PCP path on aux_for() returning None, as
  main does; only the PCP split path reads _captures_aux
  (test_deepseek_v41_mono_hi_served_step).
- test_sp_moe_dispatch patches tbo_on_comm instead of the removed
  moe.tbo_active.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_aux_rows

Non-PCP steps no longer query the PCP group, which is a MagicMock under the
aiter stub and broke test_deepseek_v41_mono_hi_served_step in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itly

The 1110 port defaulted cu_q_per_seq_gpu and block_tables_gpu to the
whole batch's, so a PCP+TBO group caller that forgot them would build
indices from the wrong rows without an error. Make both keyword-only
and required; the whole-batch caller passes its own rows. Also note in
pcp_aux_rows that target_pcp_split implies PCP is enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitingw1

Copy link
Copy Markdown
Contributor Author

@valarLip Thanks for the second pass. All ten inline comments are addressed in c9b9de7 and the follow-ups, ported onto current main in 1e582ae, with CI fixes in aee343c and 5e4e8bb, and c5b86da making the group-level cu_q / block_tables arguments required. The biggest change is in the paged-prefill metadata: instead of building whole-batch metadata and reindexing it, each rank and each TBO group now stages its own indptrs and launches the index kernel only over the tokens it owns. Gathers are grouped, DSpark quantizes before gathering, and the TBO mark primitive is private behind tbo_on_comm. E2E gsm8k after the port: pcp8_tbo 0.970, pcp8 0.965, dspark 0.975 (acceptance 63.76%). TTFT for the metadata rework plus grouped gathers: −10 % at ISL 16K (pcp 8). Not done: a few single-tensor gathers that aren't back-to-back are left as is.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Drop §5 "Compile cache key" from the PR description. kv_cache_dtype is already in Config.compute_hash on main via #3072 (atom/config.py:3338), and after aee343c this PR no longer touches atom/config.py. Remove the §5 section and its row in the summary table so the description matches the diff.
  2. Fix the float-carrier comment this PR left stale. tests/model_ops/test_comm_fused_moe.py:528-529 still says "ids travel as their float carrier", but the new gather_rows (atom/distributed/comm_utils.py) sends int32 as int32. Change the comment to "ids travel as int32" and drop the .view(torch.int32), which no longer does anything (Fix-then-sweep).
  3. Wait for the heavy accuracy jobs at c5b86da to finish green. Pre Checkin, Ruff/Black, the non-GPU unit tests and the offline smoke test pass. The ATOM accuracy matrix (including DeepSeek-V4-Pro and V4-Pro DSpark) is still pending. If DeepSeek-V4-Pro DSpark fails with "VllmBackend can only be called once", that also fails on main and is not from this PR.

Both review rounds' asks are addressed (checked at head): per-rank and per-group ub{i}_ paged-prefill staging replaces the reindex, so upload_numpy, the host mirror and the length asserts are gone. DSpark now quantizes before the grouped gather, and the proposer goes through DSparkDraftModel.write_context_kv. gather_rows lives in atom/distributed/comm_utils.py, and pcp_allgather_rerange is _many([x])[0] with the MLA, V2 and sglang call sites grouped. Compressor.project and forward are split. The comm-phase primitives are private behind tbo_on_comm, with MoRI as the documented exception. The DP and TP all-reduce paths are swept. ROCm/aiter#7149 is merged (11-02). No aiter probing, no new shuffle interfaces. The branch merges cleanly into main.


Re-reviewed at c5b86da. Already fixed: upload_numpy/host-mirror reindex, contiguous TBO-group reindex, DSpark W-fold quantization, proposer reaching into layers, gather_rows home, two all-gather paths, Compressor two modes, public mark/yield primitives, source-scan enforcement, 6-rank fallback (aiter#7149), side-stream record_stream, DP/TP tbo_on_comm sweep.

yitingw1 and others added 2 commits November 13, 2026 02:10
Combine main's `slot_mapping` and this branch's `pcp_num_tokens` on
`write_context_kv` into one signature; the PCP stripe path also takes
main's trim to `positions`' length and passes `slot_mapping` through.
KimiK3DSparkGQA's new override now forwards `pcp_num_tokens`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`gather_rows` passes int32 through unchanged since it moved to
`comm_utils`; the float-carrier comment and its `.view` were stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitingw1

Copy link
Copy Markdown
Contributor Author

@valarLip Thanks for the re-review.

  1. Dropped §5 and its summary-table row from the description; compute_hash already keys on kv_cache_dtype via fix(config): key the compile cache on max_num_batched_tokens and kv_cache_dtype #3072.
  2. Fixed in 32bf41e: the comment now says ids travel as int32, and the no-op .view(torch.int32) is gone.
  3. The head moved since your review: 4149e4e merges main to clear a conflict with feat(kimi-k3): DFlash GQA speculative draft model #2581, where write_context_kv gained slot_mapping alongside this PR's pcp_num_tokens. Both are now taken, and KimiK3DSparkGQA's new override forwards pcp_num_tokens. The accuracy matrix is re-running on the new head; I'll follow up once it's green.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (1)

  1. Drop the unreachable PCP branch from DeepseekV41DSpark.write_context_kv. V4.1 refuses context parallel at startup (atom/models/deepseek_v41/config.py:363-366), and the proposer passes pcp_num_tokens only after a PCP split (dspark_proposer.py:736-746, pcp_aux_rows), so the pcp_num_tokens parameter and branch at atom/models/deepseek_v41/dspark.py:296-306 never run. They are only there to satisfy test_v41_dspark_write_context_kv_takes_the_split_stripe (tests/test_pcp_ops.py:736) and the AST scan in test_every_dspark_draft_override_takes_pcp_num_tokens (:749). Revert V4.1 to main's signature, delete the V4.1 test, and drop DeepseekV41DSpark from the scan's expected set (:805), or scope the scan to drafts that can run under PCP. Small cleanup while you're there: the Test Plan still lists tests/test_compile_hash_spec.py, which this PR no longer touches since §5 was removed.

All three asks from 11-12 are fixed at head. §5 and its table row are gone from the description. tests/model_ops/test_comm_fused_moe.py:528 now says ids travel as int32, and the .view is gone (32bf41e). The full ATOM accuracy matrix is green at 32bf41e, including DeepSeek-V4-Pro, V4-Pro DSpark and V4.1-Flash DSpark. I re-reviewed the whole diff at head, not only the delta, and found no correctness issues:

  • Per-rank and ub{i}_ paged-prefill staging matches the whole-batch build, checked by CPU probes at ws 2/3/4/6/8, including ranks that own no tokens.
  • Dropping csa_indices.fill_(-1) is safe because the buffer is allocated per forward and every cell is written.
  • tbo_on_comm holds every returned tensor, nested ones included.
  • The write_context_kv(aux, positions, slot_mapping, pcp_num_tokens) merge resolution lines up across the base class, V4, V4.1 and Kimi-K3.
  • extra_write=0 on PCP+TBO groups is equivalent to before, because PCP+TBO refuses spec decode.
    The hard rules also hold: no aiter probing and no new shuffle interfaces. The branch merges cleanly into main.

Re-reviewed at 32bf41e. Already fixed: description §5, stale float-carrier comment and no-op view, accuracy matrix green; plus every item from the 10-29 and 11-02 rounds (re-checked at head).

V4.1 refuses context parallel at startup, so the proposer never passes it
`pcp_num_tokens`. Restore main's signature, drop its split test, and keep
it out of the override scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitingw1

Copy link
Copy Markdown
Contributor Author

@valarLip Thanks for the full re-review.

  1. Fixed in f7d83ab: DeepseekV41DSpark.write_context_kv is back to main's signature with the PCP branch removed, so deepseek_v41/dspark.py no longer appears in this PR's diff. The V4.1 split test is deleted, and the override scan now skips V4.1, with the reason (it refuses context parallel in validate_runtime_config) in its docstring.
  2. Dropped tests/test_compile_hash_spec.py from the Test Plan.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM

The one ask from 11-13 is fixed at f7d83ab. DeepseekV41DSpark.write_context_kv is back to main's (aux_concat, positions, slot_mapping=None), so atom/models/deepseek_v41/dspark.py no longer appears in the diff. The V4.1 split test is gone. The override scan (tests/test_pcp_ops.py:736) skips V4.1 and cites the refusal that makes this safe, validate_runtime_config (atom/models/deepseek_v41/config.py:338,363-366). The Test Plan no longer lists tests/test_compile_hash_spec.py.

I re-reviewed the whole diff at head, not only the delta, and found no correctness issues. The proposer passes **{"pcp_num_tokens": n} only after a PCP split (dspark_proposer.py:736-771). Every draft that can run under PCP takes that argument: the base class, KimiK3DSparkGQA's forwarding override, and V4 / Kimi-K3 / MiniMax-M3 through the base class. Per-rank and ub{i}_ staging, gather_rows' uint8 path, and the tbo_on_comm holds were all re-checked. With aiter absent, the 8 affected CPU test files give 130 passed and 69 skipped. There is no aiter probing and no new shuffle interface. The branch merges cleanly into main.

CI at f7d83ab is green, including V4-Pro, V4-Pro DSpark, V4.1-Flash DSpark and Kimi-K3 DSpark (plus PP2). The one exception is Kimi-K3 (non-DSpark), which is still queued for a runner. It passed at 32bf41e, and the only commit since then touches V4.1 DSpark and a test.


Re-reviewed at f7d83ab. Already fixed: unreachable V4.1 PCP branch and its test, Test Plan entry for test_compile_hash_spec.py; plus every item from the 10-29, 11-02, 11-12 rounds (re-checked at head).

@valarLip valarLip added the LGTM waiting pass for CI label Nov 13, 2026

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Approved at f7d83ab.

@valarLip
valarLip merged commit 0dd1f88 into main Nov 14, 2026
100 of 103 checks passed
@valarLip
valarLip deleted the yitingw1/pcp_opt branch November 14, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:atom LGTM waiting pass for CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants