Repository navigation
[PCP] Fewer V4 prefill gathers and fix PCP+TBO accuracy - #2789
Conversation
Each V4 attention layer under PCP prefill issued 3-5 separate all-gathers (main compressor input, indexer compressor input, extend K -- two under fp8 KV -- and positions), each through torch.distributed. Now it issues two: - `pcp_allgather_rerange_many` gathers several tensors in one grouped `all_gatherv` and restores token order per tensor. One-byte dtypes travel as uint8, since past aiter's custom all-gather limit the call falls back to PyNCCL, which has no fp8 dtype. - The extend K is gathered together with the positions its SWA write addresses the ring by. Positions are gathered rather than read off the forward context: an MTP draft pass runs at the target's positions + 1. - `_launch_compressors` projects both compressors' inputs and gathers them in one collective; `Compressor.forward(combined=)` takes the result. V4-Pro (30 CSA + 31 HCA layers): 397 -> 245 collectives per prefill step. `pcp_reindex_ragged` sized its output with a blocking `.item()`, four times per prefill step (sixteen under PCP+TBO). The builder already computes the four prefill indptrs in numpy; it now keeps them on the metadata (`paged_prefill_indptr_cpu`) and each rank's local indptr is computed on the host (`pcp_local_indptr`), in both the full-batch and the per-request-group paths. `pcp_round_robin_split` returns unit strides: `.contiguous()` leaves a one-row shard at row stride pcp_size, which aiter's cached-positions RoPE rejects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After a PCP prefill the aux hooks hold only this rank's round-robin rows. `compute_draft_kv` all-gathered the three aux hidden states (~42 KiB a row) and every rank then projected all rows into window KV, the same work on every rank. The projection (main_proj + norm, then per stage wqkv_a + norm + RoPE + fake-quant) works row by row, so project only the local rows at their own positions, gather the projected KV rows (1 KiB a row a stage, one grouped collective for all stages), and write them as before. `DSparkLayer.write_context_kv` is split into `context_kv_rows` and `write_context_kv_rows`. A draft opts in with `context_kv_by_rows`; only V4 DSpark does, and any other draft keeps gathering the aux states. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…unts aiter's custom all-reduce, which `all_gatherv` tries first, runs at 6 ranks, but its all-gatherv kernel exists for 2, 4 and 8 only and throws otherwise. The attention-side gathers went through torch.distributed before the grouped gather, so PCP 6 worked; take one tensor at a time there again. Also assert that the grouped inputs share a row count (a mismatch threw on the custom path but passed on PyNCCL), and fix two review nits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The DSpark drafter branches on the KV cache dtype while tracing (`dspark_fp8_planned`), but `Config.compute_hash` left it out -- the known limitation #2566 deferred. A DSV4-Pro DSpark run with `--kv-cache-dtype bf16` after an fp8 one in the same container hashed to the same directory, loaded the fp8 drafter graph and died in warmup on every rank (`IndexError: list index out of range` on the split graph's outputs). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…eads them Under TBO a PCP gather runs on the comm stream, so its output is allocated there, while the compute stream reads it. When the forward frees it, the caching allocator hands the memory straight back to the comm stream, and the comm stream waits only on its own ubatch's compute: the other ubatch's next collective can overwrite it before this ubatch's kernels have read it. On main this corrupts the gathered extend K. DSV4-Pro, pcp8 + `--enable-tbo`, gsm8k (200 q): main 0.69 / 0.675 on two nodes. With only the extend-K gathers on the main code path: 0.71 (K alone: 0.70, positions alone: 0.975); the same plus `record_stream` on the gathered tensors: 0.965. The grouped gathers of this branch have the same shape, and score 0.96 only by timing. `tbo_mark_used_on_compute` records the compute stream on such tensors after `tbo_switch_to_compute_sync`, for the attention-side gathers (extend K and positions, the compressor inputs) and FusedMoE's PCP routed-rows gather and reduce-scatter. The DP and mori TBO paths have the same shape and are left alone here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The previous commit marked the comm-stream outputs by hand at four call sites, and nothing would notice one going missing: this branch scores 0.96 under TBO with or without the marks. `tbo_on_comm(fn, *args)` makes the comm phase one call -- yield, `fn` on the comm stream, switch back, hold every tensor it returns -- so a phase cannot skip the hold, and a test fails if V4 attention or `FusedMoE._run_experts` writes a phase out by hand. `tbo_mark_used_on_compute` now takes the compute stream from the TBO context instead of the current stream, so it no longer depends on being called after the switch back. `pcp_aux_rows` documents that it describes the base layer-hook capture; EAGLE3 strips aux off the target's output instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valarLip
left a comment
There was a problem hiding this comment.
Thanks — I reviewed HEAD 9d1db12b2. The core changes check out: the grouped gather, host-side indptr, record_stream in tbo_on_comm, and per-row DSpark KV projection. I also confirmed aiter's custom_all_gatherv returns a freshly allocated output (no aliasing with the registered buffer), the GPU/CPU kv_indptr_* copies are written in one place (_build_paged_prefill_meta) and stay consistent, and dummy runs don't enter the new DSpark branch.
Six inline comments. The two most worth acting on:
- Under PCP + TBO, DSpark's context KV is reconstructed as one global stripe while the buffer is per-ubatch and padded per group, so draft KV lands on wrong rows (
dspark_proposer.py:645). - The comm-stream lifetime hazard this PR fixes via
tbo_on_commremains on the DP / TP-allreduce / MORI paths that still hand-write yield → collective → switch (ubatching.py:615).
… gathers Review follow-ups on #2789: - tbo_all_reduce and FusedMoE's DP gather / reduce-scatter run through tbo_on_comm, like the PCP comm phases, so their comm-stream outputs are held for the compute stream; the hold walks nested lists and tuples. Mori is unchanged: it returns views of its own per-ubatch buffers. - Under a graph the V4 compressors read their PCP-gathered inputs on side streams; hold them for those streams. - gather_rows moves to pcp_utils and carries one-byte dtypes as uint8 and the rest as is (aiter's all_gatherv copies bytes; only its PyNCCL fallback lacks fp8 / e8m0). MoE's DP, PCP and comm-fused gathers and the PCP grouped gather share it. - The allocator-reuse positive control skips under expandable_segments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tbo_all_reduce imports it ahead of the TBO branch, and the non-GPU CI's aiter has no aiter.dist.communication_op. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ROCm/aiter#7149 makes custom_all_gatherv decline group sizes it has no kernel for, so all_gatherv falls back to NCCL there. The grouped gather now takes the one-collective path at every PCP size, 6 included. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@valar-cop review this pr |
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed
any perf number for tp>1 / pp+pcp? and a gpu run of pcp+tbo and pcp+dspark?
ci green.
_apply_pcp_reindex now derives each rank's indptr from the host copy the builder publishes (paged_prefill_indptr_cpu); the test builds its metadata by hand, so it sets that copy beside kv_indptr_extend. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Re the review: perf was measured at |
The file failed 20 of 22 cases on main with RUN_H2D_GPU_TESTS=1, all from fixtures that fell behind the code: - set_pcp patched pcp_round_robin_query_indices (removed in #2688); the builder reads get_pcp_rank. - make_runner lacked forward_vars["batch_info"], config.max_num_seqs and the builder's csa_layers, max_model_len_idx, _local_heads and index_logits_workspace. - The padded decode case modelled the bucket width through UBatchWrapper._decode_ub_running_bs; since #2692 the split comes from _prepare_ubatch_decode's split_bs, so the test passes the bucket. - prepare_prefill was called with a total_tokens_num the fake batch lacks. 22/22 pass on this branch and on its merge base. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed, the pcp 6 fallback removal is risky..
any perf number at tp>1? and the graph-pool footprint from record_stream is still not measured..
ci green.
Both PCP reindex paths (rank and PCP+TBO group) compute local offsets from the host copy and reindex the GPU copy; a length mismatch would silently reindex against a different query set. Clarify that after either reindex both copies describe the same rank- or group-local queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. Pushed fd912b7 (host/device indptr length asserts + docstring); replies inline. On the summary points:
|
valarLip
left a comment
There was a problem hiding this comment.
Re-read at c889ff9, by reading the code and running the 7 affected CPU test files (163 pass). Thanks for the rounds since 10-29. The tbo_on_comm sweep, the unified gather_rows, the side-stream record_stream, and the graph-pool measurement all read right. I'm not repeating what's been answered: the PCP+DSpark+TBO startup refusal, -pcp 6 on aiter#7149, and the indptr length asserts.
Performance (the three I'd most like addressed)
This PR's point is to stop every rank redoing the whole prefill. Three places still pay per step for work that's already known on the host, or that each rank repeats:
pcp_reindex_raggeduploads withupload_numpyevery forward, per field (pcp_utils.py:322).upload_numpy's own docstring says "A buffer uploaded every forward should be aCpuGpuBuffer". That's 4 pageable H2Ds per prefill step, and 12+ under PCP+TBO, plus the arange / searchsorted / gather kernels behind each.- The PCP+TBO group path reindexes a contiguous range with the general machinery (
deepseek_v4_attn.py:3333). A group owns[gts, gte). For a contiguous range, the indices are a viewindices[cpu[gts]:cpu[gte]]and the indptr iscpu[gts:gte+1] - cpu[gts]: no kernel, no upload. Today each group pays 3–4 kernels and an upload per field. - DSpark gathers bf16 KV rows and then every rank quantizes all
nrows (dspark_proposer.py:658). The quantization is per row, so quantizing the local rows first and gathering the 2buff result cuts the wire (by my estimate about 640 B per row instead of 1 KiB) and removes the W-fold repeated quantization.
Items 1 and 2 share a root fix, which is the first inline comment on deepseek_v4_attn.py:406: build the rank- or group-local indptr in the builder's numpy pass and upload it once.
Structure
The rest is inline: the proposer reaching into the draft model, gather_rows' home, the two all-gather paths, Compressor's two modes, and the TBO comm-phase API.
|
ci triage for f7d83ab:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inish split Port the per-rank paged-prefill build onto main's stage_prefill / finish_prefill structure (#3136): - Under PCP, stage_prefill stages this rank's own rows (pcp_split) and finish_prefill reindexes them; pad query segments get length 0 and the per-step indptr/indices fill and reindex helpers are gone. - Under PCP+TBO each group stages and finishes its own ub{i}_ inputs (_stage_pcp_group / _finish_pcp_group) with group-rebased cu_seqlens_q, block_tables and replay ranges. - _pcp_split_rows takes scheduled and running token counts separately. - Move shared comm helpers into atom/distributed/comm_utils.py. - Tests: rank segments with running > scheduled, PP relay _aux_blocks in the DSpark drafter stubs. gsm8k 5-shot 200q, V4-Pro -tp 1 -pcp 8: 0.965, +TBO prefill 0.970, DSpark k=5 0.975 (acceptance 63.76%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve dspark_proposer.compute_draft_kv against #3091's served_by_round early return, and fix three tests this branch broke: - Drafter._aux_blocks becomes the module-level aux_blocks(drafter), so SimpleNamespace stubs with only _aux_buffers/_relayed_aux keep working (test_pp_aux_relay). - compute_draft_kv keys the non-PCP path on aux_for() returning None, as main does; only the PCP split path reads _captures_aux (test_deepseek_v41_mono_hi_served_step). - test_sp_moe_dispatch patches tbo_on_comm instead of the removed moe.tbo_active. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_aux_rows Non-PCP steps no longer query the PCP group, which is a MagicMock under the aiter stub and broke test_deepseek_v41_mono_hi_served_step in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itly The 1110 port defaulted cu_q_per_seq_gpu and block_tables_gpu to the whole batch's, so a PCP+TBO group caller that forgot them would build indices from the wrong rows without an error. Make both keyword-only and required; the whole-batch caller passes its own rows. Also note in pcp_aux_rows that target_pcp_split implies PCP is enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@valarLip Thanks for the second pass. All ten inline comments are addressed in c9b9de7 and the follow-ups, ported onto current main in 1e582ae, with CI fixes in aee343c and 5e4e8bb, and c5b86da making the group-level |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Drop §5 "Compile cache key" from the PR description.
kv_cache_dtypeis already inConfig.compute_hashon main via #3072 (atom/config.py:3338), and after aee343c this PR no longer touchesatom/config.py. Remove the §5 section and its row in the summary table so the description matches the diff. - Fix the float-carrier comment this PR left stale.
tests/model_ops/test_comm_fused_moe.py:528-529still says "ids travel as their float carrier", but the newgather_rows(atom/distributed/comm_utils.py) sends int32 as int32. Change the comment to "ids travel as int32" and drop the.view(torch.int32), which no longer does anything (Fix-then-sweep). - Wait for the heavy accuracy jobs at c5b86da to finish green. Pre Checkin, Ruff/Black, the non-GPU unit tests and the offline smoke test pass. The ATOM accuracy matrix (including DeepSeek-V4-Pro and V4-Pro DSpark) is still pending. If DeepSeek-V4-Pro DSpark fails with "VllmBackend can only be called once", that also fails on main and is not from this PR.
Both review rounds' asks are addressed (checked at head): per-rank and per-group ub{i}_ paged-prefill staging replaces the reindex, so upload_numpy, the host mirror and the length asserts are gone. DSpark now quantizes before the grouped gather, and the proposer goes through DSparkDraftModel.write_context_kv. gather_rows lives in atom/distributed/comm_utils.py, and pcp_allgather_rerange is _many([x])[0] with the MLA, V2 and sglang call sites grouped. Compressor.project and forward are split. The comm-phase primitives are private behind tbo_on_comm, with MoRI as the documented exception. The DP and TP all-reduce paths are swept. ROCm/aiter#7149 is merged (11-02). No aiter probing, no new shuffle interfaces. The branch merges cleanly into main.
Re-reviewed at c5b86da. Already fixed: upload_numpy/host-mirror reindex, contiguous TBO-group reindex, DSpark W-fold quantization, proposer reaching into layers, gather_rows home, two all-gather paths, Compressor two modes, public mark/yield primitives, source-scan enforcement, 6-rank fallback (aiter#7149), side-stream record_stream, DP/TP tbo_on_comm sweep.
Combine main's `slot_mapping` and this branch's `pcp_num_tokens` on `write_context_kv` into one signature; the PCP stripe path also takes main's trim to `positions`' length and passes `slot_mapping` through. KimiK3DSparkGQA's new override now forwards `pcp_num_tokens`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`gather_rows` passes int32 through unchanged since it moved to `comm_utils`; the float-carrier comment and its `.view` were stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@valarLip Thanks for the re-review.
|
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (1)
- Drop the unreachable PCP branch from
DeepseekV41DSpark.write_context_kv. V4.1 refuses context parallel at startup (atom/models/deepseek_v41/config.py:363-366), and the proposer passespcp_num_tokensonly after a PCP split (dspark_proposer.py:736-746,pcp_aux_rows), so thepcp_num_tokensparameter and branch atatom/models/deepseek_v41/dspark.py:296-306never run. They are only there to satisfytest_v41_dspark_write_context_kv_takes_the_split_stripe(tests/test_pcp_ops.py:736) and the AST scan intest_every_dspark_draft_override_takes_pcp_num_tokens(:749). Revert V4.1 to main's signature, delete the V4.1 test, and dropDeepseekV41DSparkfrom the scan's expected set (:805), or scope the scan to drafts that can run under PCP. Small cleanup while you're there: the Test Plan still liststests/test_compile_hash_spec.py, which this PR no longer touches since §5 was removed.
All three asks from 11-12 are fixed at head. §5 and its table row are gone from the description. tests/model_ops/test_comm_fused_moe.py:528 now says ids travel as int32, and the .view is gone (32bf41e). The full ATOM accuracy matrix is green at 32bf41e, including DeepSeek-V4-Pro, V4-Pro DSpark and V4.1-Flash DSpark. I re-reviewed the whole diff at head, not only the delta, and found no correctness issues:
- Per-rank and
ub{i}_paged-prefill staging matches the whole-batch build, checked by CPU probes at ws 2/3/4/6/8, including ranks that own no tokens. - Dropping
csa_indices.fill_(-1)is safe because the buffer is allocated per forward and every cell is written. tbo_on_commholds every returned tensor, nested ones included.- The
write_context_kv(aux, positions, slot_mapping, pcp_num_tokens)merge resolution lines up across the base class, V4, V4.1 and Kimi-K3. extra_write=0on PCP+TBO groups is equivalent to before, because PCP+TBO refuses spec decode.
The hard rules also hold: no aiter probing and no new shuffle interfaces. The branch merges cleanly into main.
Re-reviewed at 32bf41e. Already fixed: description §5, stale float-carrier comment and no-op view, accuracy matrix green; plus every item from the 10-29 and 11-02 rounds (re-checked at head).
V4.1 refuses context parallel at startup, so the proposer never passes it `pcp_num_tokens`. Restore main's signature, drop its split test, and keep it out of the override scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@valarLip Thanks for the full re-review.
|
valarLip
left a comment
There was a problem hiding this comment.
✅ LGTM
The one ask from 11-13 is fixed at f7d83ab. DeepseekV41DSpark.write_context_kv is back to main's (aux_concat, positions, slot_mapping=None), so atom/models/deepseek_v41/dspark.py no longer appears in the diff. The V4.1 split test is gone. The override scan (tests/test_pcp_ops.py:736) skips V4.1 and cites the refusal that makes this safe, validate_runtime_config (atom/models/deepseek_v41/config.py:338,363-366). The Test Plan no longer lists tests/test_compile_hash_spec.py.
I re-reviewed the whole diff at head, not only the delta, and found no correctness issues. The proposer passes **{"pcp_num_tokens": n} only after a PCP split (dspark_proposer.py:736-771). Every draft that can run under PCP takes that argument: the base class, KimiK3DSparkGQA's forwarding override, and V4 / Kimi-K3 / MiniMax-M3 through the base class. Per-rank and ub{i}_ staging, gather_rows' uint8 path, and the tbo_on_comm holds were all re-checked. With aiter absent, the 8 affected CPU test files give 130 passed and 69 skipped. There is no aiter probing and no new shuffle interface. The branch merges cleanly into main.
CI at f7d83ab is green, including V4-Pro, V4-Pro DSpark, V4.1-Flash DSpark and Kimi-K3 DSpark (plus PP2). The one exception is Kimi-K3 (non-DSpark), which is still queued for a runner. It passed at 32bf41e, and the only commit since then touches V4.1 DSpark and a test.
Re-reviewed at f7d83ab. Already fixed: unreachable V4.1 PCP branch and its test, Test Plan entry for test_compile_hash_spec.py; plus every item from the 10-29, 11-02, 11-12 rounds (re-checked at head).
Requires ROCm/aiter#7149 (6-rank groups).
Size. This PR is 29 files / ~1.9K lines, above the usual split threshold. Most of the growth after the first version answers two review rounds: the paged-prefill metadata rework, grouped gathers, DSpark local quantization and the private TBO mark primitive touch the same builder and gather paths, so splitting by comment would leave intermediate states that fail the PCP+TBO accuracy check (gsm8k 0.68 on
main). Tests account for about half of the diff.Motivation
At small prefill batches PCP on DeepSeek-V4-Pro is host-bound: a step at
-tp 1 -pcp 8launches ~3,200 GPU ops, and the CPU, not the GPU, sets its length (~190 ms of host time per step, with the GPU idle for up to 126 ms of it). Every attention layer issued 3–5 separatetorch.distributedall-gathers, and the PCP reindex blocked on a.item()per ragged field. DSpark gathered three full aux hidden states per token and then projected every row on every rank. Validating this exposed a PCP+TBO accuracy bug onmain.-pcp 8, concurrency 1.item(); TTFT −10 % at ISL 16KTechnical Details
1. Grouped gathers
pcp_allgather_rerange_manygathers several tensors in oneall_gatherv(aiter's custom all-gather: one launch, one barrier; PyNCCL group past its size limit) and restores token order per tensor. Each layer now issues one gather for the two compressor inputs (Compressor.projectin_launch_compressors, gathered, thenCompressor.forward(projected)) and one for the extend K with its positions:maink_packed,k_rope, positionsk_packed,k_rope, positionsMoE's gather / reduce-scatter (2 per layer) are unchanged. Positions are gathered rather than read off the forward context because an MTP draft pass runs at the target's positions + 1.
gather_rows(now inatom/distributed/comm_utils.py, shared with MoE's DP / PCP / comm-fused gathers) carries one-byte dtypes as uint8 and everything else as is: aiter's groupedall_gathervcopies bytes, and only its PyNCCL fallback lacks fp8 / e8m0 (MoE used to dress int32 ids as fp32, a constraint of the per-tensor custom all-gather it no longer calls). At group sizes aiter's custom all-gatherv has no kernel for (6), this PR requires ROCm/aiter#7149, which makes aiter decline them and fall back to NCCL.2. Per-rank paged-prefill metadata
mainbuilt whole-batch paged-prefill metadata and reindexed it per rank (pcp_reindex_ragged, sized by an.item()per ragged field). Each rank now stages its own indptrs from the builder's numpy arrays through the CpuGpuBuffer publication group and launches the index kernel only over the tokens it owns; the host mirror, pad / reindex kernels,upload_numpyand thecsa_indices.fill_(-1)are gone. Under PCP+TBO each request group stages its ownub{i}_rows the same way. V4-Pro-tp 1 -pcp 8, ISL 16K, mbt 4096: TTFT −9.8 % (conc 1) / −11.1 % (conc 8); with--enable-tbo prefill−11.0 % / −8.5 % (includes the grouped gathers of §1).3. DSpark context KV
The projection from aux hidden states to window KV (
main_proj+ norm, then per stagewqkv_a+ norm + RoPE + fake-quant) is row by row, so each rank now projects only its own rows at their own positions, gathers the projected KV, and writes as before.DSparkLayer.write_context_kvis split intocontext_kv_rows+write_context_kv_rows. The V4 DSpark draft overrideswrite_split_context_kv: it projects and quantizes its own rows into the fp8 2buff layout and sends every layer's rows in one grouped gather; other flavors keep the basewrite_split_context_kv, which gathers the aux states.pcp_round_robin_splitnow returns unit strides:.contiguous()left a one-row shard (a 2-token chunk at pcp 8) at row stride 8, which aiter's cached-positions RoPE asserts on.4. PCP+TBO: comm-stream outputs read on the compute stream
Under TBO a PCP gather runs on the comm stream, so its output is allocated there, while the compute stream reads it. Once the forward frees it, the caching allocator hands that memory straight back to the comm stream, and the comm stream waits only on its own ubatch's compute (
ubatching.py, one shared comm and compute stream): the other ubatch's next gather of the same layer can overwrite the extend K before this ubatch's attention has read it.-pcp 8 --enable-tbo, gsm8k 200 qmain(two nodes)main's path / K only / positions onlymain's path +record_streamtbo_on_comm(fn, *args)now runs every TBO comm phase whose output the caching allocator owns: yield,fnon the comm stream, switch back, andrecord_streamthe TBO context's compute stream on every tensorfnreturns (nested lists / tuples included). That is the extend K and positions, the compressor inputs, FusedMoE's PCP and DP routed-rows gathers and reduce-scatters, and pure TP's overlappedtbo_all_reduce(-tp 8 --enable-tbo prefill: gsm8k 0.96). Mori keeps its hand-written phases: dispatch / combine return views of each ubatch's own mori buffers, which the allocator never hands out. The grouped gathers passedmain's accuracy only by timing (0.96 without the hold), so gsm8k cannot pin the call sites; a test fails instead if any of these writes a comm phase out by hand. Under a graph the V4 compressors read their PCP-gathered inputs on side streams, so those are held for the side streams the same way.Test Plan
Test Result
Unit tests: 700 passed, 12 skipped; ruff and black clean. Each new test fails with its change reverted (22 mutations: 17 on the trees before this review round, 5 on the final one). aiter's
test_custom_allgatherv.py -t 6passes with its fix and fails with the world-size gates forced open;-t 8unchanged.-tp 1 -pcp 8unless notedmain0.69 / 0.675)-tp 4 -pcp 2, 20K-token prompts (PyNCCL fallback for fp8) + 200 q, final commit-tp 1 -pcp 6with ROCm/aiter#7149 (no ATOM 6-rank special case), 100 q-tp 8 --enable-tbo prefill, mbt 16384, TBO from 1024 tokens (64 of 80 prefill steps in a profiled probe ran TBO), 200 qmainKnown Limitations
-pcpwith speculative decoding and--enable-tbostays refused at startup ([PCP] Fix sampled-id sync, DSpark aux gather and FP4 indexer padding #2125): PCP+TBO stripes each request group on its own, while DSpark's aux gather and context-KV projection, like the draft prefill split, read the batch as one stripe. Lifting that needs all three to follow the groups.--enable-tbo allthe TP all-reduce and DP collectives nowrecord_streamwhile decode graphs capture. Measured at-tp 8 --enable-tbo all(DSV4-Pro, FULL graphs, 8 of 10 buckets recorded split): 1952 holds per rank during capture, 135 MiB held in total, and the graph pool grows by 0.00 GB with or without the hold; KV blocks unchanged.-tp 1 -pcp 8(concurrency 1 and 8, with and without--enable-tbo prefill), where every attention gather takes the custom all-gather;-tp 4 -pcp 2was checked for correctness only.DeepseekV2ForCausalLMhas already gathered); pre-existing, for a separate fix.tests/test_fused_compress_ragged.pyfails 4 cases on this image onmaintoo; the PCP unit tests import aiter, so a CI runner without it skips them.Submission Checklist
🤖 Generated with Claude Code