[wanzhenchn]

Motivation

Track Kimi-K3 AgentX 1M P/D performance in CI with one cell per InferenceX concurrency point (C1–C88), each on the prefill topology that measured best at that point, so regressions in the Kimi-K3 PP8 P/D stack (#2709, #3068, #3069, #3070) and in #2786 show up. The decode side follows InferenceX's AgentX recipe (kimik3-fp4-mi355x-atom-agentic-mtp, SemiAnalysisAI/InferenceX#4085 and #4307). They replace all of Kimi-K3's older nightly cells.

Changes

• Nine cells in .github/benchmark/models_atomesh.yaml, plus the case_names help in .github/workflows/atomesh-benchmark.yaml.
• All 20 older nightly cells retired. Kimi-K3's nightly is now just the nine cells.
  • 13 sat at the same concurrencies: TP8 DSpark7 C1/2/4, DSpark3 C16, no-draft C32–C88, and the TP8×DCP8-prefill DSpark4 C48–C88.
  • The other seven ran a TP8×DCP8 prefill: DSpark3 C8, no-draft C64, 1P2D C48, AGG C72 and 2P1D C72/128/256.
  • The old C4's GSM8K eval moves to the new C4. C1 runs GSM8K too, at concurrency 1, since no other cell runs mono. C88 does not: C72's GSM8K gates OPUS, which C88's decode runs as C72's does.
  • main's MoRIIO GSM8K cell ([ATOMesh][MoRIIO] Enable Kimi K3 and update the MoRI IO changes to support Kimi KDA state for KV. #2570) took its server args from the retired no-draft C32 cell's anchor. It now spells out the same three: gpu_memory_utilization: 0.90, max_num_seqs: 64 and max_num_batched_tokens: 8192.
  • recipes/mesh/Agentic-Kimi-K3.md now names the PP8 cells as its CI cases. They run its decode and state cache behind a PP8 prefill. Its decode exports AITER_MLA_USE_OPUS=1 at C72 and C88, as the -opus cells do. Its online-quant exclude list now keeps the DSpark draft in bf16 ("context_proj", "layers", the draft's unprefixed module names), as the cells and recipes/Kimi-K3.md do.
• The profile suite's torch-profile cell now profiles the nightly PP8 C72 instead of the retired TP8×DCP8 C72, so a trace shows the configuration the dashboard publishes. It differs from the nightly C72 only in its name, the profile window env, and running no GSM8K.
• OPUS and mono are visible on the dashboard, which labels a series by display_topology. C72, C88 and the profile cell end in -opus and their label in -OPUS. C1's label ends in -MONO, matching its name's -mono.
• Four cell keys, in pd_matrix.py, pd_submit.sh and pd_server_atom.sh:
  • online_quant_extra_exclude_layer, one pattern or a list, set in server.common_args or on a role. It is appended to the model's online_quant_config exclude list on that role's command line. The cells no longer copy the JSON, and no role override depends on argparse last-wins.
  • requires_atom_env lists the ATOM settings a cell is measured with. Before starting any server, the launcher loads the atom/utils/envs.py of the ATOM under test and fails the cell if one of them is undefined. The PP8 cells require ATOM_PD_MLA_STAGING_CHUNK_PAGES and ATOM_PD_MLA_STAGING_STREAM_PRIORITY, and C88 ATOM_PP_MAX_INFLIGHT, all from perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070. The C1–C16 cells require none.
  • expect_log is set per role, one line or a list. It names lines every server of that role must have logged while starting, and node 0 checks them once every server is healthy, before any request. The C1 decode expects two:
    • Kimi-K3 mono decode armed: its config admits mono. A decode whose config mono refuses serves the plain model.
    • Kimi-K3 mono decode on: a decode step reached mono and its runner was built. While starting, the only such step is the warmup forward before a FULL CUDA graph capture, so the line shows mono was built, not that a served request used it. Under --enforce-eager nothing is captured, the line first comes with a request, and the cell fails.
  • reject_log is set per role, one line or a list. No server of that role may log any of them. Node 0 checks before the first request and again after the last. A line's absence cannot be retried the way expect_log retries a line's presence, so the second check first waits out the same NFS window, 60 s by default. The C1 decode rejects Kimi-K3 mono decode off and Kimi-K3 mono decode declined the 8-row build. Mono logs these when it was armed but then cannot serve C1's steps: the first if its runner fails to build, the second if the width C1 serves, one request plus seven drafts, gets no build. Its capture also builds 16 rows, which C1 never serves, so that width's line is allowed, as is declined a step for a step mono does not serve at all, such as a prefill.
  • An aggregated cell has no prefill server, so a prefill expect_log or reject_log fails it instead of being skipped.
  • pd_submit.sh hands both keys to the launcher as JSON lists, and a key left unset or null checks nothing. Any other value there fails the cell, since a bare string would be checked char by char and an empty list or entry would check nothing. pd_submit.sh itself fails a cell that sets an empty list.
• The launcher exposes TP × PP GPUs to a role by default (pd_server_atom.sh). It exposed only TP, so the PP8 prefill had to list its eight GPUs in the cell. PP is parsed from the role's server args, as the packed layout already did. A cell that sets HIP_VISIBLE_DEVICES keeps its value.
• A PP8 profile run now ends on its own. wait_for_torch_traces (pd_server_atom.sh) counted only rank_*/ dirs, but a PP stage writes pp<k>_rank_<tp>/. Every PP profile run therefore waited out ATOM_PROFILER_TIMEOUT ("1/9 rank traces still missing after 1800s") and never finished. It now waits on the trace dirs stop_profile reported, plus their siblings that ModelRunner named for a rank: rank_<tp> or dp<d>_tp<tp>, prefixed pp<k>_ under PP. Any other dir is ignored. A rank is done once its dir holds a .gz whose name was not there when the window opened, and no raw .json. It goes by names because on the shared log mount, mtimes come from clocks that disagree by more than a window lasts: a run compared them with a marker and waited out the timeout on five traces already written (41266652593).
• This PR changes no ATOM code. It touches only .github/, tests/ and the recipe. The level-3 trace gzip that shortens a PP export is in perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 (item 7).

CONC	Prefill → decode	tok/s/GPU
1	TP8EP8 → TP8 DSpark7, mono	912
2	TP8EP8 → TP8 DSpark7	920
4	TP8EP8 → TP8 DSpark7	1,457
16	TP8EP8×DCP8 → TP8×DCP8 DSpark4	5,336
32	PP8 → TP8×DCP8 DSpark4	10,441
48	PP8 → TP8×DCP8 DSpark4	13,291
56	PP8 → TP8×DCP8 DSpark4	14,684
72	PP8 → TP8×DCP8 DSpark4, MLA on OPUS	17,118
88	PP8 → TP8×DCP8 DSpark4, MLA on OPUS	18,902

Why each setting

Every setting below was A/B'd on the same trace.

• Prefill topology by concurrency. TP8EP8 wins at low concurrency: 1,434 vs 1,406 at C4, 5,317 vs 5,204 at C16 (PP8 second). PP8 wins from C32: 10,409 vs 9,196 (+13%).
• C1 mono decode (needs perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786): 851 vs 761 tok/s/GPU (+11.8%) with the same prefill.
• ReplaySSM is off at C1–C4 and on from C16, as in the recipe. It only pays when there are drafts to roll back. The model-level note now says this too.
• On-demand state cache from C48 on the prefill (ATOM_STATE_CHECKPOINT_DEMAND, a 128 GiB state cap in a 224 GiB per-rank pool, as in InferenceX#4307): C72 +9.9%, C88 +14.1%. Only the prefill has a CPU tier, 128 GiB per rank, at C16 and C32.
• MLA on aiter's OPUS kernel at C72 and C88 (AITER_MLA_USE_OPUS=1 on the decode; this PR adds it to the recipe too).
  • Same image and nodes, OPUS off vs on: C72 16,425 vs 17,118 (+4.2%), ITL p50 21.96 vs 19.92 ms; C88 17,962 vs 18,902 (+5.2%), ITL p50 26.31 vs 23.54 ms.
  • aiter falls back to its asm kernel without a log line when OPUS does not apply, so -opus names the kernel requested. The A/B above is what shows it was taken.
  • OPUS's error against fp32 is larger than the asm kernel's. GSM8K exact_match,flexible-extract, the score the 0.94 gate reads, at C72 over three OPUS runs was 0.9553, 0.9409 and 0.9416 (this PR). The same cell without OPUS, on the same image, gave 0.9522, 0.9530, 0.9409 and 0.9416. A single run's stderr is about ±0.0062, so these runs neither show a drop nor rule out one of about 0.7 points.
  • The gate stays at 0.94, the model's threshold, though C72's runs sit close to it: the mean of all seven is 0.9465, and the lowest is 0.0009 above the gate.
  • Lower concurrencies are unmeasured.
• C88 prefill.
  • Two offload save/load threads: +1.7%, P90 TTFT −13%.
  • ATOM_PP_MAX_INFLIGHT=16: ~+15% vs 8.
• PP8 prefills: 128-page MLA staging slots, on streams at priority −1. perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070 makes the slots opt-in, so these cells now set ATOM_PD_MLA_STAGING_CHUNK_PAGES: "128", the old default that the runs here used. Priority −1: TTFT P90 13.26 s vs 15.13 s at priority 0.
• Online quant exclusions.
  • Every cell excludes 14 layers: the model's seven, and the DSpark drafter's six linears and context_proj, which stay bf16 like its checkpoint, which has no FP8 QAT.
  • The C1 mono decode excludes five more, as in InferenceX#4307: the KDA in_proj, f_b_proj, o_proj, and the shared experts' gate_up_proj and down_proj. Mono does not start with these layers in FP8.
  • The PP8 prefill also keeps the KDA f_b_proj in bf16. At TP1 that GEMM is unsharded (N 12288, K 128); aiter has no tuned FP8 config for it, and its untuned FP8 GEMM needs K above 192. In FP8 the prefill crashed on its first full chunk (40679836806).
• Every cell.
  • No fresh KV poll, base64 prompt ids or early T0 any more. perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 dropped ATOM_KV_POLL_FRESH_WAIT_MS because on C88 it left TTFT P50 (4.62 → 4.60 s) and throughput (18,052 → 18,073 tok/s/GPU) flat, and dropped ATOM_PD_PROMPT_IDS_B64 and ATOM_PD_EARLY_T0 until the unified entrypoint.
  • The runs under Validation still had all three on, the fresh poll at 5 ms. On C88 base64 ids and early T0 moved throughput by −0.7% (18,073 → 17,952 tok/s/GPU), inside the ±1–2% between node pairs, and did not improve end-to-end latency.
• ATOM_MONO_ENABLE=0 on the PP8 cells, as on the recipe's non-mono points.

Dependencies and merge order

This PR is the top of the Kimi-K3 PP8 P/D stack. It targets #3070, so the diff shown is its own:

1. feat(kimi-k3): PP8 P/D prefill 1/4 — PAGE-backed state checkpoints under PP #2709: PAGE-backed KDA state checkpoints under PP. Merged.
2. feat(kimi-k3): PP8 P/D prefill 2/4 — LMCache state tier under PP #3068: Kimi-K3 LMCache state tier under PP.
3. feat(kimi-k3): PP8 P/D prefill 3/4 — KV and KDA state from a PP prefill to a DCP × TP decode #3069: KV and KDA state from a PP prefill to a DCP × TP decode.
4. perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070: MLA staging for a DCP consumer, and more PP batches in flight.
5. This PR.

The PP8 cells need all four, and the C1 mono cell needs #2786, which targets main on its own. Merge this PR last. On an ATOM without #3070's settings, the PP8 cells fail at launch on requires_atom_env instead of publishing numbers without them. If this PR lands before #2786, the C1 cell fails on expect_log, since its decode does not arm mono. It also fails on expect_log if its decode arms mono but never turns it on, and on reject_log if mono then turns itself off.

Validation

The table in Changes is from a sweep of the pre-split branch on rocm/atom-dev:nightly_202610241500; its C72 and C88 rows are the OPUS A/B runs on that image (40546704877, 40617525682).

All nine cells and the profile cell ran on #2709 + #2786 + this PR (zwan/k3-agentic-verify) in rocm/atom-dev:nightly_202610301458, the image main's nightly resolves to. The verify branch carried #2709 at bd26837 and #2786 at a5f0fc4, and adds only GSM8K on C88. With this PR, C2, C4, C56 and the profile cell ran at 08a84e6, C1, C16, C32, C48 and C72 at 048c838, and C88 at 2840ea1. Between those this PR changed only the profile wait, C1's last reject_log check, how the launcher finds envs.py, and pd_submit.sh's check for empty lists, and atom/ was the same in all three. Since 08a84e6 this PR has merged #2709 seven times. Three follow #2365's head: 6bb16af, which brings main 99ebf63 and with it #2871; 9cc4609, with #2365's review commits up to 3fc180e; and 8379b0e, which brings main 825b63b. The fourth, 5c76693, brings #2709 at 9e9f07e: #2365's later heads up to b62bf7a, with main a8946ff, and #2709's own review fixes. The fifth, 12c280e, brings #2709 at d718458, which follows #2365 onto main ab4ada4 and with it Kimi-K3.1 (#3003). The sixth, 2e7c236, brings #2709 at b82e2d0, which follows #2365 onto main 058c219 and with it Chat token logprobs (#2856). The seventh, d661fb2, brings #2709 at 0ab7e65, which merges main 64412f1, where #2365 has landed, and with it #2913. This PR's own commits since 08a84e6, 642c78d and 22a7112, change only comments, the recipe's text and the C1 mono log test; b1140ca and 7c75287 cancel out. The cells have not been re-run on that stack. After #2709 was split, this PR's diff was rebuilt as one commit on #3070; the ATOM code under it is #2709's previous head, 4034713, merged with main 0fe9932, plus the stack's review fixes since. tok/s/GPU is computed as InferenceX does, over a 3600 s window, against the pre-split sweep:

Cell	Run	tok/s/GPU	Pre-split	ITL p50 (ms), this vs pre-split	P90 TTFT	GSM8K
C1	41196318546	984.8	912.0 (+8.0%)	4.51 vs 5.04	2.32 s	0.9629, at concurrency 1
C2	41280920165	926.8	920.2 (+0.7%)	6.10 vs 6.17	1.76 s	–
C4	41306544667	1,447.3	1,457.3 (−0.7%)	6.18 vs 6.26	2.01 s	0.9629
C16	41266734501	5,318.5	5,335.8 (−0.3%)	8.47 vs 8.37	2.23 s	–
C32	41196315911	10,303	10,441 (−1.3%)	11.35 vs 10.91	4.34 s	–
C48	41272956009	12,999	13,291 (−2.2%)	15.56 vs 14.52	5.02 s	–
C56	41279674613	14,201	14,684 (−3.3%)	17.76 vs 16.75	5.99 s	–
C72	41268221658	16,843	17,118 (−1.6%)	20.80 vs 19.92	11.29 s	0.9560
C88	41165801494	18,688	18,902 (−1.1%)	24.40 vs 23.54	13.47 s	0.9591

GSM8K is exact_match,flexible-extract, the score the 0.94 gate reads. C88's comes from the verify branch only.

The cells launch the same servers as the pre-split runs.

• I expanded every Kimi-K3 cell before and after through pd_matrix.py and pd_submit.sh. The nine cells are identical, except in the PP8 prefills. Those no longer set HIP_VISIBLE_DEVICES, since the launcher derives the same 0,…,7 from TP1 × PP8. They also set ATOM_PD_MLA_STAGING_CHUNK_PAGES to 128, the value they used to get by default. C1 sets ATOM_MONO_ENABLE=1 on its decode only, so its prefill and router now get the common 0. That prefill never armed mono: its TP8/EP8 servers logged Kimi-K3 mono decode off: expert parallel.
• Every role's effective --online_quant_config holds the same set of exclusions: 14 on the C1 prefill and 19 on its decode, 15 on the PP8 prefills and 14 on their decodes. The GSM8K runs of C1, C72 and C88 above used them.
• In C1's run above, node 0 found both expect_log lines in the decode's log before the first request, in the benchmark and in the eval. It found neither reject_log line, either before the first request or after the 60 s wait that follows the last.
• requires_atom_env, run against each ATOM's envs.py, fails the PP8 cells on main's and on perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786's, naming the staging settings, and in C88 the in-flight cap as well. C1 to C16 require nothing. On this stack's it passes every cell.
• On 7d1572b the generated matrix differs from cacf08d's only by those two settings: each of the nine cells loses ATOM_PD_PROMPT_IDS_B64 on its prefill, ATOM_PD_EARLY_T0 on its decode, and both from requires_atom_env. On bec310d, this PR's last commit of its own, it differs from a9c19ef's only by the fresh poll: each of the nine cells and the profile cell loses ATOM_KV_POLL_FRESH_WAIT_MS from its env and from requires_atom_env.
• Unit tests: on this head, 9ab67e3, tests/test_atomesh_*.py give 240 passed. The head restores tests/test_atomesh_cell_checks.py and tests/test_atomesh_torch_traces.py, which the split had dropped, and adds a test of the per-role --online_quant_config. They cover the empty-list failure, a reject_log line that reaches the log while the launcher waits, C1's 1 + num_speculative_tokens width, the envs.py probe and the name-based trace wait. The ATOMesh config job runs these tests on a runner without torch, so the one test that needs torch, C1's mono log lines, is skipped there and runs in Pre Checkin. main's ci: remove Atomesh TW weekly schedule #3071 and Add defense-in-depth guards for self-hosted runner jobs #3153 reach two of this PR's files, atomesh-benchmark.yaml and test_atomesh_v4_workflow.py, in lines this PR does not change. Pre Checkin on 9ab67e3 (perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070's head 01ac6d6 merged in): 13,243 passed, 1,299 skipped, 3 xfailed (43225368625).

Torch profile with this PR at 08a84e6 (41276944873, the PP8 C72 profile cell, g08 prefill, g20 decode). The head's profile window is the same.

• stop_profile returned at 06:18:38Z, 38 s after start_profile. Node 0 logged "all 16 rank traces on disk" at 06:18:43 and stopped aiperf.
• The prefill wrote one trace into each of torch_traces/prefill-rank-0/pp0_rank_0 … pp7_rank_0, 23.9–26.0 MB each. The decode wrote one into each of torch_traces/decode-rank-1/rank_0 … rank_7, 54.7–54.9 MB each. No raw .json was left.
• The names carry export times from 06:18:32 to 06:18:35, while the mtimes run from 06:18:08 to 06:19:02, five of them before the export. That skew is why the first run at 048c838 waited out the timeout (41266652593). That run's prefill also crashed later with a heap corruption, which the rerun did not hit.

[valarLip]

Thanks — I reviewed HEAD 66fc5a30a (base zwan/k3-pp); the related unit tests (118) pass locally, and I expanded the new cells with pd_matrix.py to check the merged env/args per cell.

The two points that matter most:

1. The C1 -mono cell can't arm mono at this head — Kimi-K3's config_refusal refuses any KV connector, and a P/D decode always has one, so non-mono numbers would be published under the mono name (models_atomesh.yaml:1279).
2. The async gzip thread is non-daemon and never joined, so shutdown can kill it mid-compression; the root cause of the stall is gzip's default compresslevel=9 (model_runner.py:1307).

Also: three new env vars are only implemented on #2786, with nothing enforcing the merge order (:1309). The rest are docs/comments, parsing, tests and CI cost.

Checked and fine: the */ glob, the exception paths after the _on_trace_ready split, the merged YAML (max_num_seqs, CPU memory tiers, OPUS, PP layer split totalling 93), and the bash under set -e.

[wanzhenchn]

Thanks for the careful review. All 11 points are addressed on the new head a4d277e, force-pushed on the same base, and the description is updated. One validation is still open: a PP8 profile run with the new gzip level (point 2) is waiting for a free node pair.

1. -mono cell can't arm mono at this head: fixed by failing the cell when mono does not arm. At this head config_refusal does refuse any KV connector. #2786 lifts that refusal for a NIC-only P/D consumer (_nic_only_pd_consumer), which the C1 decode is, and the C1 run in the description logs Kimi-K3 mono decode armed (TP8) on all eight decode ranks. To stop relying on merge order, a role can now declare expect_log (0fe07ed). Before the benchmark, node 0 checks every server log of that role and fails the cell if any of them lacks the line, or if the role left no log. The C1 decode expects Kimi-K3 mono decode armed, so a decode whose config mono refuses fails the cell instead of publishing plain-model numbers under the -mono name. Point 5's gate also stops these cells on an ATOM without #2786.

2. Async gzip thread never joined: fixed by removing the thread. I agree the root cause is the gzip level. ATOM_PROFILER_ASYNC_GZIP and the thread are gone. The trace is gzipped inside stop_profile as before, at compresslevel=3 instead of 9 (a4d277e). On a 265 MB trace from a PP8 stage, level 3 takes 1.3 s against level 9's 8.4 s, for a file about 25% larger. Every caller of stop_profile gets the shorter stop, and there is no shutdown race and no half-written file. A PP8 profile run on this head is waiting for a free node pair; I'll post it here.

3. Doc's completion rule: moot. The doc row went with the flag. With gzip back inside stop_profile, a trace is complete when stop_profile returns. wait_for_torch_traces keeps its rule (a .gz newer than the window start and no raw .json).

4. == "1" instead of _flag_env: moot. The variable is removed.

5. Three env vars implemented only on #2786: gated. A cell's requires_atom_env now lists the atom.utils.envs names it is measured with (12a1d3e).

• Before starting any server, the launcher loads the envs.py of the ATOM under test on its own (it imports only the standard library). If a listed name is undefined, it exits and names the missing settings.
• The nine cells require ATOM_KV_POLL_FRESH_WAIT_MS, ATOM_PD_PROMPT_IDS_B64 and ATOM_PD_EARLY_T0.
• The check passes on perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786's ATOM and fails on feat(kimi-k3): PP8 P/D prefill 1/4 — PAGE-backed state checkpoints under PP #2709's. If this PR lands first, the nine cells fail loudly instead of running without those settings.
• I used an explicit list rather than checking every ATOM_* name in the YAML, because some of those names are not in envs.py on purpose. For example, ATOM_HOST_IP is read directly by atom/utils/network.py, and ATOM_PD_RANK_MAPPING_POLICY is the launcher's own.

6. "PP engines (the prefill) ignore it" sits in env.common: comment fixed. The fresh poll runs on any engine with requests parked on a KV load:

• every decode, which waits on the prefill's KV;
• the C16 TP8EP8 prefill, whose CPU-tier loads park requests too.

The C1–C4 prefills park nothing, and the PP8 prefills ignore the setting, since PPEngineCore._poll_kv_transfer_progress has no fresh wait. The comment now says this. The config is unchanged.

7. Gzip failure logged as an export failure; stale stop_profiler docstring: fixed. The export moved to atom/utils/trace_export.py. An export failure logs "failed to export the trace to …". A compression failure logs "exported the trace, but failed to compress … to …". Either way neither file is left behind. With gzip back inside stop_profile, the docstring is accurate again.

8. ReplaySSM note says "off again from CONC 32": the note was stale. ReplaySSM defaults to on whenever drafts run (gdn_attn.py). AgentX runs it off at CONC 1/2/4 and on from CONC 8. The old "off from CONC 32" described the no-draft TP8 cells, not a band, and the existing DSpark4 C48–C88 cells already set "1". The note now says: off at C1/2/4, on from C8 wherever drafts run, and off on the cells without drafts. The PP8 cells keep "1".

9. No tests for the model_runner path: added for what remains. With the async path gone, tests/test_trace_export.py covers the rest:

• a successful export: the .gz decompresses to the trace, the raw JSON is removed, and the gzip level is 3;
• an export failure and a compression failure: each leaves no file and logs its own message.

The new launcher checks have tests in tests/test_atomesh_cell_checks.py and tests/test_atomesh_online_quant.py.

10. Copy-pasted online_quant_config and argparse last-wins: fixed. A new key, online_quant_extra_exclude_layer (a155ecf), can be set in server.common_args for both roles or on a role. pd_submit.sh appends it to the model's online_quant_config exclude list and exports one config per role. The launcher then passes --online_quant_config once per role, built from that config.

• The cells now carry only the additions: the drafter's seven linears (shared), mono's five projections on the C1 decode, and *self_attn.f_b_proj on the PP8 prefills. No JSON is copied, and no cell's quantization depends on argument order.
• A role extension without a base config, or a base config that is not JSON, fails the cell at submit.
• I expanded all 30 Kimi-K3 cells through pd_matrix.py and pd_submit.sh before and after the change. Every role's effective exclusions are the same set: 14 on the C1 prefill and 19 on its decode, 15 on the PP8 prefills and 14 on their decodes.

11. Nine new cells roughly double Kimi-K3's nightly: they now replace the old ones. The nine cells replace the 13 older 1P1D cells at the same concurrencies (edc2f1b): TP8 DSpark7 C1/2/4, DSpark3 C16, no-draft C32–C88, and the TP8×DCP8-prefill DSpark4 C48–C88.

• Kimi-K3's nightly goes from 20 cells to 16, and each concurrency publishes one set of numbers.
• The DSpark3 C8, no-draft C64, 1P2D, AGG, 2P1D and profile cells stay, and they expand to exactly the same cells as before. The old C4's GSM8K eval moves to the new C4.
• recipes/mesh/Agentic-Kimi-K3.md pointed at the retired DSpark4 cells. It now names the PP8 cells, which run its decode and state cache behind a PP8 prefill.
• The dashboard's TP8-prefill curves at those concurrencies end with this change.

273 tests pass across tests/test_atomesh_*.py, tests/test_envs.py and tests/test_trace_export.py.

[valar-cop]

not ready yet..

any perf number for the C16 prefill cell? gsm8k at C88 with opus? and the PP8 profile run on this head?

[valar-cop]

some changes needed

any perf number? gsm8k for C1 and C88? and the C16/PP8 runs are still pending?

[wanzhenchn]

Thanks. Your review is on a4d277e; since then 324361a trimmed Kimi-K3's nightly to the nine AgentX cells and moved the profile cell to the nightly PP8 C72. Seven points are addressed in three commits on top (e184df3, 8f2484c, and 17242ca, now the head; no force-push). Two are answered below without a change. For your three questions: the PP8 profile run is going, and C16 and C88 GSM8K will run when node pairs free up.

-mono can't arm here: kept, and it fails rather than mismeasures. At this head config_refusal does refuse any KV connector. #2786 lifts that for a NIC-only P/D consumer (_nic_only_pd_consumer), which this decode is, and this PR merges after #2786. Until then two checks keep the cell from publishing plain-model numbers under -mono:

• The C1 decode declares expect_log: "Kimi-K3 mono decode armed". Before the benchmark, the cell fails if any decode rank lacks that line.
• requires_atom_env fails all nine cells on an ATOM without perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786.

On #2709 + #2786 + this PR, the C1 run (40634392873) logged Kimi-K3 mono decode armed (TP8) on all eight decode ranks.

*/ counts any subdir as a rank: fixed (8f2484c). Only the names ModelRunner gives a rank's dir count: rank_<tp> or dp<d>_tp<tp>, prefixed pp<k>_ under PP. A worker with no rank dir yet still counts as one pending rank. New tests cover a stray tmp/ and rank_0.cache/ next to the rank dirs, and DP rank dirs.

trace_export.py for a CI fix: agreed, reverted (e184df3). _on_trace_ready is back to main's inline export. The only ATOM change is compresslevel=3 on its gzip.open, with a comment on why, and atom/utils/trace_export.py and its test are gone. The separate export and compress failure messages were for the background gzip thread, which the previous round removed. The level isn't CI-only, though: every stop_profile caller waits for this gzip.

Envs only on #2786: gated by merge order, enforced at launch. Each cell's requires_atom_env lists the three settings. Before any server starts, the launcher loads the envs.py of the ATOM under test and fails the cell, naming what is missing. If this PR landed first, the nine Kimi-K3 cells would fail at launch rather than run without the settings, and other models' cells are not affected. An image-tag gate would not help: scheduled runs use atom_source=mounted, so the ATOM under test is the checked-out main, whatever the image. The order is #2709, then #2786, then this PR.

Kernel-shape exclusion in the YAML: comment added (17242ca). The comment now says to drop the exclusion once aiter's FP8 GEMM serves K 128. aiter has no tuned FP8 config for that GEMM (N 12288, K 128), and its untuned FP8 GEMM needs K above 192. A fallback for that shape is an aiter change, so until then the exclusion stays, scoped to the PP8 prefill only.

OPUS not in the name: added (17242ca). C72, C88 and the profile cell now end in -opus. The dashboard labels a series by display_topology, so their label ends in -OPUS too, and C1's label ends in -MONO for the same reason. Result file names use the topology, not the label, and these cells have no dashboard history yet, so nothing else moves. I didn't keep a non-OPUS cell, since the nightly is now one cell per concurrency. The OPUS A/B is in the description: C72 +4.2%, C88 +5.2%.

fail() exits 0 and prints exit 1: fixed (8f2484c). fail() now also exits 1. Its only caller, eval "$(python3 - <<'PY' … PY)", ignores the status, so the printed exit 1 is still what stops the submit, and a caller that checks the status now sees the failure. The comment states both, and the test asserts both. The existing unsupported-server_args check goes through fail() too.

expect_log before warmup: contract stated (8f2484c). The line must be logged while the server starts. The check runs once every server is healthy and before any request, aiperf's warmup included; only /health and /v1/models are probed before it. A line first logged while serving would fail a healthy cell, and the comment now says so. Mono's line qualifies: install_mono_decode logs it from ModelRunner.__init__, at model load.

C88's 176 vs "neighbouring band": comment made precise (17242ca). 176 is 2 × 88, not C72's band. The recipe sets max-num-seqs to 2 × CONC at every point (144 at C72), and C88 follows that rule. "Takes the neighbouring band" did read as if max_num_seqs came along, so the comment now says CONC 32 and 88 take the settings of the recipe point below them, CONC 16 and 72, except max_num_seqs.

Your three questions:

• C16: the description's table has the C16 cell at 5,336 tok/s/GPU from the pre-split sweep (10-24 image), and the C16 prefill A/B was TP8EP8 5,317 vs PP8 5,204. It hasn't run on the split yet. I'll run it on perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this head when a node pair frees and post the number.
• GSM8K at C88 with OPUS: not measured yet. Only C72 was: 0.9560, 0.9416 and 0.9424 strict over three OPUS runs. zwan/k3-agentic-verify is perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this head, plus run_eval on C88 for that run only. It will run there when nodes free.
• PP8 profile on this head: running as 41023308365: the PP8 C72 OPUS profile cell on perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this head, g13/g51. I'll post the result here.

272 tests pass across tests/test_atomesh_*.py and tests/test_envs.py. The nine nightly cells expand as before apart from the renames and labels.

[wanzhenchn]

Thanks. This PR now changes no ATOM code: I force-pushed to drop the two commits that touched model_runner.py (a4d277e, e184df3), and the gzip level moved to #2786. The other commits are otherwise unchanged (git range-diff), so the ones named in my last reply have new ids: 324361a → fab94aa, 8f2484c → aad2480, 17242ca → ef89e34. Four points are fixed in a1c7dc4, the new head, and four are answered below without a change.

ATOM change in a CI PR, and 20% vs 25%: moved to #2786 (1671386dc). It is #2786's item 7 now. The comment there gives one figure, the measured one: on a PP8 stage trace, level 3 is ~6x faster for a file ~25% larger (1.3 s vs 8.4 s on 265 MB).

Hard-coded 8 devices: fixed in the launcher (a1c7dc4). A role's default HIP_VISIBLE_DEVICES now covers TP × PP, with PP parsed from the role's server args as the packed layout already did, and the PP8 cells no longer set it. Expanded through pd_matrix.py, the cells differ only in that missing key, and the launcher derives the same 0,…,7 from TP1 × PP8. A cell that sets the variable keeps its value, so the GLM PP cells, which list their GPUs, are unchanged. tests/test_atomesh_role_gpus.py covers PP, TP × PP, a decode without PP and an explicit value.

A worker with no rank dir holds the wait open: fixed (a1c7dc4). The wait no longer assumes a pending rank per worker. The ranks are the trace_dirs the stop files reported, plus their siblings named like a rank, so a dir no stop file points at, and that isn't named like a rank, can't hold it open. The new tests cover a reported dir with a non-rank name, which still counts, and a reported dir with no trace, which stays pending.

fail() comment: claim dropped (a1c7dc4). It now says only that pd_submit.sh runs this through eval "$(...)", which ignores the status, so the printed exit 1 is what stops it. That eval is the only caller.

Explicit ATOM_MONO_ENABLE: "1" over the merged "0": comment added (a1c7dc4). It says the override is on purpose, since C1 is the only cell that runs mono.

expect_log on every node: it runs on node 0 only. check_expected_logs is called only from run_benchmark_and_eval, and all five call sites run on node 0 alone: the aggregated branch exits on other ranks first, the packed branch calls it under NODE_RANK -eq 0, and the other three are NODE_RANK -eq 0 branches. Node 0 reads the other nodes' server logs from the shared RUN_DIR, as the comment above the function says.

Log string from #2786: it is main's, and a changed line fails the cell. Kimi-K3 mono decode armed (TP%d) is logged by atom/models/kimi_k3/mono/dispatch.py on main today; #2786 doesn't add or change it. If the wording changes, C1 fails at launch, so it can't publish plain-model numbers under -mono. A stable marker such as ATOM_MONO_ARMED would be an ATOM change, so I'd add it in a separate ATOM PR rather than here, and point expect_log at it then.

Mono exclusions in the cell: kept. They are a property of the mono deployment, not of the model: the other Kimi-K3 cells quantize those five layers. InferenceX#4307 does the same, putting them in its CONC 1 point's online_quant_config. Making mono apply them by itself would be an ATOM change to mono's config, so like the marker it belongs in an ATOM PR. Until then the cell is the only place that knows the decode runs mono.

Your three questions:

• Perf numbers: the description's Validation table has the split's numbers on feat(kimi-k3): PP8 P/D prefill 1/4 — PAGE-backed state checkpoints under PP #2709 + perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this PR: C88 18,564, C72 17,143 (GSM8K 0.9424), C32 10,186 and C1 984 tok/s/GPU, each with its run.
• GSM8K at C1 and C88: not measured yet. zwan/k3-agentic-verify is now perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 (1671386) + this head, with run_eval on C1 and C88 for those runs only.
• C16 and PP8 profile: still pending. The profile run in my last reply (41023308365) was cancelled while it waited for its node pair, which another job held. The profile, C88, C1 and C16 runs go out in that order as node pairs free up, and I'll post the results here.

278 tests pass across tests/test_atomesh_*.py and tests/test_envs.py.

[valar-cop]

not ready yet..

any perf number? gsm8k at C1/C88? and the new tests are not run by this pr's ci

ci green.

[valar-cop]

some changes needed, mostly doc/comment fixes and two coverage questions..

any e2e run of the nine cells at this head? gsm8k at C1/C88? C16 on the split?

ci green.

[wanzhenchn]

Thanks. The three inline points are fixed in 7961a19aa, the new head. Its parent commits are unchanged: #2709's head moved to 0c3b9f1, which changes only its last commit's date, and this branch was rebased onto it (git range-diff: every commit =).

A single exclusion pattern became one-character patterns: fixed. pd_submit.sh now reads online_quant_extra_exclude_layer, on server.common_args and on a role, as pd_matrix.normalize_list does: one value is a list of one. The base config's exclude_layer is read the same way. test_a_single_pattern_is_one_pattern gives the decode "*self_attn.o_proj*" as a string and checks that it lands as one pattern on the decode and not on the prefill.

Non-object JSON died later on an unrelated unbound variable: fixed. If online_quant_config parses to anything but an object, the cell now fails right there with online_quant_config is a JSON list, not an object (or whatever type it is), via fail(). test_a_config_that_is_not_an_object_fails_the_cell checks that message and the trailing exit 1.

Mono in env.common reached the prefill: moved to env.decode. C1's common block keeps the shared ATOM_MONO_ENABLE: "0", and its decode sets "1". Expanded through pd_matrix.py, the only difference across all Kimi-K3 cells is that C1's prefill and router now get 0. The prefill never armed mono anyway: on the C1 run its TP8/EP8 servers logged Kimi-K3 mono decode off: expert parallel, so its servers behave as before. The decode still expects Kimi-K3 mono decode armed.

The new tests and this PR's CI. Pre Checkin, which runs tests/ through run_unit_tests.sh, triggers only on PRs into main. This PR targets zwan/k3-pp and #2709 targets #2365's branch, so neither got it. I dispatched it on both heads, and both pass: this one in 41044757047 (10,240 passed, 1,091 skipped) and #2709's in 41044760804 (10,212 passed, 1,091 skipped). The job collects all of tests/ except tests/plugin/. All 16 tests/test_atomesh_*.py files ran, this PR's new ones included, and none of their tests were skipped. Locally, 281 tests pass across tests/test_atomesh_*.py and tests/test_envs.py.

Your other questions:

• Perf numbers: the description's Validation table has the split's numbers on feat(kimi-k3): PP8 P/D prefill 1/4 — PAGE-backed state checkpoints under PP #2709 + perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this PR, each with its run: C88 18,564, C72 17,143 (GSM8K strict 0.9424 ± 0.0064), C32 10,186 and C1 984 tok/s/GPU.
• GSM8K at C1 and C88: not measured yet. zwan/k3-agentic-verify is now perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 (a5f0fc4) + this head, with run_eval on C1 and C88 for those runs only, so the C1 run will also exercise the decode-only mono setting. The runs go out as node pairs free up, and I'll post the results here.

[wanzhenchn]

Thanks. Two points are fixed in 2c1ad68a4, the new head, now on #2709's 687a3ba (git range-diff: every earlier commit =). The other two are answered below.

Profiler comment over build_online_quant_args: fixed. The function now sits above that comment, which is back on build_torch_profiler_args.

The recipe says OPUS but never sets it: fixed. Its decode block now exports AITER_MLA_USE_OPUS=1 at CONC 72 and 88, as the -opus cases do, and the table has a column for it. The default CONC=72 therefore reproduces the OPUS decode.

Dropping the agg 1M, 1P2D and 2P1D cells: intended. They were added to check that CI's agg path and those topologies worked and performed as expected. This nightly now covers only P/D, on the InferenceX AgentX recipe points. Kimi-K3 aggregated serving is still covered nightly by ATOM's own benchmark (atom-benchmark.yaml) and accuracy (models_accuracy.json, TP8 with DSpark), though not at 1M agentic.

C72 OPUS gate margin. The gate compares exact_match,flexible-extract with the model's 0.94. Flexible-extract on the three OPUS runs:

Run	strict	flexible-extract	margin over 0.94
40546704877	0.9560	0.9553	+0.0153
40421767877	0.9416	0.9409	+0.0009
40690480833	0.9424	0.9416	+0.0016

The low runs are not OPUS's alone. The same cell without OPUS, on the same image, gave 0.9522 and 0.9530 (40525157531, 40292434362), and also 0.9409 and 0.9416 (40184716007, 40351829339). Over these seven runs the mean is 0.9465, against a single-run stderr of about ±0.0062. So at 0.94, roughly one run in seven would fail with no regression behind it. I've kept the model's 0.94 for this cell rather than give it its own lower threshold.

Your other questions:

• An e2e run of the nine cells at this head: not yet. The last full set predates the split, and the split's runs are those in the description's Validation table. zwan/k3-agentic-verify (perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786 + this head) runs the profile, C88 and C1 with GSM8K, and C16, as node pairs free up.
• GSM8K at C1 and C88: not measured yet; both are in that verify set.
• C16 on the split: not run yet; it is the last of that set.

I'll post the results here.

[valar-cop]

some changes needed, the guards are not fail-loud yet..

any e2e run of the nine cells at this head? gsm8k flexible-extract number?

ci green.

[valar-cop]

a few small things..

e2e of the nine cells at this head incl C16? gsm8k at C1? pp8 C72 profile on this head?

ci green.

[wanzhenchn]

e2e of the nine cells at this head incl C16? gsm8k at C1? pp8 C72 profile on this head?

All nine are in the description's Validation table, C16 included (41266734501, 5,318.5 tok/s/GPU, −0.3% against the pre-split sweep). They ran on zwan/k3-agentic-verify in rocm/atom-dev:nightly_202610301458. That branch is #2709 at bd26837, #2786 at a5f0fc4 and this PR, which C2, C4, C56 and the profile cell carried at 08a84e6 and the rest at 048c838 and 2840ea1. Between those this PR changed only the profile wait, C1's last reject_log check, how the launcher finds envs.py and pd_submit.sh's empty-list check.

• GSM8K at C1: 0.9629 (±0.0052), strict 0.9629, at concurrency 1 (41196318546).
• PP8 C72 profile: 41276944873, at 08a84e6. stop_profile returned 38 s after start_profile, and all 16 rank traces were on disk 5 s later. The head's profile window differs from it only by b1140ca.

Not at this head: since 08a84e6 this PR has followed #2709 onto #2365's head (6bb16af, which brings main 99ebf63 and with it #2871) and added b1140ca. The cells have not been re-run on that stack.

[valar-cop]

a few small things..

ci green.

[valar-cop]

a few small things..

ci green.

[valarLip]

⚠️ Action required before merge (4)

1. Hold the merge until the stack and #2786 land. This PR targets zwan/k3-pp-perf (#3070 → #3069 → #3068, all still open), and the C1 mono cell's expect_log/reject_log lines (Kimi-K3 mono decode armed/on/declined the 8-row build) exist only on #2786 (open, targets main). On main today the PP8 cells fail on requires_atom_env and C1 fails on expect_log, so the nightly would go red. Merge after all four.
2. Restore the launcher tests that the split dropped. tests/test_atomesh_cell_checks.py and tests/test_atomesh_torch_traces.py are gone at this head. The PR now adds ~240 lines to .github/scripts/atomesh/pd_server_atom.sh (parse_log_lines, check_expected_logs/check_rejected_logs, check_required_atom_env, the name-based wait_for_torch_traces with pp<k>_rank_*, TP×PP HIP_VISIBLE_DEVICES) and role_online_quant_config/role_log_lines to pd_submit.sh, but only stubs them out (tests/test_atomesh_v4_workflow.py:110-111, tests/test_atomesh_mooncake_store.py:379). Earlier review replies said tests covered the empty-list failure, the delayed reject_log line, C1's 1 + num_speculative_tokens width and the envs.py probe; none of those tests is in this diff. Bring them back, or split them into a follow-up PR that lands with this one, so the guards are tested.
3. Fix the CONC 32/88 comment. .github/benchmark/models_atomesh.yaml:1012-1014 says CONC 32 and 88 "each takes the settings of the recipe point below it, CONC 16 and 72, except max_num_seqs". But C32 runs the PP8 prefill (C16 runs TP8EP8×DCP8), and C88 adds ATOM_PP_MAX_INFLIGHT=16, two offload workers and run_eval: false. Scope the sentence to the decode settings and list the exceptions.
4. Fix the garbled comment. At .github/scripts/atomesh/pd_server_atom.sh:242, "sets each role's from its …" is missing its object. Write "sets PREFILL_/DECODE_ONLINE_QUANT_CONFIG from each role's online_quant_extra_exclude_layer".

---

_{Re-reviewed at 4ecc15f. Already fixed: C1 mono now gated by #2786 plus expect_log/reject_log; gzip/async-thread and _flag_env items moved to #2786; three unread env vars dropped, merge order now enforced by requires_atom_env; online-quant JSON deduped via online_quant_extra_exclude_layer; old nightly cells retired; ReplaySSM note fixed; trace-dir glob limited to rank names; empty-list and reject-after-NFS-wait handled; C1 GSM8K enabled. CI: lint and unit tests green, Pre Checkin green; the one red "ATOM Test" run was cancelled and superseded.}

[wanzhenchn]

@valarLip Thanks. Items 2 to 4 are in 49b967c and 0f385d4; item 1 stands as the merge plan.

1. Agreed. This PR stays based on perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070 and lands only after feat(kimi-k3): PP8 P/D prefill 2/4 — LMCache state tier under PP #3068, feat(kimi-k3): PP8 P/D prefill 3/4 — KV and KDA state from a PP prefill to a DCP × TP decode #3069, perf(kimi-k3): PP8 P/D prefill 4/4 — MLA staging for a DCP consumer, more PP batches in flight #3070 and perf(kimi-k3): serve mono decode behind a Mooncake P/D decode; gzip torch traces at level 3 (from Kimi-K3 AgentX 1M) #2786. Until then requires_atom_env fails the PP8 cells on main, and the C1 decode never logs the armed line its expect_log needs, since main refuses mono behind Mooncake.
2. Restored tests/test_atomesh_cell_checks.py and tests/test_atomesh_torch_traces.py. Both pass unchanged on this tree: they cover parse_log_lines, check_expected_logs and check_rejected_logs (including the empty list and a reject_log line that reaches the log during the wait), check_required_atom_env against envs.py, C1's 1 + num_speculative_tokens width, role_log_lines, and the name-based wait_for_torch_traces with pp<k>_rank_*. 0f385d4 adds SubmitOnlineQuantTest for role_online_quant_config: a role's extra exclusions reach only its own command line, and extras without a shared config fail the cell. The TP × PP HIP_VISIBLE_DEVICES is top-level launcher code rather than a function; the PP8 runs log [runtime] HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 for TP1 × PP8. tests/test_atomesh_*.py: 240 passed.
3. The note now scopes "the recipe point below it" to the decode and lists the prefill differences: CONC 32 runs the PP8xTP1 prefill instead of CONC 16's TP8EP8xDCP8, and CONC 88 adds ATOM_PP_MAX_INFLIGHT=16 and two offload copy and load workers to CONC 72's and skips GSM8K.
4. Now reads "pd_submit.sh sets PREFILL_/DECODE_ONLINE_QUANT_CONFIG from each role's online_quant_extra_exclude_layer."