Skip to content

Commit 3c61360

Browse files
Jasen2201claude
andcommitted
Merge remote-tracking branch 'origin/main' into jasen/glm52-mi355x-atom-disagg-agentx
Resolve the perf-changelog conflict with prepare_perf_changelog_merge resolve-conflict, which keeps this PR's entry as the tail append, and keep both rows in runners/srt-slurm/patches/README.md: 602 from main and 604 from this PR. All three patches apply in order on srt-slurm 545233a0d. 用 prepare_perf_changelog_merge resolve-conflict 解决 perf-changelog 冲突, 本 PR 的条目仍是末尾追加;runners/srt-slurm/patches/README.md 保留 main 的 602 和本 PR 的 604 两行。三个补丁按顺序都能打到 srt-slurm 545233a0d 上。 [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 parents 4b2c5ea + c2aa0c3 commit 3c61360

56 files changed

Lines changed: 4269 additions & 692 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/PULL_REQUEST_TEMPLATE/pull_request_template.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,8 @@
3232
- [ ] I have completed the AI model disclosure and kept it current
3333
- [ ] I have tested my changes locally
3434
- [ ] I have updated documentation if necessary
35-
- [ ] **For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of `inferencex-e2e/perf-changelog.yaml` and have not edited historical entries**
36-
- [ ] **If this PR can affect benchmark performance or adds or modifies a recipe, it carries exactly one primary sweep label** (a maintainer applies it on fork PRs): `full-sweep-fail-fast` (recommended), `full-sweep-enabled`, or `non-canary-full-sweep-enabled`. Optional modifiers `all-evals`, `evals-only`, and `agentx-fast` require a primary label; the last two block reuse while applied.
35+
- [ ] **If `python -m infx.matrix.changed --base origin/main` lists affected config keys, I have appended one new entry covering them to the physical end of `inferencex-e2e/perf-changelog.yaml`; if it lists none, I have added none. I have not edited historical entries**
36+
- [ ] **If this PR appends an `inferencex-e2e/perf-changelog.yaml` entry, it carries exactly one primary sweep label** (a maintainer applies it on fork PRs): `full-sweep-fail-fast` (recommended), `full-sweep-enabled`, or `non-canary-full-sweep-enabled`. Optional modifiers `all-evals`, `evals-only`, and `agentx-fast` require a primary label; the last two block reuse while applied.
3737
- [ ] **Before merging via reuse, an authorized maintainer (`OWNER`/`MEMBER`/`COLLABORATOR`) has commented `/use <run_id>` (or the legacy `/reuse-sweep-run`) on this PR**. Do this **only once there is a final full sweep that is all green with evals passing**, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
3838

3939
</details>

‎.github/workflows/claude.yml‎

Lines changed: 7 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ jobs:
184184
- Only wait for the final successful run to complete before analyzing benchmark results
185185
- Do NOT claim completion until the most recent job finishes and results are analyzed
186186
- If jobs cannot be run, say exactly what you could not run and why
187-
- **Important** Modify inferencex-e2e/perf-changelog.yaml for any config changes affecting performance
187+
- **Important** Append an inferencex-e2e/perf-changelog.yaml entry when the change alters what any benchmark runs (see "Updating perf-changelog.yaml" below)
188188
189189
## Profiling (SGLang only)
190190
When asked to profile a config, dispatch the `profile.yml` workflow. **Only SGLang configs can be profiled** — the profiler uses SGLang's `/start_profile` and `/stop_profile` HTTP endpoints. Reject profiling requests for vLLM, TRT, or other frameworks.
@@ -234,7 +234,7 @@ jobs:
234234
235235
## Updating perf-changelog.yaml
236236
237-
See `AGENTS.md` → "Non-negotiable benchmark invariants" for changelog rules and `inferencex-e2e/perf-changelog.yaml` for entry format. Required whenever you change image tags, env vars, or perf-affecting params in `inferencex-e2e/configs/*-master.yaml` or `inferencex-e2e/benchmarks/*.sh`. Use `XXX` as the PR-link placeholder until the PR exists.
237+
See `AGENTS.md` → "Non-negotiable benchmark invariants" for changelog rules and `inferencex-e2e/perf-changelog.yaml` for entry format. Add an entry when the change alters what any benchmark runs: run `uv run python -m infx.matrix.changed --base origin/main` from `inferencex-e2e/` and cover every config key it lists; if it lists none (refactors, recipe moves, docs), add no entry. Use `XXX` as the PR-link placeholder until the PR exists.
238238
239239
If an entry uses `append-only: true`, require the generated base matrix to remain an immutable subset of the generated head matrix. New concurrency values or new recipe variants may be added inside a selected existing config/scenario, but no existing generated point may be removed or modified, and every addition must retain the target visual curve's single non-null image. Do not enforce a file allowlist: supporting code, benchmark scripts, launchers, helpers, and other files may change when their benchmark effect is exclusive to the corresponding newly appended points. All added changelog entries must be append-only, and eval modifiers are forbidden. Trace behavior through the complete diff; unguarded changes or paths reachable by an existing point are blocking.
240240
@@ -397,15 +397,12 @@ jobs:
397397
- For re-reviews, prefix summary with "Re-review:" and note what changed
398398
399399
## Master Config and Perf Changelog Validation:
400-
When reviewing a PR, check if any of the following master config files were modified:
401-
- `inferencex-e2e/configs/amd-master.yaml`
402-
- `inferencex-e2e/configs/nvidia-master.yaml`
400+
A PR needs an `inferencex-e2e/perf-changelog.yaml` entry exactly when it changes what a benchmark runs; the entry triggers result collection.
403401
404-
If either master config file was edited AND `inferencex-e2e/perf-changelog.yaml` was NOT edited in the same PR:
405-
- This is a 🔴 **BLOCKING** issue
406-
- Comment that `inferencex-e2e/perf-changelog.yaml` must also be updated when master config files are changed
407-
- The perf-changelog entry should document what changed in the config and include the PR link
408-
- Format: "Master config files were modified but `inferencex-e2e/perf-changelog.yaml` was not updated. When changing `inferencex-e2e/configs/amd-master.yaml` or `inferencex-e2e/configs/nvidia-master.yaml`, you must add a corresponding entry to `inferencex-e2e/perf-changelog.yaml` documenting the changes."
402+
When a PR changes master configs, recipes, benchmark scripts, or launcher code and does NOT add a `inferencex-e2e/perf-changelog.yaml` entry, run `uv run python -m infx.matrix.changed --base origin/<base branch>` from `inferencex-e2e/`:
403+
- If it lists changed config keys, this is a 🔴 **BLOCKING** issue. Format: "This PR changes benchmark points for <config keys> but adds no `inferencex-e2e/perf-changelog.yaml` entry. Append one entry covering these config keys to the end of the file."
404+
- If it lists no changed config keys and the PR changes no benchmark script or launcher behavior that a benchmark runs, it is not an issue; do not comment on it. The tool does not see script or launcher code, so judge those changes from the diff and treat a run-affecting one without an entry as 🔴 **BLOCKING**.
405+
- If it cannot run, say so in the summary comment and ask whether the change alters what any benchmark runs.
409406
410407
### Perf Changelog Entry Position:
411408
`inferencex-e2e/perf-changelog.yaml` is read in chronological order — oldest entries at the top, newest at the bottom. New entries MUST be appended to the END of the file. Never insert in the middle or prepend.

‎.github/workflows/klaud-plan.yml‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ name: Klaud Cold auto-sweep
44
on: # zizmor: ignore[concurrency-limits]
55
workflow_dispatch:
66
schedule:
7-
- cron: '0 */6 * * *'
7+
- cron: '0 */2 * * *'
88

99
permissions:
1010
contents: read
@@ -81,9 +81,9 @@ jobs:
8181
actions: read # Read workflow runs and artifacts.
8282
pull-requests: read # Read PR metadata.
8383
env:
84-
# Bound one semantic-overlap review; recently attempted same-base candidates sort last.
8584
REVIEW_BATCH_SIZE: '64'
8685
CANDIDATE_COOLDOWN_HOURS: '24'
86+
STALE_PR_DAYS: '7'
8787
outputs:
8888
review: ${{ steps.review.outputs.structured_output }}
8989
outcome: ${{ steps.review.outcome }}
@@ -122,6 +122,7 @@ jobs:
122122
--directory "$RUNNER_TEMP/klaud"
123123
--review-batch-size "$REVIEW_BATCH_SIZE"
124124
--cooldown-hours "$CANDIDATE_COOLDOWN_HOURS"
125+
--stale-pr-days "$STALE_PR_DAYS"
125126
# Uploaded before the agent runs, so selection reads the planner's files, not agent-touched ones.
126127
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
127128
with:
@@ -163,8 +164,9 @@ jobs:
163164
--json-schema '${{ steps.prepare.outputs.review_schema }}'
164165
prompt: |
165166
Use Read on candidates.json, open-prs.json and capacity.json under ${{ runner.temp }}/klaud/.
166-
Review candidates in the supplied shuffled order against current recipes and the complete open-PR index,
167-
including drafts and arbitrary branches. Changed-file paths are already fetched;
167+
Review candidates in the supplied shuffled order against current recipes and the open-PR index,
168+
including drafts and arbitrary branches. The index omits PRs idle for ${{ env.STALE_PR_DAYS }} days
169+
(other than Klaud candidate PRs); those no longer own any family. Changed-file paths are already fetched;
168170
inspect relevant PR bodies/diffs with direct gh pr view/diff commands. For API reads,
169171
use gh api --method GET. Use Read/Glob/Grep for local inspection, not shell file commands
170172
or Python. Run one allowed gh/git command per Bash call, without cd/env wrappers,
@@ -186,7 +188,10 @@ jobs:
186188
to fill the cap. Never copy private eligibility data into reasons or reports.
187189
Mark duplicate when an open PR owns that family's image refresh or overlapping
188190
compatibility work, even with another image tag; unrelated work on the same hardware
189-
is not a duplicate. Return the supplied schema: candidate-id, decision
191+
is not a duplicate. A cross-cutting PR (launcher, telemetry, refactor or shared-schema
192+
work spanning many families) is a duplicate only if its diff changes this family's
193+
image or the exact engine flags/environment a refresh edits; touching the same recipe
194+
file for unrelated settings is not overlap. Return the supplied schema: candidate-id, decision
190195
(proceed/duplicate/uncertain), family (null if unresolved), telemetry-clusters (exact
191196
target IDs, empty if unresolved), pull-requests (overlapping
192197
PR numbers), baseline-model (required for proceed; otherwise null), reason (brief evidence).

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,8 @@ inspect its generated fields, then collect applicable smoke, sweep, and eval evi
137137
## Non-negotiable benchmark invariants
138138

139139
- Every priority-scheduled benchmark job on a self-hosted cluster must request exactly one `nodes:N` label, where `N` is the positive integer number of physical Slurm nodes required. Single-node jobs use `nodes:1`; generated multi-node jobs must forward their computed `node-count`. A queued job missing this label is ineligible for priority scheduling, and labels cannot be added retroactively, so fix the source branch and dispatch a new run.
140-
- Every change that can affect benchmark performance and every recipe addition or modification requires a new `inferencex-e2e/perf-changelog.yaml` entry. The file is append-only and byte-sensitive. Preserve all existing bytes and separator whitespace, and append only at the tail.
141-
- One PR maps to one `inferencex-e2e/perf-changelog.yaml` block. By convention, a PR with a perf changelog entry is a change that affects inference performance or otherwise requires a rerun to collect up-to-date results. That block may list one or more `config-keys` to run, but do not add a second block for the same PR, even when the change evolved across commits. Revise the PR's existing block so it describes the final state of the change.
140+
- A change that alters what any benchmark runs needs a new `inferencex-e2e/perf-changelog.yaml` entry covering the affected config keys; the entry triggers result collection. A change that leaves every benchmark point unchanged (refactors, recipe moves, docs) needs none. For config and recipe changes, `python -m infx.matrix.changed --base origin/main` (run from `inferencex-e2e/`) lists the affected keys; if it lists none, add no entry. It does not see benchmark script or launcher code, so judge those changes by what they run. The file is append-only and byte-sensitive. Preserve all existing bytes and separator whitespace, and append only at the tail.
141+
- One PR maps to at most one `inferencex-e2e/perf-changelog.yaml` block. That block may list one or more `config-keys` to run, but do not add a second block for the same PR, even when the change evolved across commits. Revise the PR's existing block so it describes the final state of the change.
142142
- New `inferencex-e2e/perf-changelog.yaml` entries must be English-only. Do not add Chinese translations or bilingual descriptions; the bilingual README and GitHub-content rules do not apply to these entries. Leave historical entries unchanged.
143143
- Multi-node srt-slurm changes update the recipe YAML and matching master config together. srt-slurm recipes, fixed-sequence and AgentX, are fragments that get the master image, model, precision (and fixed-sequence lengths) from the binder.
144144
- Every speculative fixed-sequence benchmark renders prompts with the chat template: the binder (`inferencex-e2e/infx/srt_slurm/workload.py::bind_workload`) sets `benchmark.env.USE_CHAT_TEMPLATE: "true"` for single-node recipes that speculate, which `python3 -m infx.bench fixed-seq` (run by `srt_fixed_sequence.sh`) turns into `--use-chat-template` for the benchmark client.

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Every PR description must include an **AI model disclosure** section inside a co
2727
3. Ping a core maintainer on Slack for final approval, after obtaining the checklist sign-off when required.
2828
4. An authorized maintainer posts `/use <run_id>` (see below) and the PR is merged via the reuse path.
2929

30-
**Performance changelog requirement:** Every change that can affect benchmark performance and every recipe addition or modification **MUST** append a new entry to the physical end of `inferencex-e2e/perf-changelog.yaml`. Historical entries **MUST NOT** be edited.
30+
**Performance changelog:** A PR that changes what any benchmark runs **MUST** append one new entry covering the affected config keys to the physical end of `inferencex-e2e/perf-changelog.yaml`; `python -m infx.matrix.changed --base origin/main` (run from `inferencex-e2e/`) lists them. A PR that changes no benchmark point adds none. Historical entries **MUST NOT** be edited.
3131

3232
## Draft-model precision
3333

‎collectivex/README.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -221,19 +221,25 @@ level pinning but leave the RoCE-only GID override unset so NVSHMEM/NCCL can use
221221
Mixed Ethernet and InfiniBand HCA lists are rejected.
222222

223223
`rdma_fabric: "efa"` declares an AWS Elastic Fabric Adapter cluster (b300 is a p6-b300.48xlarge
224-
pool with 16 EFA devices per node). EFA is not a verbs HCA: its ports report no link layer, it has
224+
pool with 16 EFA devices per node, h100-dgxc a p5.48xlarge pool with 32). EFA is not a verbs HCA: its ports report no link layer, it has
225225
no GID table, service level or traffic class, and NCCL reaches it only through the aws-ofi-nccl
226226
libfabric plugin that the cluster's enroot hook mounts into every container. On such a fabric the
227227
probe accepts the unspecified link layer for the listed `rdma_devices` (which must still be ACTIVE
228228
on every node), the IB selector family (`NCCL_IB_*`, `NVSHMEM_HCA_LIST`/IBGDA, MoRI, UCCL) stays
229229
unset, `NCCL_NET_PLUGIN=ofi` and `FI_PROVIDER=efa` select the plugin, NVSHMEM is pointed at its
230230
libfabric transport, and backend setup fails unless the plugin library is present in the container.
231-
NCCL GIN then rides the plugin's own GIN implementation (`OFI_NCCL_GIN_TYPE` selects proxy or GDAKI),
232-
which requires GDRCopy 2.5+ (`gdrdrv` loaded, `libgdrapi` present) on the node: without it the plugin's
233-
GIN init fails, NCCL disables the Device API for the communicator, and `ncclEpCreateGroup` returns
234-
`ncclInvalidUsage` (verified on b300-dsxe 2026-09-11; that pool ships no gdrcopy, so its EP16 legs are
235-
withheld from the registry until it does). DeepEP V2 low-latency is also withheld there: its legacy
236-
Buffer initialises NVSHMEM with an RDMA transport even single-node, and on that pool NVSHMEM aborts
231+
NCCL GIN then rides the plugin's GIN through its CPU proxy (`NCCL_GIN_TYPE=2`), which requires
232+
GDRCopy 2.5+ (`gdrdrv` loaded, `libgdrapi` present) on the node: without it the plugin's GIN init fails,
233+
NCCL disables the Device API for the communicator, and `ncclEpCreateGroup` returns `ncclInvalidUsage`.
234+
The type is pinned because NCCL otherwise prefers its built-in GDAKI on the nodes' two ConnectX-7 IB
235+
ports, which are not the GPU fabric, and most ranks then fail window registration while the rest hang.
236+
Multi-node shards build aws-ofi-nccl 1.21.1 into the backend cache and load it in place of the hosts'
237+
1.20, whose proxy applies every signal with an inline GPU read-modify-write (1.3-4.5x on b300 EP16).
238+
Proxy GIN still bounds these rows well below the IB pools: GPU-initiated GIN on EFA needs EFA
239+
installer 1.50.0 on the hosts (efa.ko 3.3.0 completion counters, libfabric 2.6) and NCCL 2.31.
240+
DeepEP sizes its SMs from an `ibstat` rate probe that cannot see EFA, so `network.rdma_gbs` supplies
241+
the per-GPU scale-out rate instead. DeepEP V2 low-latency stays withheld on both: its legacy
242+
Buffer initialises NVSHMEM with an RDMA transport even single-node, and on b300 NVSHMEM aborts
237243
with "nvshmem detect topo failed" (status 28) with and without `NVSHMEM_REMOTE_TRANSPORT=none`
238244
(runs 34521825749, 34523594787); the HT ElasticBuffer path, which needs no NVSHMEM, passes.
239245

‎collectivex/bench/ep_deepep_v2.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,8 +213,12 @@ def create_buffer(self, spec):
213213
),
214214
)
215215
tuning_num_experts = int(args.experts)
216+
# DeepEP sizes SMs from the scale-out rate, which it reads from ibstat; on fabrics ibstat
217+
# cannot see (EFA) that probe returns 0 and the model divides by it, so the registry's
218+
# per-GPU rate stands in. Unset (0) keeps DeepEP's own probe.
219+
rdma_gbs = float(os.environ.get("COLLX_RDMA_GBS") or 0)
216220
self.num_sms = int(
217-
self.buffer.get_theoretical_num_sms(tuning_num_experts, args.topk)
221+
self.buffer.get_theoretical_num_sms(tuning_num_experts, args.topk, rdma_gbs=rdma_gbs)
218222
)
219223
self.num_qps = int(self.buffer.get_theoretical_num_qps(self.num_sms))
220224
realized = {

‎collectivex/bench/ep_harness.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,9 +196,10 @@ def _write_json_atomic(path: str, value) -> None:
196196

197197

198198
def kernel_generation(backend) -> str:
199-
"""The adapter's kernel family; `-cudagraph` keeps replayed rows a separate series."""
199+
"""Identify the kernel family and captured-all-reduce measurement regime."""
200200
family = backend.kernel_generation or "n-a"
201-
return f"{family}-cudagraph" if backend.cuda_graph_enabled else family
201+
# Historical -cudagraph rows used an external barrier and alignment spin.
202+
return f"{family}-cudagraph-in-graph-ar" if backend.cuda_graph_enabled else family
202203

203204

204205
class _Collectives:

0 commit comments

Comments
 (0)