[lvyufeng]

AI Agent Information

• Agent/Tool: Claude Code CLI (Opus 5, 1M context)
• Model: Claude Opus 5
• Human Reviewer: @lvyufeng
• Session Summary: A DCU issue sweep came to the codegen/conf boundary: FLAGOS_CODEGEN_ALL=1 python3 scripts/codegen/codegen_ops.py, the regeneration command the project's own docs prescribe, ended with two extra writers that rebuilt backends_metax.conf and backends_dcu.conf from a fallback set that had drifted away from the confs. Issue [AI] [BUG][DCU] codegen_ops.py still rewrites backends_dcu.conf and silently reverts the DCU routing policy #459 recorded the measurement; this PR deletes both writers, leaves gen_vendor_confs.py as the sole owner of the vendor confs, relocates the diagnosis they carried into BOXING_GAP_NOTES["metax"], and adds a source-level guard test that fails against the pre-fix file.

Summary

scripts/codegen/codegen_ops.py no longer writes torch_fl/configs/backends_metax.conf or torch_fl/configs/backends_dcu.conf. Both were written at the end of every all_cuda run from op_info plus a local fallback set in the pre-#272 cuda/flagos_python vocabulary, and both overwrote their conf unconditionally — so the documented one-command regeneration silently reverted the routing those platforms record and dropped three ops the generator's op universe carries on top of op_info. scripts/codegen/gen_vendor_confs.py, which CI already enforces with --check, becomes the only writer of every vendor conf; a run whose FlagGems coverage moved now prints the follow-up command instead of running it. No route changed: all six vendor confs keep their route lines and the two regenerated files differ by comments alone. tests/unit/test_vendor_conf_single_writer.py pins the invariant at source level, with no build, no torch import and no hardware.

Change Type

• Bug Fix
• New Feature
• Performance Optimization
• Refactoring
• Documentation
• Testing
• CI/Infrastructure
• Breaking Change

Platforms Affected

• CUDA
• MetaX
• Ascend
• PPU
• Platform-agnostic (all platforms)

The change is in the generators and in the conf ownership rule, not in any route: MetaX and DCU's confs are regenerated with added comments and byte-identical route lines, and PPU, CUDA, MUSA, Ascend and GCU's confs are untouched.

Problem Analysis

What was broken/missing?

FLAGOS_CODEGEN_ALL=1 python3 scripts/codegen/codegen_ops.py is the full regeneration command that scripts/README.md and docs/development/testing.md both prescribe. It wrote torch_fl/configs/backends_cuda.conf (correct: that file is the CUDA boxing table for the wrappers that run generated), and it also wrote backends_metax.conf and backends_dcu.conf, from:

• op_info — this run's op set, which is not the op universe gen_vendor_confs.py routes over. The generator widens that universe from register.inc with vendor-only ops read out of a conf (matmul, matmul_backward) plus EXTRA_ROUTED (scaled_dot_product_attention). Those three ops are in the committed confs and were silently deleted by the codegen run.
• flaggems_forced_cuda plus a per-platform literal (metax_triton_fallback, dcu_triton_fallback) spelled in the pre-ci: align backend routing with platform FlagGems availability #272 cuda | flagos_python vocabulary, at a time when the confs had already moved on. The MetaX literal in particular named 24 ops of which mm, mm.out, bmm, bmm.out, mean.dim, add.Tensor, div.Scalar, div.Tensor, _conj, sort, sort.stable, relu, relu_, sum.dim_IntList, embedding_dense_backward, i0, i0.out, reflection_pad2d, soft_margin_loss, special_i0e, special_i1, im2col, zero and mul_.Tensor were already cuda in the committed conf, and 9 of which (conv_transpose2d, scaled_mm, as_strided_copy, prelu, lift_fresh_copy, resolve_conj, t_copy, …) are absent from the conf altogether.

The route writer's answer for "not a FlagGems gap" was flagos_python, which is not what the conf says today: the confs are FlagGems-first and pin the exceptions to cuda.

Why did it happen?

The generator reached the confs first and the confs then grew a second, authoritative writer (gen_vendor_confs.py, enforced by .github/workflows/codegen-checks.yml:62-63). Nobody deleted the first one, so from then on two writers disagreed about the same artifact, and the one that ran on the CUDA path won. The drift was invisible to CI because the damage was never committed — a maintainer running the documented command would see a large diff in the confs and would have reason to read it as expected churn.

Investigation process:

1. Read scripts/codegen/codegen_ops.py end to end around the conf writers (the all_cuda block) and scripts/codegen/gen_vendor_confs.py (BOXING_PLATFORMS, VENDORS, route_boxing(), BOXING_GAP_NOTES).
2. Traced how the MetaX pins actually survive today: not from the literal, but from BOXING_PLATFORMS's diff branch — the generator reads backends_metax.conf back on every run, so an op FlagGems covers whose conf line reads cuda is the gap set. route_boxing() checks that set before the C++ route, which is why only 12 of METAX_CPP_MEASURED's 17 ops reach flaggems_cpp and bmm, bmm.out, sort, sort.stable and sum.dim_IntList stay boxed.
3. Extracted the pristine 514c58d tree (git archive HEAD | tar -x) and ran python3 scripts/codegen/gen_vendor_confs.py --check in it: clean, exit 0. The round trip was sound before the fix, which is why the clobber only ever appeared as an uncommitted diff.
4. Implemented the deletion, then measured the alternative the issue offers (calling gen_vendor_confs.py from codegen_ops.py) before choosing the printed hint — see "Key design decisions".
5. Regenerated, then diffed the results against the pristine tree at route-line granularity and at route-set-SHA granularity.
6. Falsified the new guards against git show HEAD:scripts/codegen/codegen_ops.py before keeping them.

Solution Design

Implementation approach:

• Delete both writers and their 165 + 28 lines of justification comments from codegen_ops.py; the module stops naming any conf but backends_cuda.conf.
• Keep the write above them — render_flaggems_coverage(...) producing scripts/codegen/backend_coverage.py — because that file is the input to gen_vendor_confs.py and codegen_ops.py is the only place the FlagGems Python surface is discovered.
• Print the follow-up command, so a run whose coverage moved still tells the caller what to run.
• Move the per-op diagnosis the MetaX literal carried into BOXING_GAP_NOTES["metax"] in gen_vendor_confs.py, beside the DCU entry that already lives there and next to the routing it explains. The gap set was never restated — it is read back out of the conf — so only the prose moved.
• Fix the two files' prose that pointed at the deleted literals (gen_vendor_confs.py's METAX_CPP_MEASURED comment, codegen_ops.py's _conj comment, tests/integration/ops/test_sum_dispatch.py, tests/integration/ops/test_metax_flaggems.py ×3, docs/reference/operator-support.md ×3 including the historical slice.Tensor row), and correct .claude/skills/pre-pr-checks/SKILL.md, whose conflict procedure told the reader to regenerate torch_fl/configs/backends_*.conf with codegen_ops.py — which is the bug this PR removes.

Key design decisions:

The follow-up is printed, not run — a deliberate deviation from the issue's first suggestion. Issue #459's Proposed Solution item 1 says "have codegen_ops.py call gen_vendor_confs.py at the end of an all_cuda run", and item 2 offers the printed command as the alternative "if invoking one generator from another is judged too coupled". I implemented item 2 and measured why item 1 is wrong before doing so:

codegen_ops.py discovers the FlagGems coverage from the installed flag_gems, so the ceiling is environment-dependent. In this container (flag_gems 5.3.4.post1.dev1) it discovers 533 of the 639 ops committed in backend_coverage.py. A delegated variant of this fix therefore regenerated six vendor confs from that ceiling, not two: backends_metax.conf moved 95 routes (flaggems → cuda) and backends_ppu.conf was rewritten as well — neither platform is on the CUDA path this script owns, and the diff a reviewer would read is CUDA-binding churn. Every op it moved is a member of both coverage sets, so the movement is not an artifact of the version gap. Stating the follow-up keeps one property that is the whole point of #459 — a full regeneration performs no routing change — while a run that really did move the ceiling shows it right above the hint and in the printed command.

Jumping to cuda first would invert the generator's decision. The deleted writers answered "not a FlagGems gap" with flagos_python; the shipped confs answer it with cuda, and which one is right is per-platform and recorded in the gap sets. Deleting the writers is what makes that decision single-sourced.

The guard is source-level on purpose. The regression is cheap to reintroduce (the shortcut "just write the conf here too" is always available) and expensive to notice (generated-file churn). tests/unit/test_vendor_conf_single_writer.py uses ast on codegen_ops.py and imports gen_vendor_confs.py by path, so it costs no build, no torch import and no hardware, and it fails at the line that would do the writing.

Code changes by file:

• scripts/codegen/codegen_ops.py (2629-2633, 3085-3122): removed the backends_metax.conf and backends_dcu.conf writers and their comment blocks (279 lines net); replaced them with the printed catch-up hint; removed the now-unused import subprocess; repointed the _conj comment at BOXING_GAP_NOTES.
• scripts/codegen/gen_vendor_confs.py (169-180, 322-395, 1584-1597): rewrote the BOXING_PLATFORMS comment to describe the diff round trip and the single-writer precondition it depends on; added the BOXING_GAP_NOTES["metax"] entry carrying the diagnosis relocated out of metax_triton_fallback; repointed the METAX_CPP_MEASURED comment at the gap set and the notes.
• tests/unit/test_vendor_conf_single_writer.py (new, 253 lines): six tests — the conf-path scan, cuda's absence from build_all(), the legacy-set scan, the no-shell-out scan, the printed-command scan, and the render_flaggems_coverage pin.
• torch_fl/configs/backends_dcu.conf (+17 lines), torch_fl/configs/backends_metax.conf (+52 lines): regenerated; comments only, zero route-line changes.
• tests/integration/ops/test_sum_dispatch.py (154), tests/integration/ops/test_metax_flaggems.py (42, 78, 262): pointed the MetaX sum.dim_IntList and slice.Tensor explanations at BOXING_GAP_NOTES["metax"] instead of the deleted literal. Comments and docstrings only.
• docs/reference/operator-support.md: repointed three prose references, and added a 2026-09-28 Update History row recording the comment-only regeneration, the before/after conf and route-set SHAs, and the fact that no survey was rerun because no route moved.
• .claude/skills/pre-pr-checks/SKILL.md (56-84): split the "regenerate rather than merge" procedure by artifact owner — csrc/aten/generated/* is codegen_ops.py's, torch_fl/configs/backends_*.conf is gen_vendor_confs.py's — and fixed the idempotency check, which compared the worktree against a freshly checked-out index and so reported "NOT idempotent" for any correct run.

Changes by commit:

1. d18914b - fix: make gen_vendor_confs.py the sole writer of the vendor confs: the whole change, as described above.

Verification

Pre-submission Checklist

• Linting passed (ruff check, ruff format --check)
• Type checking passed (if applicable — no type checker in this repository)
• All tests pass (unit + integration where the platform is available; see the evidence gaps)
• Manual testing completed (a full post-fix regeneration, and the guards falsified against the pre-fix file)
• No debug/temporary code (no print statements added beyond the intentional user-facing hint, no TODOs)
• Documentation updated (operator support report, pre-PR skill, and the stale references listed above)
• Commit messages follow conventions (type: description format)
• All text in English

Linting Results

$ ruff check
All checks passed!

$ ruff format --check
334 files already formatted

Test Results

# Command:
pytest tests/unit/test_gen_vendor_confs.py tests/unit/test_conf_registration_consistency.py tests/unit/test_vendor_conf_single_writer.py -q

# Output (including test count):
79 passed, 11 warnings in 0.90s

The warning is the pre-existing unregistered anyplatform mark, not a new one.

The three new guards, run against the pre-fix codegen_ops.py (git show HEAD:... swapped in for the fix):

# Command:
pytest tests/unit/test_vendor_conf_single_writer.py -q    # with HEAD's codegen_ops.py in place

# Output:
FAILED tests/unit/test_vendor_conf_single_writer.py::test_codegen_ops_names_no_generated_vendor_conf
FAILED tests/unit/test_vendor_conf_single_writer.py::test_no_local_vendor_fallback_table_is_left_behind
FAILED tests/unit/test_vendor_conf_single_writer.py::test_codegen_ops_points_at_the_generator
3 failed, 3 passed in 0.26s

with, verbatim:

E  AssertionError: codegen_ops.py names ['backends_dcu.conf', 'backends_metax.conf']; every
   torch_fl/configs/backends_<platform>.conf except backends_cuda.conf is written by
   scripts/codegen/gen_vendor_confs.py
E  AssertionError: ['dcu_triton_fallback', 'metax_triton_fallback'] reappeared in codegen_ops.py;
   per-platform FlagGems fallbacks are stated in gen_vendor_confs.py, which renders the confs
E  AssertionError: codegen_ops.py no longer prints a runnable command for refreshing the vendor
   confs after a run whose FlagGems coverage moved

The wider unit suite cannot run in this container at the moment, for a reason that is not this change. import torch_fl raises

torch_fl/__init__.py:2118: in <module>
    _phase_claim()
torch_fl/__init__.py:2007: in _phase_claim
    torch_fl._C._set_backend_config_path(_BACKEND_CONFIG_PATH)
E   AttributeError: module 'torch_fl._C' has no attribute '_set_backend_config_path'

because every torch_fl/_C*.so on this box predates that pybind setter (strings -a torch_fl/_C.cpython-310-x86_64-linux-gnu.so | grep -c _set_backend_config_path → 0, for the copy in this tree, in the main checkout and in every sibling worktree). Every test module that imports the package therefore errors at collection. This is shown to be independent of the change by running the same five files in the pristine 514c58d tree and in this branch, back to back; the output is identical, including the collection errors:

########## BASELINE 514c58d ##########
ERROR tests/unit/test_flaggems_pointwise_dispatch.py - AttributeError: module...
ERROR tests/unit/test_musa_rng_bridge.py - AttributeError: partially initiali...
2 errors in 2.53s
########## WORKTREE d18914b (this branch) ##########
ERROR tests/unit/test_flaggems_pointwise_dispatch.py - AttributeError: module...
ERROR tests/unit/test_musa_rng_bridge.py - AttributeError: partially initiali...
2 errors in 2.53s

The five files (test_flaggems_pointwise_dispatch.py, test_musa_rng_bridge.py, test_startup_subprocess.py, test_transformers_hf_tests.py, test_vendor_libtorch_overlay.py) were also grepped for configs/, codegen and backends_ — zero matches — and the change adds no code to any import path they take. An earlier run of the same five files in this container, from before the extension drift, collected 78 tests and failed 11 of them, all of them in the same environment-dependent categories. None of that is claimed as green here: what is claimed is that the three suites above (79 passed) exercise every file this PR changes, and that the failures outside them reproduce on the base commit.

Manual Verification

The generator round trip is idempotent and clean, before and after:

# In the pristine 514c58d tree, and in this branch:
$ python3 scripts/codegen/gen_vendor_confs.py --check
note: 2 vendor-only op(s) added to the op list: matmul, matmul_backward
all vendor confs up to date
# exit 0, and 0 again on a second run

A full post-fix regeneration leaves the vendor confs bit-identical. Run in a scratch copy of 514c58d with this branch's codegen_ops.py in place:

$ FLAGOS_CODEGEN_ALL=1 python3 scripts/codegen/codegen_ops.py
...
   regenerated backends_cuda.conf with 0 flaggems and 2034 cuda routes ...
   updated backend_coverage.py: FLAGGEMS_PYTHON_OPS = 0 ops (0 override-only excluded)
   vendor confs unchanged; if the coverage above moved, refresh
   /<tree>/torch_fl/configs with:
     python3 scripts/codegen/gen_vendor_confs.py

Done. Files in: /<tree>/csrc/aten/generated
exit=0

# The vendor confs after that run, byte for byte identical to before it:
$ sha256sum torch_fl/configs/backends_dcu.conf torch_fl/configs/backends_metax.conf
1d0c00e852a8923528ba6577010777969ee9cfff27e1d4995120556142da6da2  torch_fl/configs/backends_dcu.conf
bb1dc5c4550339dcd44ac438b2981e703c882025e477221e86cac37d833f58f2  torch_fl/configs/backends_metax.conf

Before the fix those two SHAs changed on every such run; #459's measurement of one of them is the 2037 → 2034 op regression, 531 moved routes and 121 deleted # Note: lines, after which gen_vendor_confs.py --check failed on three confs.

No route moved anywhere. Route lines and route-set SHA-256 (sha256 of the active-route list) before → after:

Conf	Routes	Route-set SHA-256	File SHA-256
backends_dcu.conf	454 → 454	2c51253253a328f9… unchanged	1d0c00e8… → 0dd915f8b1377125ae0b2700c4a444489bed80f4156ef9907fa7c80fd9da6198 (17 # Note: lines added)
backends_metax.conf	592 → 592	b07ca8a02a76986b… unchanged	bb1dc5c4… → 8af0f654bb27358c93444b5a1674d447d566f6d3c535d591ee80714da7e95a22 (52 comment lines added)
backends_ppu.conf	591 → 591	c39b39635122b644… unchanged	byte-identical
backends_cuda.conf	416 → 416	aacb66bca5dfc08d… unchanged	byte-identical
backends_musa.conf, backends_ascend.conf, backends_gcu.conf	—	—	byte-identical

Code Quality Verification

Style Consistency

• Matched existing code style in modified files
• Followed naming conventions (checked similar code)
• Comment density matches surrounding code — the deleted blocks were long-form diagnosis, and the replacement is the same kind of prose relocated next to the routing it explains
• Used project's existing utilities/helpers (no reinventing)

Edge Cases Considered

1. A run whose FlagGems coverage moved. The hint prints unconditionally, so the caller is told the confs may need a refresh even when the move is invisible in this run's own output.
2. backends_cuda.conf. It is deliberately kept: it is not a vendor conf, it is the CUDA boxing table of the wrappers this run generated, and it is not an input to gen_vendor_confs.py (which enumerates from csrc/aten/generated/register.inc). The guard pins that cuda is not in build_all()'s platform set, so the exemption cannot silently widen.
3. A future reader grepping for metax_triton_fallback. Six references pointed at the deleted literal; all six now point at the conf and BOXING_GAP_NOTES["metax"], including the operator-support row from 2026-09-16, which keeps its original wording and carries a pointer.
4. import gen_vendor_confs. Rejected and pinned: the module parses sys.argv at import time, and codegen_ops.py is imported by the FlagGems register generators, so importing one from the other would consume the caller's arguments. This is why the guards assert the absence of an import as well as of a subprocess call.
5. Deliberate divergence between the two trees. MetaX's slice_backward and DCU's slice_backward are pinned for unrelated causes and the relocated notes say so, so the shared name does not read as a shared reason.

Potential Risks

1. Someone runs the vendor-conf regeneration and commits churn. That is the pre-existing workflow, unchanged here; the difference is that the CUDA path can no longer produce it silently.
2. The relocated MetaX prose goes stale the way the literal did. It is now adjacent to the routing it describes, which is the mitigations the file already uses for DCU; it is still prose and nothing enforces it.

Rollback Plan

Revert d18914b. The change is confined to two generators, two conf files (comments only) and tests, and it moves no runtime route, so a revert restores the previous bytes for every conf.

Related Work

• Fixes [AI] [BUG][DCU] codegen_ops.py still rewrites backends_dcu.conf and silently reverts the DCU routing policy #459
• Related to [AI][BUG][DCU] Track FlagGems native_batch_norm input mutation with absent running statistics #295 and fix: pin DCU's native_batch_norm to CUDA boxing #458 (the DCU native_batch_norm pin the deleted writer used to revert)

Explicitly Not Included

• Rerunning tests/manual/flaggems_overload_survey.py. No route moved, so there is nothing to re-measure; the report's Update History row records the conf SHAs and says so explicitly rather than implying a survey was taken.
• Any change to backend_coverage.py or to the FlagGems coverage discovery in codegen_ops.py. The 533-of-639 gap in this container is an environment fact, not a defect this PR fixes, and the coverage file is untouched by the commit.
• A CI job that runs codegen_ops.py and asserts the confs are unchanged. That is what the unit guards do at source level, without a build; a build-and-run CI job would be a larger change and belongs to the maintainer's call.
• Fixing the three sum.out-style follow-ups [AI] [BUG][DCU] codegen_ops.py still rewrites backends_dcu.conf and silently reverts the DCU routing policy #459 mentions in passing, if any are real: out of scope here.

Human Review Notes

Areas needing special attention:

1. The deviation from the issue's first suggestion. [AI] [BUG][DCU] codegen_ops.py still rewrites backends_dcu.conf and silently reverts the DCU routing policy #459 proposes delegating to gen_vendor_confs.py first and the printed command second; this PR takes the second. The measurement behind that is in "Key design decisions" and its blast radius (six confs, 95 MetaX routes, a PPU rewrite) is reproducible by adding the subprocess.run call back and diffing the tree.
2. The relocation of the MetaX diagnosis. BOXING_GAP_NOTES["metax"] is prose carried across from the deleted literal; the assertions it makes about mm/bmm/mean.dim, the scalar group, _conj, sort and the device-guard class come from the deleted comment verbatim, while the survey verdicts it points at live in tests/integration/ops/test_metax_flaggems.py and docs/reference/operator-support.md. If anything there reads as new, it is a copy error and worth flagging.
3. The added Update History row. It documents comment-only drift in two generated files; if the project prefers conf SHA churn not to be recorded, that row should be dropped rather than reworded.

Questions for reviewer:

1. Is the printed hint the right resting place, or would you rather codegen_ops.py gain a --refresh-vendor-confs flag that runs the generator explicitly (still not by default)?
2. Should the single-writer rule also be asserted in CI against a real FLAGOS_CODEGEN_ALL=1 run, given the cost?

Additional Context

Evidence gaps: no accelerator was used for this change. The confs were regenerated on a host with no DCU, MetaX, PPU, Ascend, GCU or MUSA device, so no platform is revalidated by it, and no row of theirs moves in docs/reference/operator-support.md. The coverage ceiling codegen_ops.py discovers is environment-dependent (533 of 639 committed ops here), which is the reason the follow-up is printed rather than acted on, and the reason a reviewer should not read the printed number in any log excerpt as the repository's coverage.

---

🤖 Generated with Claude Code