Skip to content

Commit 41f1721

Browse files
lvyufengclaude
andauthored
docs: mark the two stale issue references in the coverage reports (#431)
Both reference documents carry claims that stopped being true when the issues behind them closed, and neither had a record of what replaced them. `docs/reference/hf-coverage.md` lists five tracked causes from the 2026-09-02 MUSA baseline and still presents all five as open work. Four are closed: #250/#265 by PR #280 and #282, #262/#268 by PR #398, #266 by PR #278, #264 by PR #419. Only #263 is still open. A new "What has changed since this baseline" section records each closure with the issue and the commit that landed, and states what was actually re-measured: the eight SDPA nodeids #268 borrowed, with `{"PASS": 8}` under the harness's device shims and an `allclose` residual without them that belongs with #248. The baseline's own `Affected tests` counts are deliberately left alone — they are dated measurements of `64e60dd`, and rewriting them against a later tree would stop them matching the run they came from, which is what the dedup parser reads them for. `docs/reference/operator-support.md` records the 2026-08-30 cohort as excluding "the known float64 `mm` gap". PR #275 moved `mm` and `mm.out` onto the FlagGems route on 2026-09-15, so that exclusion is historical; the later entries in the report count the op. A paragraph after the row says so and names what replaced the mudnn capability gap — a different defect on the route that now serves the op, the FlagGems fp64 tile exceeding the device's 192 KiB of shared memory past `M > 32` and `N > 32`, filed as #428. Documentation only; no code, no routing and no test changes. Verified: `ruff check .` -> All checks passed; `ruff format --check .` -> 311 files already formatted; `pytest tests/unit/test_transformers_automation.py -q` -> 61 passed; `pytest tests/unit/ -q` -> 6 failed, 761 passed, 131 skipped, and the same six fail identically with these edits stashed, so they pre-exist on d5f82ae. `transformers_deduplicate.extract_baseline_fingerprints` still reads the edited coverage doc without error. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d5f82ae commit 41f1721

2 files changed

Lines changed: 57 additions & 1 deletion

File tree

‎docs/reference/hf-coverage.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,12 @@ The corrected verification grouped the 20 occurrences into these tracked causes:
5151
| | `FEATURE_UNSUPPORTED` | TorchInductor/Triton requires CUDA libraries | 2 | [#264](https://github.com/flagos-ai/Torch-FL/issues/264) |
5252
| | `OP_UNSUPPORTED` | mudnn `TRUEDIV` with `INT64` | 2 | [#266](https://github.com/flagos-ai/Torch-FL/issues/266) |
5353

54+
`Affected tests` is the count this baseline measured on `64e60dd` and is not
55+
updated here: the table's job is to say which occurrence a re-measurement is
56+
looking at, so a count rewritten against a later tree would stop matching the run
57+
it came from. Which rows are still live work is recorded in prose below, where a
58+
closure can carry the commit that closed it.
59+
5460
The `Fingerprint` column is what deduplication keys on: `transformers_deduplicate.py`
5561
reads it from each baseline's cause table, and `transformers_file_issues.py`
5662
fills it in when it records a newly filed issue. The five rows above are left
@@ -75,6 +81,46 @@ suite measurement, but the corrected per-test isolation is the evidence used for
7581
root-cause filing. Raw JSON remains outside the repository at
7682
`/tmp/qwen3.json` and `/tmp/qwen3_isolated_results_v2.json`.
7783

84+
### What has changed since this baseline
85+
86+
Four of the five causes have a fix; the fifth is the only reason this baseline is
87+
not fully resolved.
88+
89+
- **#250, #265** (device context poisoning, and its `test_model_parallelism`
90+
trigger) — closed 2026-09-15, the same day PR #280 (`6b2af4b`, "stage
91+
cross-device copies in MudnnCopy instead of handing mudnn two devices") landed,
92+
with PR #282 ("order the peer device against a cross-device D2D copy") covering
93+
the ordering half.
94+
- **#262, #268** (mudnn softmax rejects non-contiguous input) — PR #398
95+
(`6a9c101`, 2026-09-23, "materialize strided softmax operands before mudnn on
96+
MUSA"), which closed #262 the same day. #268 was closed three days later, on
97+
2026-09-26, against that fix after its borrowed nodeids were re-measured. The eight
98+
SDPA nodeids this row borrowed were re-measured on `main @ 48829b8` with the
99+
harness's device shims enabled and report `{"PASS": 8}`; the
100+
`RuntimeError: _softmax failed: INVALID_PARAMETER` signature the run recorded is
101+
raised by none of them. With the shims disabled
102+
(`HF_TEST_NO_DEVICE_SHIMS=1`) the same eight fail, but on the 80 % `allclose`
103+
check and not in the operator: `tests/test_modeling_common.py:486` falls through
104+
to fp32's `atol = 1e-7, rtol = 1e-4` for a device name transformers does not
105+
list, which is the fall-through #248 documents for GCU. That residual belongs
106+
with #248. Only those eight were re-run.
107+
- **#266** (mudnn `TRUEDIV` with `INT64`) — PR #278 (`251c9c8`, 2026-09-15, "route
108+
MUSA integer division through mudnn and promote true division").
109+
- **#264** (TorchInductor/Triton requires CUDA libraries) — PR #419 (`48829b8`,
110+
"ask torch, not the redirected probe, before aliasing cuda to flagos").
111+
- **#263** (ProcessGroupGloo rejects `flagos` tensors) — still open. Its six
112+
FSDP2 nodeids are the remaining known failures of this baseline, and the failure
113+
reproduces today: `RuntimeError: ProcessGroupGloo::allreduce: unsupported device
114+
type flagos`.
115+
116+
Because `Affected tests` and the baseline header above describe `64e60dd`, the
117+
whole table is a dated snapshot and not a statement about the current tree.
118+
Re-running the baseline is what refreshes it — and is also what fills in the
119+
`Fingerprint` column, per the note above. The closures above are recorded from the
120+
issues and the commits that landed; only the #268 re-measurement re-ran any of
121+
this baseline's tests, and it covers eight nodeids out of twenty, so the other
122+
four causes are **not revalidated** here.
123+
78124
### Failed test inventory
79125

80126
The 20 failed tests were grouped for follow-up investigation:
@@ -110,3 +156,11 @@ SDPA-named tests reproduced the same non-contiguous softmax failure rather than
110156
pure tolerance mismatch. Future measurements must also run with
111157
`FLAGOS_LOG=fallback` and report any passing operation that used CPU fallback
112158
as an accelerator coverage gap.
159+
160+
That last clause is worth revisiting now that #398 has landed. With the strided
161+
operands materialized before mudnn, a tolerance mismatch is exactly what those
162+
eight show: they pass under the harness's device shims and fail without them, at
163+
the `allclose` check rather than in the operator (the re-measurement above). The
164+
baseline's reading was right for the tree it measured — and the behaviour it ruled
165+
out is what remains once the operator is fixed, which is why the residual is
166+
tracked by #248 and not as a softmax defect.

‎docs/reference/operator-support.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5257,7 +5257,9 @@ Measured on the eight-device Moore Threads MTT S5000 host with CPU PyTorch 2.10.
52575257
- `tests/integration/ops/test_pow_dispatch.py -m anyplatform`: **22 passed, 4 deselected**; coverage includes both `pow.Tensor_Scalar` and `pow.Tensor_Tensor` empty outputs, empty broadcasting, the zero-length narrow backward path, and non-empty parity.
52585258
- `tests/integration/ops/test_narrow_dispatch.py -m anyplatform`: **17 passed**, including `test_narrow_backward_edge_cases[1-2-0]`, which is no longer deselected in the MUSA CI manifest.
52595259
- `tests/integration/ops/test_musa_dispatch.py -m musa`: **89 passed**.
5260-
- The CI operator cohort (excluding the separately triaged RNG file and the known float64 `mm` gap) reached **480 passed, 14 skipped, and 3 xpassed**; three existing FlagGems configuration-consistency assertions failed because they inspect unrelated generic FlagGems routes, not MUSA native kernels.
5260+
- The CI operator cohort (excluding the separately triaged RNG file and the float64 `mm` gap as it stood on this date) reached **480 passed, 14 skipped, and 3 xpassed**; three existing FlagGems configuration-consistency assertions failed because they inspect unrelated generic FlagGems routes, not MUSA native kernels.
5261+
5262+
The `mm` exclusion is historical and no longer describes the cohort: PR #275 (`6b978c0`, 2026-09-15) moved `mm` and `mm.out` onto the FlagGems route, so every entry dated from 2026-09-15 onward counts the op. The figure above is therefore a measurement of the 2026-08-30 tree and not a current one; the later entries in this report record the cohort without that exclusion, and their manifests differ from this one, so they are not a correction of the number — they are separate measurements. What replaced the mudnn capability gap is a *different* defect on the route that now serves the op: the FlagGems float64 tile exceeds the device's 192 KiB of shared memory once both `M > 32` and `N > 32`, filed as #428. That does not restore the exclusion this row records, and it is not read from the routing table — the boundary is measured.
52615263

52625264
The empty-output path stays on `flagos:0` and preserves shape and dtype; no host round trip is used. The MUSA operator support cohort is otherwise unchanged.
52635265

0 commit comments

Comments
 (0)