Repository navigation
fix(signoff): accept removed points under the head frontier in Check 18(g) / fix(signoff):Check 18(g) 接受位于 head 前沿线下方的被移除点 - #4589
Conversation
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
| for low, high in zip(frontier, frontier[1:]): | ||
| if low[x] < point[x] < high[x]: | ||
| line = low[y] + (high[y] - low[y]) * (point[x] - low[x]) / (high[x] - low[x]) | ||
| return f"between {low['label']} and {high['label']}" if line >= point[y] else None | ||
| return None | ||
|
|
||
|
|
||
| def dominance(before: list[dict], after: list[dict]) -> dict[str, Any]: | ||
| """Removed points must be dominated by an after point; shared points report deltas.""" | ||
| """Removed points must lie under the after frontier; shared points report deltas.""" | ||
| for point in (*before, *after): | ||
| if not isinstance(point.get("label"), str) or not all( | ||
| isinstance(point.get(metric), int | float) |
There was a problem hiding this comment.
🔴 The CI lint job fails on this PR, because two new lines break the Ruff format and lint checks it runs. Line 465 is 103 characters, over the line-length = 100 in infx/ruff.toml, so ruff format --check wants it rewrapped. Line 304, zip(frontier, frontier[1:]), has no strict= and is a pairwise idiom, which select = ["ALL"] flags (B905 / RUF007). Fix: get both ruff check and ruff format --check clean on infx. Use itertools.pairwise(frontier) at line 304, as results/power/common.py does, and let ruff format split the covers comprehension at line 465.
Why this was flagged
The CI job at .github/workflows/ci.yml:59-60 runs ruff check and ruff format --check --diff over inferencex-e2e/infx. That tree is configured by infx/ruff.toml: line-length = 100, select = ["ALL"], and E501 is ignored, so only the formatter enforces the length. signoff_facts.py:465 (other["measured"] | {"label": other["label"]} for other in head_points if other["measured"]) is 103 characters. The formatter would split that comprehension over several lines, so the format check reports a diff and fails. signoff_facts.py:304 calls zip(frontier, frontier[1:]) with no strict= argument. With target py312 this trips B905, and RUF007 also flags it as zip-instead-of-pairwise. Every other zip( under infx passes strict=, and the successive-pair loops use itertools.pairwise. On the base branch the lint job passes for this file. After this change it fails until both lines are fixed.
Verification: Trigger: the CI Lint leg (.github/workflows/ci.yml:59-60) runs ruff check and ruff format --check --diff over inferencex-e2e/infx. signoff_facts.py:304 for low, high in zip(frontier, frontier[1:]): is the zip(x, x[1:]) pattern that RUF007 (zip-instead-of-pairwise) flags. Result: the required Lint check goes red on this PR, and it would fail on main if merged.
| `check_18g_points` lists `undominated_removed` (FAIL, unless head-point measurements | ||
| the facts missed cover them) and `unmeasured_removed`. For unmeasured | ||
| points, take measurements from the most recent sweep run that measured them | ||
| (`gh run download`) and run `signoff_facts dominance`. Points with the same shape and concurrency but a new |
There was a problem hiding this comment.
🟡 (optional) Verifiers re-checking removed points with signoff_facts dominance can now get a false PASS by passing only some head points as after. Under the old rule a partial list could only fail too often. _cover builds its segment from whatever after it is given. If a lower non-dominated head point between the bracketing pair is left out, the line rises and a removed point above the real head curve reports "between A and B". Fix: require after to hold every measured head point of the config key, i.e. the facts' head measurements plus any the facts missed. Also have check_18g_points output the head points it used so the verifier can copy them.
Why this was flagged
Check 18(g) at .github/codeowner-signoff-verify-prompt.md:934-937 tells the verifier to run signoff_facts dominance for unmeasured points, and to look at head measurements "the facts missed". It never says which head points go into after. points_diff (signoff_facts.py:442-517) does not report the full head set it fed into _cover, only added and removed. So the verifier rebuilds after from a run download, as the PR's own replays did for #4281/#4357 ("the head points of the validating run"). That leaves out unchanged head points measured only on the ingested curve. In _cover (signoff_facts.py:300-307), dropping a non-dominated head point C that sits below the A-B chord removes the A-C-B kink. The line at the removed point's mean_intvty then rises and the point is reported as covered, even though the full head frontier fails it. On the base branch dominated_by needed one dominating point, so a partial after could only cause a false FAIL, never a false PASS.
Verification: Trigger: a verifier runs signoff_facts dominance with an after list that leaves out a non-dominated head point. In signoff_facts.py, _cover builds frontier only from the others it is passed. The old rule was monotone: a subset of after could never produce a false PASS. Check 18(g) at .github/codeowner-signoff-verify-prompt.md:934-937 never says that after must contain every measured head point.
Summary
Check 18(g) rejects a removed sweep point unless a single head point beats it on both
tput_per_gpuandmean_intvty. A point removed from the middle of a curve almost never passes that test, even when the new curve clearly sits above it. Since the rule landed in #4475 it has rejected #3835, #4281, #4357 and #4516. In #4516, the removed 1P2D c56 (13,058 at 114.2 tok/s/user) sits between head c48 (11,792 at 133.7) and c64 (15,322 at 103.5). The head curve reaches about 14,071 at that interactivity, but neither neighbor dominates c56 on its own.This PR lets a removed point pass when the measured head curve covers it. Either a head point dominates it, or it lies on or under the straight line between the two adjacent non-dominated head points that bracket its
mean_intvty. Dominated head points are dropped before the line is drawn, and nothing is extrapolated past the head's measured range.signoff_facts.py: new_cover()used by bothpoints_diff(the precomputedcheck_18g_pointsfacts) and thesignoff_facts dominancecommand.dominated_bynames either the dominating point or the segment, e.g.between c64 and c48.codeowner-signoff-verify-prompt.md: Check 18(g) states the same rule so the verifier does not overrule the facts.test_signoff_facts.py: covers the Refresh Kimi-K3 GB300 AgentX points on the latest vLLM nightly / 基于最新 vLLM nightly 更新 Kimi-K3 GB300 AgentX 点位 #4516 case, a point above the segment, a point past the head range, and a dominated head point that must not lower the segment.Risk: a straight segment can bridge a wide gap on a sparse curve (#4357's head spans 261 to 497 tok/s/user with no points in between). Check 14 still reports Pareto coverage, but only as a warning. The segment is stricter than the chart's monotone spline for the usual concave frontier, so a point just under the drawn curve can still fail.
Validation details
Replayed the four rejected PRs against this branch. #4516 used the full
signoff_facts reporton the PR head. The others usedsignoff_facts dominancewith the removed-point measurements quoted in each verdict and the head points of the validating run.#3835 is a real regression at that interactivity, so it still fails as intended.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists no affected config keys; noperf-changelog.yamlentry added中文
概述
Check 18(g) 只有在某个 head 点同时在
tput_per_gpu和mean_intvty上都优于被移除的点时才放行。从曲线中段移除的点几乎永远无法满足这个条件,即使新曲线明显高于它。该规则自 #4475 合入以来已拒绝 #3835、#4281、#4357 和 #4516。在 #4516 中,被移除的 1P2D c56(13,058,114.2 tok/s/user)位于 head 的 c48(11,792,133.7)和 c64(15,322,103.5)之间。head 曲线在该交互性处约为 14,071,但两个相邻点单独都无法支配 c56。本 PR 改为:只要实测的 head 曲线覆盖被移除的点即可放行。要么有 head 点支配它,要么它位于包围其
mean_intvty的两个相邻非支配 head 点之间连线上或线下。连线前会先去掉被支配的 head 点,且不会外推到 head 实测范围之外。signoff_facts.py:新增_cover(),供points_diff(预计算的check_18g_points)和signoff_facts dominance命令共用。dominated_by给出支配点或线段,例如between c64 and c48。codeowner-signoff-verify-prompt.md:Check 18(g) 写明相同规则,避免验证器推翻预计算结果。test_signoff_facts.py:覆盖 Refresh Kimi-K3 GB300 AgentX points on the latest vLLM nightly / 基于最新 vLLM nightly 更新 Kimi-K3 GB300 AgentX 点位 #4516 的情形、高于线段的点、超出 head 范围的点,以及不得拉低线段的被支配 head 点。风险:在点稀疏的曲线上,直线段可能跨越很大的空档(#4357 的 head 在 261 到 497 tok/s/user 之间没有点)。Check 14 仍会报告 Pareto 覆盖情况,但只是警告。对于常见的凹形前沿,直线比图表的单调样条更严格,所以略低于图上曲线的点仍可能失败。
验证
用本分支重放了四个被拒绝的 PR。#4516 在 PR head 上运行完整的
signoff_facts report;其余三个使用各自 verdict 中引用的被移除点数据和验证 run 的 head 点运行signoff_facts dominance。#4516、#4281、#4357 现在全部被覆盖;#3835 的 TP4 c15 仍然 FAIL(该处 head 连线约为 15,100,低于 22,342),这是该交互性下的真实退化,符合预期。AI 模型使用说明
改动类型
python -m infx.matrix.changed --base origin/main未列出受影响的配置键,未添加perf-changelog.yaml条目。