Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/codeowner-signoff-verify-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -927,9 +927,12 @@ Entries are permanent, so this fails here; PR-body claims stay under Check 15.

### (g) Removed points were off the frontier
A removed sweep point is allowed only when it was not on the combined Pareto frontier,
or a measured head point dominates it in both `tput_per_gpu` and `mean_intvty`. Per
config key, `check_18g_points` lists `undominated_removed` (FAIL, unless head-point
measurements the facts missed dominate them) and `unmeasured_removed`. For unmeasured
or the measured head curve covers it: a head point dominates it in both `tput_per_gpu`
and `mean_intvty`, or it lies on or under the straight line between the two adjacent
non-dominated head points whose `mean_intvty` brackets its own (no extrapolation past
the head's range). `dominated_by` names the point or segment. Per config key,
`check_18g_points` lists `undominated_removed` (FAIL, unless head-point measurements
the facts missed cover them) and `unmeasured_removed`. For unmeasured
points, take measurements from the most recent sweep run that measured them
(`gh run download`) and run `signoff_facts dominance`. Points with the same shape and concurrency but a new
Comment on lines +934 to 937

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Verifiers re-checking removed points with signoff_facts dominance can now get a false PASS by passing only some head points as after. Under the old rule a partial list could only fail too often. _cover builds its segment from whatever after it is given. If a lower non-dominated head point between the bracketing pair is left out, the line rises and a removed point above the real head curve reports "between A and B". Fix: require after to hold every measured head point of the config key, i.e. the facts' head measurements plus any the facts missed. Also have check_18g_points output the head points it used so the verifier can copy them.

Why this was flagged

Check 18(g) at .github/codeowner-signoff-verify-prompt.md:934-937 tells the verifier to run signoff_facts dominance for unmeasured points, and to look at head measurements "the facts missed". It never says which head points go into after. points_diff (signoff_facts.py:442-517) does not report the full head set it fed into _cover, only added and removed. So the verifier rebuilds after from a run download, as the PR's own replays did for #4281/#4357 ("the head points of the validating run"). That leaves out unchanged head points measured only on the ingested curve. In _cover (signoff_facts.py:300-307), dropping a non-dominated head point C that sits below the A-B chord removes the A-C-B kink. The line at the removed point's mean_intvty then rises and the point is reported as covered, even though the full head frontier fails it. On the base branch dominated_by needed one dominating point, so a partial after could only cause a false FAIL, never a false PASS.

Verification: Trigger: a verifier runs signoff_facts dominance with an after list that leaves out a non-dominated head point. In signoff_facts.py, _cover builds frontier only from the others it is passed. The old rule was monotone: a subset of after could never produce a false PASS. Check 18(g) at .github/codeowner-signoff-verify-prompt.md:934-937 never says that after must contain every measured head point.

recipe fingerprint are re-tuned, not removed; they appear under `retuned_deltas`.
Expand Down
15 changes: 15 additions & 0 deletions inferencex-e2e/infx/tests/workflows/test_signoff_facts.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,21 @@ def test_dominance_requires_removed_points_to_be_dominated():
assert report["deltas"] == [{"label": "keep", "tput_per_gpu_pct": 10.0, "mean_intvty_pct": 0.0}]


@pytest.mark.parametrize(
("removed", "covered_by"),
[
(point("c56", 13058, 114.2), "between c64 and c48"),
(point("above", 14500, 114.2), None),
(point("past", 5000, 140.0), None),
],
)
def test_dominance_covers_points_under_the_head_frontier_segment(removed, covered_by):
# "slow" is dominated by c48; as a segment end it would drop the line under c56.
after = [point("c48", 11792, 133.7), point("c64", 15322, 103.5), point("slow", 9000, 120.0)]
report = signoff_facts.dominance([removed], after)
assert report["removed"] == [{"label": removed["label"], "dominated_by": covered_by}]


def test_dominance_rejects_missing_metrics():
with pytest.raises(ValueError, match="positive"):
signoff_facts.dominance([point("a", 0, 50)], [])
Expand Down
43 changes: 31 additions & 12 deletions inferencex-e2e/infx/workflows/signoff_facts.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import argparse
import io
import itertools
import json
import math
import re
Expand Down Expand Up @@ -287,8 +288,29 @@ def _dominates(a: dict, b: dict) -> bool:
return a[x] >= b[x] and a[y] >= b[y] and (a[x] > b[x] or a[y] > b[y])


def _cover(point: dict, others: list[dict]) -> str | None:
"""What covers a removed point: a dominating point, or the frontier segment over it.

The segment is the straight line between the two adjacent non-dominated points that
bracket the point's interactivity. Nothing covers a point outside their range.
"""
cover = next((other for other in others if _dominates(other, point)), None)
if cover:
return cover["label"]
x, y = "mean_intvty", "tput_per_gpu"
frontier = sorted(
(other for other in others if not any(_dominates(o, other) for o in others)),
key=lambda other: other[x],
)
for low, high in itertools.pairwise(frontier):
if low[x] < point[x] < high[x]:
line = low[y] + (high[y] - low[y]) * (point[x] - low[x]) / (high[x] - low[x])
return f"between {low['label']} and {high['label']}" if line >= point[y] else None
return None


def dominance(before: list[dict], after: list[dict]) -> dict[str, Any]:
"""Removed points must be dominated by an after point; shared points report deltas."""
"""Removed points must lie under the after frontier; shared points report deltas."""
for point in (*before, *after):
if not isinstance(point.get("label"), str) or not all(
isinstance(point.get(metric), int | float)
Expand All @@ -304,8 +326,7 @@ def dominance(before: list[dict], after: list[dict]) -> dict[str, Any]:
for point in before:
if point["label"] in after_by_label:
continue
cover = next((other for other in after if _dominates(other, point)), None)
removed.append({"label": point["label"], "dominated_by": cover and cover["label"]})
removed.append({"label": point["label"], "dominated_by": _cover(point, after)})
deltas = [
{
"label": point["label"],
Expand Down Expand Up @@ -424,7 +445,7 @@ def points_diff(base: dict, head: dict, ingested: list[Any], new_run: list[Any])

Points match measurements by recipe fingerprint and concurrency. Base points use the
ingested curve; head points prefer the PR's validated sweep, then the ingested curve.
A removed point passes only when a measured head point dominates it.
A removed point passes only when the measured head frontier covers it.
"""
from infx.matrix.changed import compare

Expand All @@ -441,18 +462,16 @@ def measured(point: dict, table: dict[tuple, dict]) -> dict:
measured(point, after) for point in head.get(change["config-key"], ({}, []))[0].values()
]
added = [measured(point, after) for point in change["added"]]
covers = [
other["measured"] | {"label": other["label"]}
for other in head_points
if other["measured"]
]
removed = []
for point in change["removed"]:
item = measured(point, before)
if item["measured"]:
item["dominated_by"] = next(
(
other["label"]
for other in head_points
if other["measured"] and _dominates(other["measured"], item["measured"])
),
None,
)
item["dominated_by"] = _cover(item["measured"], covers)
removed.append(item)
added_labels = {item["label"] for item in added}
replaced = {item["label"]: item for item in added if item["measured"]}
Expand Down
Loading