Skip to content

fix(claude): the gate reads GitHub's timeline, not the model's formatting - #236

Merged
elewashy merged 2 commits into
mainfrom
fix/gate-reads-github-not-formatting
Nov 21, 2026
Merged

elewashy merged 2 commits into
mainfrom
fix/gate-reads-github-not-formatting

Conversation

@elewashy

@elewashy elewashy commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #73

Found by running the finished reviewer live on a canary pull request (#235), not by reading it.

What happened

The reviewer found the planted defect at 🔴 with the right mechanism. It then posted its summary as one flattened paragraph — Code review. Base <sha>, Head <sha>. Reviewed 1 file… — with no ## Code review heading and no bold field names.

The gate parses exactly that heading and those fields. So the next decision was:

review: no_earlier_review (tier: first review)

which is both failures the gate exists to prevent, in one line: a re-review announcing itself as a first review (the #231 failure, reintroduced by a formatting difference), and record is None meaning no empty push can ever be skipped. The no-op optimisation was dead in production while every unit test was green.

Fix

A policy decision was depending on model prose. It now depends on GitHub wherever it can:

  1. Looser parse — Base <sha> / Head <sha> in a sentence is a record; a bot comment without the heading counts when it names both commits. Full 40-character shas only, so a finding that quotes a 7-character prefix cannot become a baseline.
  2. Timeline fallback — GET /issues/{n}/timeline: committed events carry each pushed sha, reviewed events carry the commit_id the review was submitted against. The reviewer writes neither.
  3. No skip without a base — the timeline has no base commit, and a base that moved changes what the pull request means. A timeline-only baseline reports history_incomplete and reviews.

The skill now says how to post the summary: a file and gh pr comment --body-file, since a one-line --body string is what arrives flattened.

Evidence

Read-only replay against the real pull request, real history, real API:

#235, head 6cfac7e, base 615ad97 decision
before review: no_earlier_review (tier: first review)
after skip: no_reviewable_changes (tier: re-review)

test_review_gate.py 39 → 45 tests: the exact flattened body as a fixture, the timeline paths, a human's review event, a short hex string in prose, and a timeline-only record never licensing a skip. Together with test_claude_integration.py: 78 passed.

…ting

First live run of the finished reviewer, on #235: it found the planted defect at 🔴 and posted a correct finding, then
wrote its summary as one flattened paragraph — `Code review. Base <sha>, Head <sha>. …`, no heading, no bold fields.
The gate parses the heading and the bold fields, so it reported `no_earlier_review` and `first review` (the #231
failure, reintroduced by a formatting difference), and no empty push could ever be skipped. Both decisions the gate
exists to make were dead, and every unit test was green.

A model's prose is not a place to keep a policy decision. Three changes, in that order:

1. The summary is parsed loosely as well as strictly: `Base <sha>` / `Head <sha>` inside a sentence is a record, and a
   comment without the heading still counts when it names both commits. A full 40-character sha is required, so a
   finding quoting a 7-character prefix cannot become a baseline.
2. `GET /issues/{n}/timeline` is the fallback: `committed` events carry every pushed sha, `reviewed` events carry the
   `commit_id` a review was submitted against. The reviewer does not write either, so no wording can break the tier.
3. A timeline-only baseline never skips: the timeline has no base commit, a moved base changes what the pull request
   means, and the answer to an unknown is a review.

Verified against the real pull request, read-only: the gate now reports `skip: no_reviewable_changes (tier: re-review)`
for #235 where it previously reported `review: no_earlier_review (tier: first review)`.

The skill now says how to post the summary — a file and `--body-file`, because a one-line `--body` string is what
arrives flattened.

Refs #73, #231, #233
Comment thread .github/scripts/review_gate.py Resolved
@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Code review

Base: 615ad97999e75ffb562e298c0b235e0dca972959 · Head: 479fe7164e3038fd65b0ef36d46c72fd46986232 · Reviewed: 4 files, +185/-20 · Tier: first review (no_earlier_review)

What the change does

review_gate.py now accepts a flattened prose summary (Base <40-hex>, Head <40-hex>) as a record. It falls back to the PR timeline for the tier when no summary is recognised, and a timeline-only baseline never licenses a skip. The skill now tells the reviewer to post its summary with --body-file.

Findings

# Severity Location Issue
1 🟡 .github/scripts/review_gate.py::fetch_history A failed timeline fetch returns earlier_review=False, so the gate reports "first review" instead of unknown history.

Verified

  • Read the full diff and classify end to end. A base_sha == "" record always yields review=True, history_incomplete, so the timeline path cannot cause a skip.
  • The 40-hex requirement in the loose patterns keeps a short hex string in prose from becoming a baseline.
  • The workflow grants issues: read, which the timeline call needs.

Not verified

  • I did not run test_review_gate.py: the shell command was not approved in this session.
  • I did not replay the gate against perf(purge): take a shared lock over purge candidates #235 or call the live timeline API.
  • Real committed and reviewed event shapes are assumed from the fixtures. The timeline orders committed events by commit date, so the recovered head may be off. This cannot cause a skip, because the base is empty.

Earlier findings

None: first review.

… stops promising a test run

Both from the review of this pull request, which found a real defect in it.

1. `fetch_history` returned `earlier_review=False` when the summary was unrecognisable *and* the timeline call failed,
   so a transient API error re-created the #231 failure: a re-review announcing itself as a first review. It now reports
   an unreadable history, which `classify` turns into an unknown tier and a review. Pinned by a test that makes the
   timeline call fail.

2. The skill told the reviewer to "reproduce it if you can … `uv run --no-sync pytest`" and to reconstruct a parent
   with `git stash`/`git checkout`. None of those are in the workflow's allow list, and the review job has no database,
   so the instruction asked for checks the reviewer cannot make — an invitation to claim verification it did not
   perform (it did not, and said so under "Not verified"). Reading is what this job can do; the instruction now says
   that, and says a refused command is not a check.

Engine 4114 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (105 refs) clean.

Refs #236
@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Code review

Base: 615ad97999e75ffb562e298c0b235e0dca972959 · Head: 8d12c4bbc134a84c734878e284717e9b39babb56 · Reviewed: 4 files, +208/-26 · Tier: re-review (history read)

What the change does

review_gate.py accepts a flattened prose summary as a review record, and falls back to the PR timeline for the tier when no summary is recognised. A timeline-only baseline never licenses a skip. The latest push makes an unreadable timeline report unknown history instead of a first review.

Findings

No blocking findings.

Verified

  • Read fetch_history at the head (.github/scripts/review_gate.py lines 251-270). When the comments are readable but unrecognised and the timeline call fails, it now returns History(available=False, ...). The new test stubs _api_list to produce that case and expects classify to give review=True, tier=unknown, reason=history_unavailable.
  • The push also edits .claude/skills/review-pr/SKILL.md to match the read-only tool set. This weakens no check, and REVIEW.md still requires every finding to carry its evidence.

Not verified

  • I did not run test_review_gate.py. No test runner is available in this session.
  • I did not replay the gate against the live timeline API.

Earlier findings

  1. 🟡 fetch_history reporting a first review on a failed timeline fetch: fix verified, thread open (not permitted: viewerCanResolve is false for this token, so a human can resolve it).

@elewashy
elewashy merged commit e1ecbfc into main Nov 21, 2026
8 checks passed
@elewashy
elewashy deleted the fix/gate-reads-github-not-formatting branch November 21, 2026 11:57
elewashy added a commit that referenced this pull request Nov 21, 2026
…ds (#237)

Measured on #236, not inferred: the reviewer verified its own 🟡 finding, tried to resolve the thread, and reported
`thread open: not permitted (no Contents: write)`. `resolveReviewThread` is gated on the token's **Contents: write**;
`pull_requests: write` alone is refused with `Resource not accessible by integration` (github/gh-aw#35726,
oven-sh/bun#36959 and #40468, all with live logs).

That is not cosmetic here: branch protection has `required_conversation_resolution` enabled, so an unresolved thread the
bot left behind **blocks the merge**. #236 could not be merged until a human resolved it.

So both review jobs now grant `contents: write` — one permission, for one GraphQL mutation — and the compensating
controls are what keep it narrow: `persist-credentials: false` on every checkout, `git push`/`commit`/`merge`,
`gh pr merge/close/review/edit` denied, and every REST write verb (`gh api -X …`, `gh api --method …`) denied, leaving
GraphQL as the only mutation path available to the job.

The reviewer still checks `viewerCanResolve` first and reports `not permitted` rather than assuming; if an installation
cannot grant Contents: write, the human fallback is unchanged and documented.

Refs #236, #233
elewashy added a commit that referenced this pull request Nov 21, 2026
…olves them

`resolveReviewThread` is gated on Contents: write, and the token the model runs with is the Claude App installation's,
which this repository cannot grant. Measured, not assumed, and measured twice: #236 and #238 both reported
`viewerCanResolve: false` and `thread open: not permitted`, truthfully. So every finding the reviewer verified left a
thread open — and branch protection requires conversations resolved, so a bot's own verified finding blocked the merge
(#236 needed a human to resolve it).

The job's own token does hold the permission: a probe workflow ran the mutation with `secrets.GITHUB_TOKEN` and got
`isResolved: true` back. So the split is now explicit: **the reviewer decides, the job acts.**

* the review ends its summary with `<!-- resolve-threads: PRRT_… -->`, naming only threads whose fix it verified in the
  code;
* `resolve_threads.py` (policy code from the base branch, like the gate) re-checks that each id is an *unresolved* thread
  of *this reviewer's* — an unknown id, a human's thread, or one already resolved is refused with a reason — then
  resolves the rest and prints `isResolved: true` per thread. Nothing it cannot confirm is reported as resolved;
* a refusal is a warning and an open thread, never a failed review: the findings are already posted by then.

This also removes the model-side mutation entirely, so the skill no longer instructs the reviewer to run something it
cannot run, and the resolution is verified by the job rather than claimed by the model.

Tests: 16 for the resolver (marker parsing, id validation, authorship, idempotency, confirmation, fail-soft, 3.12 syntax),
plus the integration tests now pin the step in both review paths and the marker in the skill. Engine 4136 passed /
21 skipped; root 15; ruff, format, pyright, actionlint, mechanical checks, statistics, plan refs, coverage, evidence
(48 refs) all clean.

Refs #236, #238, #73
elewashy added a commit that referenced this pull request Nov 21, 2026
… verified (#238)

* feat(claude): a review that plans, admits doubt, and can post what it wrote

Three defects the live runs on #235 exposed, and the review-quality work they justified.

1. The skill told the reviewer to post its summary with `--body-file <path>`. `Write` is denied in this job, so the model
   tried, was refused seven times (`permission_denials_count: 7`), and the run ended with no comment at all. It now posts
   from a quoted heredoc — `gh pr comment "$PR_NUMBER" --body-file -` — which needs no file and keeps the newlines the
   record parser reads.

2. Sonnet reviews what it is pointed at. The skill now opens with a plan (what the change claims, which guarantees it can
   break, the three files to read line by line), and every finding carries `**Basis:** proven|suspected` and its `**Fix:**`
   line. What the procedure cannot establish — a migration's behaviour, a race, a query plan — is not guessed: it writes
   `escalate: opus` and the question, and a maintainer re-runs with `CLAUDE_REVIEW_MODEL=claude-opus-5-5`. The model never
   switches itself and never claims a run it did not make.

3. `REVIEW.md` gains the order to work in (`How a review proceeds`), the domains a review is asked about — correctness,
   security, integrity and failure, concurrency, performance, compatibility and migrations, tests, simplicity — and the
   rule the #235 finding needed: *a consequence is a claim about a constraint*. That review said a weaker purge lock would
   lose a citation silently; `memory_derivation.source_version_id` is a restrictive foreign key, so the delete aborts. The
   finding was right and the consequence was wrong.

Both files stay inside their ceilings (REVIEW.md 14,959/15,000; SKILL.md 19,986/20,000), which forced the consolidation
the split wanted anyway: the procedure stays in the skill, the repository's domains and the standard stay in REVIEW.md.

Engine 4119 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (47 refs) clean.

Refs #235, #73

* fix(claude): restore what the review of this change found it had dropped

The reviewer read the compression and found three real regressions in it, all of them mine:

1. the identity guard was gone from the skill's first code block — `[ "$REPOSITORY" = "${GITHUB_REPOSITORY:-$REPOSITORY}" ]`
   is back, so an invocation whose REPOSITORY disagrees with the run is still caught;
2. the link check read `$BODY`, a variable the heredoc no longer creates — an instruction that cannot be performed. It now
   reads the posted comment back (`gh pr view --comments | grep -o …`) and says to edit it if the owner is not unique;
3. "fix verified" had lost "and any test it asked for exists and would fail on the old code", which is the difference
   between verifying a fix and taking a push at its word.

Its second finding was about the test itself: `"Write" in job` proves the word appears, not that the tool is denied. It
now asserts the deny list entry.

Both files stay inside their ceilings (19,986 and 14,959) after re-consolidating elsewhere.

Refs #238

* feat(claude): the reviewer names the threads it verified, the job resolves them

`resolveReviewThread` is gated on Contents: write, and the token the model runs with is the Claude App installation's,
which this repository cannot grant. Measured, not assumed, and measured twice: #236 and #238 both reported
`viewerCanResolve: false` and `thread open: not permitted`, truthfully. So every finding the reviewer verified left a
thread open — and branch protection requires conversations resolved, so a bot's own verified finding blocked the merge
(#236 needed a human to resolve it).

The job's own token does hold the permission: a probe workflow ran the mutation with `secrets.GITHUB_TOKEN` and got
`isResolved: true` back. So the split is now explicit: **the reviewer decides, the job acts.**

* the review ends its summary with `<!-- resolve-threads: PRRT_… -->`, naming only threads whose fix it verified in the
  code;
* `resolve_threads.py` (policy code from the base branch, like the gate) re-checks that each id is an *unresolved* thread
  of *this reviewer's* — an unknown id, a human's thread, or one already resolved is refused with a reason — then
  resolves the rest and prints `isResolved: true` per thread. Nothing it cannot confirm is reported as resolved;
* a refusal is a warning and an open thread, never a failed review: the findings are already posted by then.

This also removes the model-side mutation entirely, so the skill no longer instructs the reviewer to run something it
cannot run, and the resolution is verified by the job rather than claimed by the model.

Tests: 16 for the resolver (marker parsing, id validation, authorship, idempotency, confirmation, fail-soft, 3.12 syntax),
plus the integration tests now pin the step in both review paths and the marker in the skill. Engine 4136 passed /
21 skipped; root 15; ruff, format, pyright, actionlint, mechanical checks, statistics, plan refs, coverage, evidence
(48 refs) all clean.

Refs #236, #238, #73

* fix(claude): the review of the resolver found two things worth fixing

Both from the on-demand review of this pull request, which is the first time the workflow-change fallback has produced
an independent review of a workflow change.

1. The 🔴 it raised on its previous pass was only half fixed: the identity-mismatch guard and "would fail on the old code"
   were back in the skill, but nothing pinned them, and my assertion (`"$GITHUB_REPOSITORY" in skill`) was satisfied by
   the derivation sentence — so the next condensation could drop either again with every test green. Pinned by
   `test_the_two_checks_a_condensation_must_not_drop`, which matches the guard line itself.

2. `OWN_AUTHORS` matched the bare login `claude`, and `claude` is a legal human username: such an account's marker would
   have resolved threads on the bot's behalf. `own_claims` now also requires REST's `user.type` to be `Bot` when present.

Engine 4136 passed / 21 skipped (63 in the two changed files); ruff, format, pyright, actionlint clean.

Refs #238

* fix(claude): the on-demand review is told its tier instead of guessing it

Its last run called itself "first review" on #238, a pull request with three earlier reviews — the third time a tier has
been inferred from nothing in this workstream. An `issue_comment` payload carries no tier, but the gate can compute it
from the review history exactly as it does for an automatic run, so the on-demand job now runs it too: `review` is
ignored (a maintainer asked), and `tier` plus the previous shas are passed into the prompt.

Also: `<!-- resolve-threads: -->` with no ids is a marker that names nothing, which the same run wrote. The skill now
says to leave the marker out when nothing was verified.

Engine 4,140+ passed; ruff, format, pyright, actionlint, root suite clean.

Refs #238

* fix(claude): resolution fails closed — the pull request never supplies its own resolver

The review of the complete diff flagged it as a 🟡: the resolve step fell back to the pull request's copy of
`resolve_threads.py` when the base had none, running it with a `contents: write` token. The gate has that bootstrap on
purpose — its safe default is a review, so it must survive being introduced — but resolution is the opposite: an open
thread is what a human sees anyway, so the safe default is to resolve nothing. The fallback is gone; without the base
copy the step says so and exits 0.

Pinned by an assertion that the resolve step contains no `pr-head archive`.

Refs #238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant