Repository navigation
fix(claude): the gate reads GitHub's timeline, not the model's formatting - #236
Conversation
…ting First live run of the finished reviewer, on #235: it found the planted defect at 🔴 and posted a correct finding, then wrote its summary as one flattened paragraph — `Code review. Base <sha>, Head <sha>. …`, no heading, no bold fields. The gate parses the heading and the bold fields, so it reported `no_earlier_review` and `first review` (the #231 failure, reintroduced by a formatting difference), and no empty push could ever be skipped. Both decisions the gate exists to make were dead, and every unit test was green. A model's prose is not a place to keep a policy decision. Three changes, in that order: 1. The summary is parsed loosely as well as strictly: `Base <sha>` / `Head <sha>` inside a sentence is a record, and a comment without the heading still counts when it names both commits. A full 40-character sha is required, so a finding quoting a 7-character prefix cannot become a baseline. 2. `GET /issues/{n}/timeline` is the fallback: `committed` events carry every pushed sha, `reviewed` events carry the `commit_id` a review was submitted against. The reviewer does not write either, so no wording can break the tier. 3. A timeline-only baseline never skips: the timeline has no base commit, a moved base changes what the pull request means, and the answer to an unknown is a review. Verified against the real pull request, read-only: the gate now reports `skip: no_reviewable_changes (tier: re-review)` for #235 where it previously reported `review: no_earlier_review (tier: first review)`. The skill now says how to post the summary — a file and `--body-file`, because a one-line `--body` string is what arrives flattened. Refs #73, #231, #233
Code reviewBase: What the change does
Findings
Verified
Not verified
Earlier findingsNone: first review. |
… stops promising a test run Both from the review of this pull request, which found a real defect in it. 1. `fetch_history` returned `earlier_review=False` when the summary was unrecognisable *and* the timeline call failed, so a transient API error re-created the #231 failure: a re-review announcing itself as a first review. It now reports an unreadable history, which `classify` turns into an unknown tier and a review. Pinned by a test that makes the timeline call fail. 2. The skill told the reviewer to "reproduce it if you can … `uv run --no-sync pytest`" and to reconstruct a parent with `git stash`/`git checkout`. None of those are in the workflow's allow list, and the review job has no database, so the instruction asked for checks the reviewer cannot make — an invitation to claim verification it did not perform (it did not, and said so under "Not verified"). Reading is what this job can do; the instruction now says that, and says a refused command is not a check. Engine 4114 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (105 refs) clean. Refs #236
Code reviewBase: What the change does
FindingsNo blocking findings. Verified
Not verified
Earlier findings
|
…ds (#237) Measured on #236, not inferred: the reviewer verified its own 🟡 finding, tried to resolve the thread, and reported `thread open: not permitted (no Contents: write)`. `resolveReviewThread` is gated on the token's **Contents: write**; `pull_requests: write` alone is refused with `Resource not accessible by integration` (github/gh-aw#35726, oven-sh/bun#36959 and #40468, all with live logs). That is not cosmetic here: branch protection has `required_conversation_resolution` enabled, so an unresolved thread the bot left behind **blocks the merge**. #236 could not be merged until a human resolved it. So both review jobs now grant `contents: write` — one permission, for one GraphQL mutation — and the compensating controls are what keep it narrow: `persist-credentials: false` on every checkout, `git push`/`commit`/`merge`, `gh pr merge/close/review/edit` denied, and every REST write verb (`gh api -X …`, `gh api --method …`) denied, leaving GraphQL as the only mutation path available to the job. The reviewer still checks `viewerCanResolve` first and reports `not permitted` rather than assuming; if an installation cannot grant Contents: write, the human fallback is unchanged and documented. Refs #236, #233
…olves them `resolveReviewThread` is gated on Contents: write, and the token the model runs with is the Claude App installation's, which this repository cannot grant. Measured, not assumed, and measured twice: #236 and #238 both reported `viewerCanResolve: false` and `thread open: not permitted`, truthfully. So every finding the reviewer verified left a thread open — and branch protection requires conversations resolved, so a bot's own verified finding blocked the merge (#236 needed a human to resolve it). The job's own token does hold the permission: a probe workflow ran the mutation with `secrets.GITHUB_TOKEN` and got `isResolved: true` back. So the split is now explicit: **the reviewer decides, the job acts.** * the review ends its summary with `<!-- resolve-threads: PRRT_… -->`, naming only threads whose fix it verified in the code; * `resolve_threads.py` (policy code from the base branch, like the gate) re-checks that each id is an *unresolved* thread of *this reviewer's* — an unknown id, a human's thread, or one already resolved is refused with a reason — then resolves the rest and prints `isResolved: true` per thread. Nothing it cannot confirm is reported as resolved; * a refusal is a warning and an open thread, never a failed review: the findings are already posted by then. This also removes the model-side mutation entirely, so the skill no longer instructs the reviewer to run something it cannot run, and the resolution is verified by the job rather than claimed by the model. Tests: 16 for the resolver (marker parsing, id validation, authorship, idempotency, confirmation, fail-soft, 3.12 syntax), plus the integration tests now pin the step in both review paths and the marker in the skill. Engine 4136 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, mechanical checks, statistics, plan refs, coverage, evidence (48 refs) all clean. Refs #236, #238, #73
… verified (#238) * feat(claude): a review that plans, admits doubt, and can post what it wrote Three defects the live runs on #235 exposed, and the review-quality work they justified. 1. The skill told the reviewer to post its summary with `--body-file <path>`. `Write` is denied in this job, so the model tried, was refused seven times (`permission_denials_count: 7`), and the run ended with no comment at all. It now posts from a quoted heredoc — `gh pr comment "$PR_NUMBER" --body-file -` — which needs no file and keeps the newlines the record parser reads. 2. Sonnet reviews what it is pointed at. The skill now opens with a plan (what the change claims, which guarantees it can break, the three files to read line by line), and every finding carries `**Basis:** proven|suspected` and its `**Fix:**` line. What the procedure cannot establish — a migration's behaviour, a race, a query plan — is not guessed: it writes `escalate: opus` and the question, and a maintainer re-runs with `CLAUDE_REVIEW_MODEL=claude-opus-5-5`. The model never switches itself and never claims a run it did not make. 3. `REVIEW.md` gains the order to work in (`How a review proceeds`), the domains a review is asked about — correctness, security, integrity and failure, concurrency, performance, compatibility and migrations, tests, simplicity — and the rule the #235 finding needed: *a consequence is a claim about a constraint*. That review said a weaker purge lock would lose a citation silently; `memory_derivation.source_version_id` is a restrictive foreign key, so the delete aborts. The finding was right and the consequence was wrong. Both files stay inside their ceilings (REVIEW.md 14,959/15,000; SKILL.md 19,986/20,000), which forced the consolidation the split wanted anyway: the procedure stays in the skill, the repository's domains and the standard stay in REVIEW.md. Engine 4119 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (47 refs) clean. Refs #235, #73 * fix(claude): restore what the review of this change found it had dropped The reviewer read the compression and found three real regressions in it, all of them mine: 1. the identity guard was gone from the skill's first code block — `[ "$REPOSITORY" = "${GITHUB_REPOSITORY:-$REPOSITORY}" ]` is back, so an invocation whose REPOSITORY disagrees with the run is still caught; 2. the link check read `$BODY`, a variable the heredoc no longer creates — an instruction that cannot be performed. It now reads the posted comment back (`gh pr view --comments | grep -o …`) and says to edit it if the owner is not unique; 3. "fix verified" had lost "and any test it asked for exists and would fail on the old code", which is the difference between verifying a fix and taking a push at its word. Its second finding was about the test itself: `"Write" in job` proves the word appears, not that the tool is denied. It now asserts the deny list entry. Both files stay inside their ceilings (19,986 and 14,959) after re-consolidating elsewhere. Refs #238 * feat(claude): the reviewer names the threads it verified, the job resolves them `resolveReviewThread` is gated on Contents: write, and the token the model runs with is the Claude App installation's, which this repository cannot grant. Measured, not assumed, and measured twice: #236 and #238 both reported `viewerCanResolve: false` and `thread open: not permitted`, truthfully. So every finding the reviewer verified left a thread open — and branch protection requires conversations resolved, so a bot's own verified finding blocked the merge (#236 needed a human to resolve it). The job's own token does hold the permission: a probe workflow ran the mutation with `secrets.GITHUB_TOKEN` and got `isResolved: true` back. So the split is now explicit: **the reviewer decides, the job acts.** * the review ends its summary with `<!-- resolve-threads: PRRT_… -->`, naming only threads whose fix it verified in the code; * `resolve_threads.py` (policy code from the base branch, like the gate) re-checks that each id is an *unresolved* thread of *this reviewer's* — an unknown id, a human's thread, or one already resolved is refused with a reason — then resolves the rest and prints `isResolved: true` per thread. Nothing it cannot confirm is reported as resolved; * a refusal is a warning and an open thread, never a failed review: the findings are already posted by then. This also removes the model-side mutation entirely, so the skill no longer instructs the reviewer to run something it cannot run, and the resolution is verified by the job rather than claimed by the model. Tests: 16 for the resolver (marker parsing, id validation, authorship, idempotency, confirmation, fail-soft, 3.12 syntax), plus the integration tests now pin the step in both review paths and the marker in the skill. Engine 4136 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, mechanical checks, statistics, plan refs, coverage, evidence (48 refs) all clean. Refs #236, #238, #73 * fix(claude): the review of the resolver found two things worth fixing Both from the on-demand review of this pull request, which is the first time the workflow-change fallback has produced an independent review of a workflow change. 1. The 🔴 it raised on its previous pass was only half fixed: the identity-mismatch guard and "would fail on the old code" were back in the skill, but nothing pinned them, and my assertion (`"$GITHUB_REPOSITORY" in skill`) was satisfied by the derivation sentence — so the next condensation could drop either again with every test green. Pinned by `test_the_two_checks_a_condensation_must_not_drop`, which matches the guard line itself. 2. `OWN_AUTHORS` matched the bare login `claude`, and `claude` is a legal human username: such an account's marker would have resolved threads on the bot's behalf. `own_claims` now also requires REST's `user.type` to be `Bot` when present. Engine 4136 passed / 21 skipped (63 in the two changed files); ruff, format, pyright, actionlint clean. Refs #238 * fix(claude): the on-demand review is told its tier instead of guessing it Its last run called itself "first review" on #238, a pull request with three earlier reviews — the third time a tier has been inferred from nothing in this workstream. An `issue_comment` payload carries no tier, but the gate can compute it from the review history exactly as it does for an automatic run, so the on-demand job now runs it too: `review` is ignored (a maintainer asked), and `tier` plus the previous shas are passed into the prompt. Also: `<!-- resolve-threads: -->` with no ids is a marker that names nothing, which the same run wrote. The skill now says to leave the marker out when nothing was verified. Engine 4,140+ passed; ruff, format, pyright, actionlint, root suite clean. Refs #238 * fix(claude): resolution fails closed — the pull request never supplies its own resolver The review of the complete diff flagged it as a 🟡: the resolve step fell back to the pull request's copy of `resolve_threads.py` when the base had none, running it with a `contents: write` token. The gate has that bootstrap on purpose — its safe default is a review, so it must survive being introduced — but resolution is the opposite: an open thread is what a human sees anyway, so the safe default is to resolve nothing. The fallback is gone; without the base copy the step says so and exits 0. Pinned by an assertion that the resolve step contains no `pr-head archive`. Refs #238
Refs #73
Found by running the finished reviewer live on a canary pull request (#235), not by reading it.
What happened
The reviewer found the planted defect at 🔴 with the right mechanism. It then posted its summary as one flattened paragraph —
Code review. Base <sha>, Head <sha>. Reviewed 1 file…— with no## Code reviewheading and no bold field names.The gate parses exactly that heading and those fields. So the next decision was:
which is both failures the gate exists to prevent, in one line: a re-review announcing itself as a first review (the #231 failure, reintroduced by a formatting difference), and
record is Nonemeaning no empty push can ever be skipped. The no-op optimisation was dead in production while every unit test was green.Fix
A policy decision was depending on model prose. It now depends on GitHub wherever it can:
Base <sha>/Head <sha>in a sentence is a record; a bot comment without the heading counts when it names both commits. Full 40-character shas only, so a finding that quotes a 7-character prefix cannot become a baseline.GET /issues/{n}/timeline:committedevents carry each pushed sha,reviewedevents carry thecommit_idthe review was submitted against. The reviewer writes neither.history_incompleteand reviews.The skill now says how to post the summary: a file and
gh pr comment --body-file, since a one-line--bodystring is what arrives flattened.Evidence
Read-only replay against the real pull request, real history, real API:
review: no_earlier_review (tier: first review)skip: no_reviewable_changes (tier: re-review)test_review_gate.py39 → 45 tests: the exact flattened body as a fixture, the timeline paths, a human's review event, a short hex string in prose, and a timeline-only record never licensing a skip. Together withtest_claude_integration.py: 78 passed.