Repository navigation
fix(claude): the reviewer's identity, its tier, and a resolution it must verify - #233
Merged
Merged
Conversation
…ust verify The canary (a real defect in a throwaway PR, `.github/workflows/README.md`) found the reviewer reasoning well and getting four mechanical things wrong. Each is fixed where it is decided, not by asking the prompt to be more careful. Identity. The skill said `https://github.com/<owner>/<repo>/…`, a placeholder, so the owner was filled from memory and one review cited `HussainAlmutawa/ChronaMind`. `REPOSITORY` is now passed by both review workflows, is the only source of a URL, and an identity that cannot be established drops the link instead of inventing one. Tier. `review-pr` inferred "am I a re-review?" from the diff. When a base branch advances, the diff range moves, the earlier findings leave it, and a second review calls itself a first — which is what happened: it re-reported nothing and left its own 🔴 thread open on a defect it had identified. `.github/scripts/review_gate.py` now reads the pull request's review history before the model runs and passes `REVIEW_TIER`, `PREVIOUS_HEAD_SHA` and `PREVIOUS_BASE_SHA`. Resolution. The skill said, in one line, to resolve a fixed thread. It now fetches threads by GraphQL, matches them by comment content and author, verifies the fix in the code, and resolves only when GitHub answers `isResolved: true` — reporting the failure and leaving the thread open otherwise. No `permissions:` change: the action authenticates as the Claude GitHub App through OIDC, and that installation is the identity that already posts the comments; the job's `GITHUB_TOKEN` stays read-only, as `test_the_review_holds_a_read_only_token_and_cannot_merge` requires. Cost. A push that changes no tree bought a full run and posted nothing. The gate compares the pushed tree with the head the last review recorded and skips before the action mints a token — no path allow list, because a change to `PLAN/V2/`, `.github/` or an instruction file is reviewable here. A crashed gate fails open to a review, never to a skip. Reachability. A finding about a guard now has to trace the path and name the guards upstream before it sets a severity, and carry a `**Reachability:**` line. The canary's finding was right and its exploit was not: `row_visible` refused the input first. A latent violation of a stated guarantee is still 🔴 — the analysis is for accuracy, not for silence. Tests: 33 new. `custom-memory-engine/tests/test_review_gate.py` covers every classification (first, re-review, base advanced, base merge, fix-only, regression-while-fixing, unreadable and unparseable history, missing commits), the summary-format seam, and the gate's own entry point. `tests/test_claude_integration.py` holds the workflow and the instructions to them, and runs the skill's documented link check against a fabricated owner. Refs #73
…e is none CI caught this on the first run: the gate step executed from the base checkout, where `.github/scripts/` does not exist until this pull request merges — `can't open file .../review_gate.py`, exit 2, a red job on the change that adds the gate. The gate decides whether a pull request is reviewed, so it is policy code and is read from the base branch, exactly as `REVIEW.md` and the skill are: a pull request must not be able to edit its own gate into skipping it. The pull request that introduces it is the one case with no base copy, so it falls back to `pr-head` and says so in the log — the same bootstrap the skill's own step already uses. With neither copy present the step does not fail: it emits `review=true`, `tier=unknown`, `reason=gate_failed`, because a gate that cannot decide is a reason to review, never a reason to skip. Pinned by `test_the_gate_comes_from_the_base_branch_and_tolerates_its_own_introduction`. Refs #73
Collaborator
Author
|
@chronamind Diagnostic only — change nothing, resolve nothing. I need to know whether the app token this action runs as is authorised to resolve review threads. Run exactly these two commands and paste the raw output of each:
Then tell me, in one line each: (a) the value of |
|
I'll analyze this and get back to you. |
…reviews the reviewer Three gaps the first round could not close, and one bug found while closing them. **Resolution needs `Contents: write`, not `pull_requests: write`.** `resolveReviewThread` is gated on repository write for a GitHub App installation token; a token holding only `pull_requests: write` is refused with `Resource not accessible by integration` (`github/gh-aw#35726`, `oven-sh/bun#36959`, `zuke-build/zuke#634`). The action's default token request does ask for `contents: write` too, so this installation should already qualify — but "should" is not grounds for writing "resolved", so the query now reads `viewerCanResolve` and the skill refuses to attempt the mutation when it is false, reporting `thread open: not permitted (no Contents: write)` instead of a silent skip. End-to-end resolution is still unproven here: it needs a review of a real fix, which needs this change on `main`. **A workflow change is never auto-reviewed, and that now says so.** The action's OIDC exchange refuses a workflow file that differs from the default branch's, so `claude-review` goes green with the review skipped — the failure shape where green means "not reviewed". The gate detects the case from the pull request's file list and prints a warning plus a step-summary notice naming the workflow and the path that works: a maintainer's `@claude review`, which runs on `issue_comment` and therefore on the default branch's workflow. No permission is widened to dodge the rule. **No-op detection, proven end to end.** The gate is now exercised as the workflow runs it — a real git repository, a stub `gh` on `PATH`, the script as a subprocess — so the "identical tree, do not review" answer and the output file the review step is gated on are both tested, not just `classify()`. A pull request that changes the tree still reviews, an unreadable file list does not block, and an unparseable history still reviews. **A bug the diagnostic found: the assistant cancelled itself.** The action posts an acknowledgement comment as it starts; that comment is an `issue_comment` event, joined the same concurrency group, and cancelled the run that had just posted it ("Claude Code is working…" 10:07:11Z, "The operation was canceled" 10:07:27Z, #233). A run started by a bot now lands in a group keyed on its own run id, so only a human's comment can supersede a human's request. Also: a controlled-canary procedure in the documentation (one CI-invisible defect, expected finding written first, never touching the workflow or the standard, closed unmerged) so the next validation is repeatable; and `SKILL.md` was consolidated — its cap sentence and "do not report" list now point at `REVIEW.md` instead of repeating it — to stay under its 20,000-byte ceiling at 19,961. The gate is pinned to Python 3.12, because Ruff with the root's 3.14 target rewrote an `except (A, B, C)` into PEP 758 syntax that is a SyntaxError on the runner. Refs #73
The notice reads to a maintainer under a green check, so it says "edits `a` and `b`, so …" rather than a comma splice. The helper is unit-tested for one, two and three items. Refs #73
Collaborator
Author
|
@claude review |
…as no remote `@claude review` is the documented way to review a pull request that edits the review workflow — the automatic job cannot, because the action's token exchange refuses a workflow file that differs from the default branch's. That path was broken from the day it was written: its checkout step ran `git init --quiet pr-head`, which creates a repository with no remote, and then `git -C pr-head fetch origin "$HEAD_SHA"` — `fatal: Could not read from remote repository`, exit 128, before the review step. Its first live attempt was #233 at 10:27Z, and it failed exactly there. It now uses `actions/checkout`, as the automatic review does, with the head taken from the repository the pull request came from (`head.repo.full_name`, newly resolved in the API step) so a fork still works when a maintainer asks for one. Authentication is the action's, and `persist-credentials: false` is kept. This is what makes an independent review of a workflow change possible at all; without it, the fallback documented for that gap is a check that goes green having reviewed nothing. Refs #73
Collaborator
Author
|
@claude review |
elewashy
added a commit
that referenced
this pull request
Nov 21, 2026
… never ran (#234) Live verification of the on-demand path after #233 merged: the job now completes (its checkout was broken until `f44ce3f`'s successor), the prompt carried `REPOSITORY`, `BASE_SHA` and `HEAD_SHA`, the model ran 17 turns for $0.328 — and it posted nothing at all: no summary, no inline comment, no review. The pull request had already been merged, which is exactly when a reader most needs telling what was reviewed. The skill now requires the summary unconditionally: say what you read, and that it is no longer actionable. That closes the last silent path in a procedure whose whole purpose is that a review cannot be mistaken for silence. `SKILL.md` was consolidated again to stay under its 20,000-byte ceiling (19,992): the same requirements, restated more tightly where two sentences said one thing. Refs #73
elewashy
added a commit
that referenced
this pull request
Nov 21, 2026
…ting (#236) * fix(claude): the gate reads GitHub's timeline, not the model's formatting First live run of the finished reviewer, on #235: it found the planted defect at 🔴 and posted a correct finding, then wrote its summary as one flattened paragraph — `Code review. Base <sha>, Head <sha>. …`, no heading, no bold fields. The gate parses the heading and the bold fields, so it reported `no_earlier_review` and `first review` (the #231 failure, reintroduced by a formatting difference), and no empty push could ever be skipped. Both decisions the gate exists to make were dead, and every unit test was green. A model's prose is not a place to keep a policy decision. Three changes, in that order: 1. The summary is parsed loosely as well as strictly: `Base <sha>` / `Head <sha>` inside a sentence is a record, and a comment without the heading still counts when it names both commits. A full 40-character sha is required, so a finding quoting a 7-character prefix cannot become a baseline. 2. `GET /issues/{n}/timeline` is the fallback: `committed` events carry every pushed sha, `reviewed` events carry the `commit_id` a review was submitted against. The reviewer does not write either, so no wording can break the tier. 3. A timeline-only baseline never skips: the timeline has no base commit, a moved base changes what the pull request means, and the answer to an unknown is a review. Verified against the real pull request, read-only: the gate now reports `skip: no_reviewable_changes (tier: re-review)` for #235 where it previously reported `review: no_earlier_review (tier: first review)`. The skill now says how to post the summary — a file and `--body-file`, because a one-line `--body` string is what arrives flattened. Refs #73, #231, #233 * fix(claude): an unreadable timeline is unknown history, and the skill stops promising a test run Both from the review of this pull request, which found a real defect in it. 1. `fetch_history` returned `earlier_review=False` when the summary was unrecognisable *and* the timeline call failed, so a transient API error re-created the #231 failure: a re-review announcing itself as a first review. It now reports an unreadable history, which `classify` turns into an unknown tier and a review. Pinned by a test that makes the timeline call fail. 2. The skill told the reviewer to "reproduce it if you can … `uv run --no-sync pytest`" and to reconstruct a parent with `git stash`/`git checkout`. None of those are in the workflow's allow list, and the review job has no database, so the instruction asked for checks the reviewer cannot make — an invitation to claim verification it did not perform (it did not, and said so under "Not verified"). Reading is what this job can do; the instruction now says that, and says a refused command is not a check. Engine 4114 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (105 refs) clean. Refs #236
elewashy
added a commit
that referenced
this pull request
Nov 21, 2026
…ds (#237) Measured on #236, not inferred: the reviewer verified its own 🟡 finding, tried to resolve the thread, and reported `thread open: not permitted (no Contents: write)`. `resolveReviewThread` is gated on the token's **Contents: write**; `pull_requests: write` alone is refused with `Resource not accessible by integration` (github/gh-aw#35726, oven-sh/bun#36959 and #40468, all with live logs). That is not cosmetic here: branch protection has `required_conversation_resolution` enabled, so an unresolved thread the bot left behind **blocks the merge**. #236 could not be merged until a human resolved it. So both review jobs now grant `contents: write` — one permission, for one GraphQL mutation — and the compensating controls are what keep it narrow: `persist-credentials: false` on every checkout, `git push`/`commit`/`merge`, `gh pr merge/close/review/edit` denied, and every REST write verb (`gh api -X …`, `gh api --method …`) denied, leaving GraphQL as the only mutation path available to the job. The reviewer still checks `viewerCanResolve` first and reports `not permitted` rather than assuming; if an installation cannot grant Contents: write, the human fallback is unchanged and documented. Refs #236, #233
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Refs #73 · do not merge: this is a review of the reviewer, and merging it is the owner's decision.
Fixes the four things the canary validation showed the automated review getting wrong, plus the reasoning weakness it exposed. Every improvement is decided before the model runs, or verified after GitHub answers — none of them is "ask the prompt to be more careful".
1. Root cause and fix, per item
A. Identity and link integrity (P0)
Root cause. The skill gave the link as
https://github.com/<owner>/<repo>/blob/<sha>/<path>#L<n>— a placeholder, and no canonical identity was passed anywhere in the invocation. A model filling<owner>/<repo>fills it from memory; the canary's inline comment citedHussainAlmutawa/ChronaMind, then corrected itself in the summary. The summary was right, the link was not, and the link is what a reader follows.Fix.
REPOSITORY=${{ github.repository }}is passed by both review workflows. Skill §1 states it is the only source of any URL, forbids writing an owner/name/branch/sha/path from memory, requires omitting the link (link omitted: identity unavailable) when the identity or sha cannot be established, and gives a pre-post check (grep -o 'github\.com/[^/]*/[^/]*'must return exactly one value, ours).B. Explicit re-review state (P0)
Root cause. The skill inferred its tier from the diff (
If you are reviewing a later commit… a synchronize event). Merging the base into the branch moves the diff range, the earlier finding leaves it, and the model concludes it is a first review. That is exactly what happened:Tier: first reviewon a two-file fix, the original 🔴 unreported, its thread left open.Fix.
.github/scripts/review_gate.pyreads the pull request's comment history before the model runs, classifiesfirst review/re-review/unknownfrom it, and passesREVIEW_TIER,REVIEW_REASON,PREVIOUS_HEAD_SHA,PREVIOUS_BASE_SHA. The skill follows the tier as an input and says so in the summary. History is read from the**Base:** … **Head:** …line the skill already writes; unavailable (API failure) and incomplete (unparseable record) are distinct states, neither of which becomes a first review or discards an earlier finding. A base that advanced isbase_advanced, notno change:BASE...pr-headmeans something different once the base moves.C. Verified review-thread resolution (P0)
Root cause. One sentence ("resolve the threads you judge fixed") with no procedure, no thread lookup, and no requirement that GitHub confirm it. The canary verified its fix and left the thread open — and this repo's branch protection requires conversation resolution.
Fix. Four explicit states — reported → fix observed → fix verified → thread resolved. Threads are fetched via GraphQL, matched by comment content and
path, and only whenauthor.loginisclaude/claude[bot]: another reviewer's thread, a human's or another bot's is never resolved.resolveReviewThreadmust returnisResolved: truebefore the word "resolved" may be written; a refused mutation is reported asfix verified, thread open (…).Every write is narrowly scoped to this. No
permissions:change: the action authenticates as the Claude GitHub App via OIDC, and that installation — not the job's read-onlyGITHUB_TOKEN— is the identity that already posts comments.test_the_review_holds_a_read_only_token_and_cannot_mergestill passes unchanged.D. No-op pushes (P1)
Root cause.
synchronizefires on any push, including an empty commit. The canary's empty commit bought a 15-turn, $0.234 run that posted nothing.Fix. The gate compares the pushed tree with the head the last review recorded (
git diff --quiet), and skips before the action mints a token. No path allow list — a change toPLAN/V2/,.github/or an instruction file is reviewable here, and an extension allow list is precisely how a governance change gets skipped as a docs change. Merge commits, rebases and force pushes reduce to a tree comparison; a commit that cannot be fetched iscomparison_unavailable, which reviews rather than skips. A skipped run is not silent:no_reviewable_changesin the outputs, a step summary naming the sha it compared, and a::notice::. A crashed gate fails open to a review.E. Reachability before severity (P0)
Root cause. The canary's finding was correct — a documented fail-closed contract was flipped — and it described the impact as a live cross-tenant leak.
row_visibleparses the same path and refuses the read first, andapp_user.idis constrained to a valid segment, so the exploit did not exist. The severity turned out right for the wrong reason.Fix.
REVIEW.mdgains "Reachability before severity": trace fromEngine.call, name the guards upstream, classify reachable today / latent / unverified, then calibrate; every such finding carries a**Reachability:**line. This does not weaken reporting: a latent violation of a stated guarantee is still 🔴 — the guard above it is a second line of defence, not a licence to delete the first. The 🔴/🟡 table gains the same distinction.2. Files
.github/scripts/review_gate.py(new)gh/gitshell.github/workflows/claude-code-review.ymlREPOSITORY, tier and previous shas into the prompt; the expensive steps conditional on the decision.github/workflows/claude.ymlREPOSITORY,BASE_SHA, and an instruction to classify itself (a comment event carries no tier).claude/skills/review-pr/SKILL.mdREVIEW.md.github/workflows/README.mdcustom-memory-engine/tests/test_review_gate.py(new)custom-memory-engine/tests/test_claude_integration.py3. Tests
test_review_gate.py— first review; normal push; base advanced; base merge into the branch; fix-only push; regression introduced while fixing; identical tree; unchanged head; unreadable history; unparseable record; missing commits;opened/ready_for_review/reopened; newest-record-wins; duplicated summaries; both bot spellings; a human quoting the heading; the empty-commit vs one-byte-edit comparison in a real repo; PLAN/workflow/instruction changes all counting as reviewable; the output contract; the crashed gate; and the summary-format seam (the gate parses exactly what the skill writes).test_claude_integration.py— identity passed and un-inventable; the old<owner>/<repo>placeholder gone; tier passed not inferred;resolveReviewThreadnamed, matched by content and author, confirmed byisResolved, failure reported, four states distinct; reachability required in both documents without weakening severity; the gate step present and ordered before the review step; no path filter; and the skill's own documented link check run against a body containingHussainAlmutawa/ChronaMind(it flags it).Local: 4095 passed / 21 skipped (baseline 4062/21 → +33 new tests, nothing removed or skipped). Root suite 15 passed. ruff, ruff format, pyright (engine and root), actionlint,
md5sum -c validation/document_hashes.txt, mechanical checks, statistics, G-TS gate,check_plan_refs,audit_status --check,plan_coverage --check, andcheck_evidence(96 refs, 0 unresolved) all clean.Verified against the real thing, read-only: the gate replayed on PR #231 returns
skip / no_reviewable_changes / re-reviewfor the actual no-op commit (the run that cost 15 turns),review / base_advancedwhen the base is moved to the old one, andskipfor the exact reviewed head. The skill's GraphQL query returns the real thread (PRRT_…, authorclaude), and the mutation with a synthetic id returnsNOT_FOUNDrather than a parse error — the schema is right.4. Before / after
<owner>/<repo>filled from memory → cited another repositoryREPOSITORY, checked before posting, omitted if unknownfirst review, earlier findings lostre-reviewfrom history,base_advancedisResolved: true**Reachability:** latent: <the guard>; still 🔴5. Limitations
test_the_instructions_stay_small_enough_to_read_on_every_review, unchanged). Further instruction work has to remove something.pull_requests: writeon the Claude GitHub App installation. That is an app setting, not a repository file; if it is missing the review reportsresolution failed (…)and leaves the thread open. It can only be proven on a PR with a fixed finding — see below.REVIEW.md/skill (a new repository, or this PR before merge), the workflow restores them from the PR head and says so, as before.**Head:**line in the summary. Unparseable → review, never skip..github/workflows/README.md.6. Canary detection preserved
The instructions that produced the canary's detection are untouched: the same evidence bar, the same "trace, do not skim" steps, the same severity table, the same file-reads.
REVIEW.mdgrew by one section (11,767 of its 15,000-byte ceiling) and the skill by a net ~5.6 KB, both under the existing ceilings, andcheck_evidenceconfirms every citation still resolves. The 33 new tests assert the workflow still passes the skill, still restores the standard from the base branch, still holds a read-only token and still denies push/merge/approve.