Skip to content

fix(claude): the reviewer's identity, its tier, and a resolution it must verify - #233

Merged
elewashy merged 5 commits into
mainfrom
fix/claude-review-hardening
Nov 21, 2026
Merged

elewashy merged 5 commits into
mainfrom
fix/claude-review-hardening

Conversation

@elewashy

Copy link
Copy Markdown
Collaborator

Refs #73 · do not merge: this is a review of the reviewer, and merging it is the owner's decision.

Fixes the four things the canary validation showed the automated review getting wrong, plus the reasoning weakness it exposed. Every improvement is decided before the model runs, or verified after GitHub answers — none of them is "ask the prompt to be more careful".

1. Root cause and fix, per item

A. Identity and link integrity (P0)

Root cause. The skill gave the link as https://github.com/<owner>/<repo>/blob/<sha>/<path>#L<n> — a placeholder, and no canonical identity was passed anywhere in the invocation. A model filling <owner>/<repo> fills it from memory; the canary's inline comment cited HussainAlmutawa/ChronaMind, then corrected itself in the summary. The summary was right, the link was not, and the link is what a reader follows.

Fix. REPOSITORY=${{ github.repository }} is passed by both review workflows. Skill §1 states it is the only source of any URL, forbids writing an owner/name/branch/sha/path from memory, requires omitting the link (link omitted: identity unavailable) when the identity or sha cannot be established, and gives a pre-post check (grep -o 'github\.com/[^/]*/[^/]*' must return exactly one value, ours).

B. Explicit re-review state (P0)

Root cause. The skill inferred its tier from the diff (If you are reviewing a later commit… a synchronize event). Merging the base into the branch moves the diff range, the earlier finding leaves it, and the model concludes it is a first review. That is exactly what happened: Tier: first review on a two-file fix, the original 🔴 unreported, its thread left open.

Fix. .github/scripts/review_gate.py reads the pull request's comment history before the model runs, classifies first review / re-review / unknown from it, and passes REVIEW_TIER, REVIEW_REASON, PREVIOUS_HEAD_SHA, PREVIOUS_BASE_SHA. The skill follows the tier as an input and says so in the summary. History is read from the **Base:** … **Head:** … line the skill already writes; unavailable (API failure) and incomplete (unparseable record) are distinct states, neither of which becomes a first review or discards an earlier finding. A base that advanced is base_advanced, not no change: BASE...pr-head means something different once the base moves.

C. Verified review-thread resolution (P0)

Root cause. One sentence ("resolve the threads you judge fixed") with no procedure, no thread lookup, and no requirement that GitHub confirm it. The canary verified its fix and left the thread open — and this repo's branch protection requires conversation resolution.

Fix. Four explicit states — reported → fix observed → fix verified → thread resolved. Threads are fetched via GraphQL, matched by comment content and path, and only when author.login is claude/claude[bot]: another reviewer's thread, a human's or another bot's is never resolved. resolveReviewThread must return isResolved: true before the word "resolved" may be written; a refused mutation is reported as fix verified, thread open (…).

Every write is narrowly scoped to this. No permissions: change: the action authenticates as the Claude GitHub App via OIDC, and that installation — not the job's read-only GITHUB_TOKEN — is the identity that already posts comments. test_the_review_holds_a_read_only_token_and_cannot_merge still passes unchanged.

D. No-op pushes (P1)

Root cause. synchronize fires on any push, including an empty commit. The canary's empty commit bought a 15-turn, $0.234 run that posted nothing.

Fix. The gate compares the pushed tree with the head the last review recorded (git diff --quiet), and skips before the action mints a token. No path allow list — a change to PLAN/V2/, .github/ or an instruction file is reviewable here, and an extension allow list is precisely how a governance change gets skipped as a docs change. Merge commits, rebases and force pushes reduce to a tree comparison; a commit that cannot be fetched is comparison_unavailable, which reviews rather than skips. A skipped run is not silent: no_reviewable_changes in the outputs, a step summary naming the sha it compared, and a ::notice::. A crashed gate fails open to a review.

E. Reachability before severity (P0)

Root cause. The canary's finding was correct — a documented fail-closed contract was flipped — and it described the impact as a live cross-tenant leak. row_visible parses the same path and refuses the read first, and app_user.id is constrained to a valid segment, so the exploit did not exist. The severity turned out right for the wrong reason.

Fix. REVIEW.md gains "Reachability before severity": trace from Engine.call, name the guards upstream, classify reachable today / latent / unverified, then calibrate; every such finding carries a **Reachability:** line. This does not weaken reporting: a latent violation of a stated guarantee is still 🔴 — the guard above it is a second line of defence, not a licence to delete the first. The 🔴/🟡 table gains the same distinction.

2. Files

File Why
.github/scripts/review_gate.py (new) tier + no-op decision, stdlib only, pure functions plus a thin gh/git shell
.github/workflows/claude-code-review.yml gate step; REPOSITORY, tier and previous shas into the prompt; the expensive steps conditional on the decision
.github/workflows/claude.yml REPOSITORY, BASE_SHA, and an instruction to classify itself (a comment event carries no tier)
.claude/skills/review-pr/SKILL.md §1 identity, §5 reachability, §6 tier-driven re-review and verified resolution
REVIEW.md reachability standard with the worked canary example; re-review tier note
.github/workflows/README.md the four mechanisms, the one write, and how a skip is reported
custom-memory-engine/tests/test_review_gate.py (new) 27 tests over the gate
custom-memory-engine/tests/test_claude_integration.py 6 tests holding the workflow and the instructions to the behaviour

3. Tests

test_review_gate.py — first review; normal push; base advanced; base merge into the branch; fix-only push; regression introduced while fixing; identical tree; unchanged head; unreadable history; unparseable record; missing commits; opened/ready_for_review/reopened; newest-record-wins; duplicated summaries; both bot spellings; a human quoting the heading; the empty-commit vs one-byte-edit comparison in a real repo; PLAN/workflow/instruction changes all counting as reviewable; the output contract; the crashed gate; and the summary-format seam (the gate parses exactly what the skill writes).

test_claude_integration.py — identity passed and un-inventable; the old <owner>/<repo> placeholder gone; tier passed not inferred; resolveReviewThread named, matched by content and author, confirmed by isResolved, failure reported, four states distinct; reachability required in both documents without weakening severity; the gate step present and ordered before the review step; no path filter; and the skill's own documented link check run against a body containing HussainAlmutawa/ChronaMind (it flags it).

Local: 4095 passed / 21 skipped (baseline 4062/21 → +33 new tests, nothing removed or skipped). Root suite 15 passed. ruff, ruff format, pyright (engine and root), actionlint, md5sum -c validation/document_hashes.txt, mechanical checks, statistics, G-TS gate, check_plan_refs, audit_status --check, plan_coverage --check, and check_evidence (96 refs, 0 unresolved) all clean.

Verified against the real thing, read-only: the gate replayed on PR #231 returns skip / no_reviewable_changes / re-review for the actual no-op commit (the run that cost 15 turns), review / base_advanced when the base is moved to the old one, and skip for the exact reviewed head. The skill's GraphQL query returns the real thread (PRRT_…, author claude), and the mutation with a synthetic id returns NOT_FOUND rather than a parse error — the schema is right.

4. Before / after

Before After
A link <owner>/<repo> filled from memory → cited another repository from REPOSITORY, checked before posting, omitted if unknown
Tier after a base move first review, earlier findings lost re-review from history, base_advanced
A fixed finding thread left open resolved only after a verified fix and isResolved: true
Empty commit full run, no output skipped before the token, with a reason
A violation described as a live leak **Reachability:** latent: <the guard>; still 🔴

5. Limitations

  • The skill is at 19,973 of its 20,000-byte ceiling (a control in test_the_instructions_stay_small_enough_to_read_on_every_review, unchanged). Further instruction work has to remove something.
  • Thread resolution needs pull_requests: write on the Claude GitHub App installation. That is an app setting, not a repository file; if it is missing the review reports resolution failed (…) and leaves the thread open. It can only be proven on a PR with a fixed finding — see below.
  • If the base branch contains no REVIEW.md/skill (a new repository, or this PR before merge), the workflow restores them from the PR head and says so, as before.
  • The gate cannot see a review that never recorded a head; the baseline is the **Head:** line in the summary. Unparseable → review, never skip.
  • Not verified live: no canary was run for this change. Detection is unchanged by construction (same skill, same standard, same scope; only the identity, tier and resolution mechanics moved), but that is an argument, not evidence. The next planted-defect PR should confirm it, per .github/workflows/README.md.

6. Canary detection preserved

The instructions that produced the canary's detection are untouched: the same evidence bar, the same "trace, do not skim" steps, the same severity table, the same file-reads. REVIEW.md grew by one section (11,767 of its 15,000-byte ceiling) and the skill by a net ~5.6 KB, both under the existing ceilings, and check_evidence confirms every citation still resolves. The 33 new tests assert the workflow still passes the skill, still restores the standard from the base branch, still holds a read-only token and still denies push/merge/approve.

…ust verify

The canary (a real defect in a throwaway PR, `.github/workflows/README.md`) found the reviewer reasoning well and
getting four mechanical things wrong. Each is fixed where it is decided, not by asking the prompt to be more careful.

Identity. The skill said `https://github.com/<owner>/<repo>/…`, a placeholder, so the owner was filled from memory and
one review cited `HussainAlmutawa/ChronaMind`. `REPOSITORY` is now passed by both review workflows, is the only source
of a URL, and an identity that cannot be established drops the link instead of inventing one.

Tier. `review-pr` inferred "am I a re-review?" from the diff. When a base branch advances, the diff range moves, the
earlier findings leave it, and a second review calls itself a first — which is what happened: it re-reported nothing
and left its own 🔴 thread open on a defect it had identified. `.github/scripts/review_gate.py` now reads the pull
request's review history before the model runs and passes `REVIEW_TIER`, `PREVIOUS_HEAD_SHA` and `PREVIOUS_BASE_SHA`.

Resolution. The skill said, in one line, to resolve a fixed thread. It now fetches threads by GraphQL, matches them by
comment content and author, verifies the fix in the code, and resolves only when GitHub answers `isResolved: true` —
reporting the failure and leaving the thread open otherwise. No `permissions:` change: the action authenticates as the
Claude GitHub App through OIDC, and that installation is the identity that already posts the comments; the job's
`GITHUB_TOKEN` stays read-only, as `test_the_review_holds_a_read_only_token_and_cannot_merge` requires.

Cost. A push that changes no tree bought a full run and posted nothing. The gate compares the pushed tree with the head
the last review recorded and skips before the action mints a token — no path allow list, because a change to `PLAN/V2/`,
`.github/` or an instruction file is reviewable here. A crashed gate fails open to a review, never to a skip.

Reachability. A finding about a guard now has to trace the path and name the guards upstream before it sets a severity,
and carry a `**Reachability:**` line. The canary's finding was right and its exploit was not: `row_visible` refused the
input first. A latent violation of a stated guarantee is still 🔴 — the analysis is for accuracy, not for silence.

Tests: 33 new. `custom-memory-engine/tests/test_review_gate.py` covers every classification (first, re-review, base
advanced, base merge, fix-only, regression-while-fixing, unreadable and unparseable history, missing commits), the
summary-format seam, and the gate's own entry point. `tests/test_claude_integration.py` holds the workflow and the
instructions to them, and runs the skill's documented link check against a fabricated owner.

Refs #73
…e is none

CI caught this on the first run: the gate step executed from the base checkout, where `.github/scripts/` does not exist
until this pull request merges — `can't open file .../review_gate.py`, exit 2, a red job on the change that adds the
gate.

The gate decides whether a pull request is reviewed, so it is policy code and is read from the base branch, exactly as
`REVIEW.md` and the skill are: a pull request must not be able to edit its own gate into skipping it. The pull request
that introduces it is the one case with no base copy, so it falls back to `pr-head` and says so in the log — the same
bootstrap the skill's own step already uses. With neither copy present the step does not fail: it emits
`review=true`, `tier=unknown`, `reason=gate_failed`, because a gate that cannot decide is a reason to review, never a
reason to skip.

Pinned by `test_the_gate_comes_from_the_base_branch_and_tolerates_its_own_introduction`.

Refs #73
@elewashy

Copy link
Copy Markdown
Collaborator Author

@chronamind Diagnostic only — change nothing, resolve nothing. I need to know whether the app token this action runs as is authorised to resolve review threads. Run exactly these two commands and paste the raw output of each:

  1. Read-only, on a closed PR, to read viewerCanResolve for the viewer:
gh api graphql -f query='query{repository(owner:"eMinders",name:"ChronaMind"){pullRequest(number:231){reviewThreads(first:10){nodes{id isResolved viewerCanResolve resolvedBy{login}}}}}}'
  1. A mutation aimed at a synthetic thread id that does not exist (it can resolve nothing):
gh api graphql -f query='mutation($t:ID!){resolveReviewThread(input:{threadId:$t}){thread{id isResolved}}}' -F t="PRRT_kwDOAAAAAAAAAAAAAAA"

Then tell me, in one line each: (a) the value of viewerCanResolve, and (b) whether the mutation failed with NOT_FOUND (authorised, but no such thread) or with FORBIDDEN / Resource not accessible by integration (not authorised). Do not comment on anything else and do not touch any real thread.

@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

…reviews the reviewer

Three gaps the first round could not close, and one bug found while closing them.

**Resolution needs `Contents: write`, not `pull_requests: write`.** `resolveReviewThread` is gated on repository write
for a GitHub App installation token; a token holding only `pull_requests: write` is refused with `Resource not
accessible by integration` (`github/gh-aw#35726`, `oven-sh/bun#36959`, `zuke-build/zuke#634`). The action's default token
request does ask for `contents: write` too, so this installation should already qualify — but "should" is not grounds for
writing "resolved", so the query now reads `viewerCanResolve` and the skill refuses to attempt the mutation when it is
false, reporting `thread open: not permitted (no Contents: write)` instead of a silent skip. End-to-end resolution is
still unproven here: it needs a review of a real fix, which needs this change on `main`.

**A workflow change is never auto-reviewed, and that now says so.** The action's OIDC exchange refuses a workflow file
that differs from the default branch's, so `claude-review` goes green with the review skipped — the failure shape where
green means "not reviewed". The gate detects the case from the pull request's file list and prints a warning plus a
step-summary notice naming the workflow and the path that works: a maintainer's `@claude review`, which runs on
`issue_comment` and therefore on the default branch's workflow. No permission is widened to dodge the rule.

**No-op detection, proven end to end.** The gate is now exercised as the workflow runs it — a real git repository, a stub
`gh` on `PATH`, the script as a subprocess — so the "identical tree, do not review" answer and the output file the
review step is gated on are both tested, not just `classify()`. A pull request that changes the tree still reviews, an
unreadable file list does not block, and an unparseable history still reviews.

**A bug the diagnostic found: the assistant cancelled itself.** The action posts an acknowledgement comment as it
starts; that comment is an `issue_comment` event, joined the same concurrency group, and cancelled the run that had just
posted it ("Claude Code is working…" 10:07:11Z, "The operation was canceled" 10:07:27Z, #233). A run started by a bot
now lands in a group keyed on its own run id, so only a human's comment can supersede a human's request.

Also: a controlled-canary procedure in the documentation (one CI-invisible defect, expected finding written first,
never touching the workflow or the standard, closed unmerged) so the next validation is repeatable; and `SKILL.md` was
consolidated — its cap sentence and "do not report" list now point at `REVIEW.md` instead of repeating it — to stay
under its 20,000-byte ceiling at 19,961. The gate is pinned to Python 3.12, because Ruff with the root's 3.14 target
rewrote an `except (A, B, C)` into PEP 758 syntax that is a SyntaxError on the runner.

Refs #73
The notice reads to a maintainer under a green check, so it says "edits `a` and `b`, so …" rather than a comma
splice. The helper is unit-tested for one, two and three items.

Refs #73
@elewashy

Copy link
Copy Markdown
Collaborator Author

@claude review

…as no remote

`@claude review` is the documented way to review a pull request that edits the review workflow — the automatic job
cannot, because the action's token exchange refuses a workflow file that differs from the default branch's. That path
was broken from the day it was written: its checkout step ran `git init --quiet pr-head`, which creates a repository
with no remote, and then `git -C pr-head fetch origin "$HEAD_SHA"` — `fatal: Could not read from remote repository`,
exit 128, before the review step. Its first live attempt was #233 at 10:27Z, and it failed exactly there.

It now uses `actions/checkout`, as the automatic review does, with the head taken from the repository the pull request
came from (`head.repo.full_name`, newly resolved in the API step) so a fork still works when a maintainer asks for one.
Authentication is the action's, and `persist-credentials: false` is kept.

This is what makes an independent review of a workflow change possible at all; without it, the fallback documented for
that gap is a check that goes green having reviewed nothing.

Refs #73
@elewashy
elewashy merged commit 18a1c76 into main Nov 21, 2026
8 checks passed
@elewashy
elewashy deleted the fix/claude-review-hardening branch November 21, 2026 10:38
@elewashy

Copy link
Copy Markdown
Collaborator Author

@claude review

elewashy added a commit that referenced this pull request Nov 21, 2026
… never ran (#234)

Live verification of the on-demand path after #233 merged: the job now completes (its checkout was broken until
`f44ce3f`'s successor), the prompt carried `REPOSITORY`, `BASE_SHA` and `HEAD_SHA`, the model ran 17 turns for $0.328 —
and it posted nothing at all: no summary, no inline comment, no review. The pull request had already been merged, which
is exactly when a reader most needs telling what was reviewed.

The skill now requires the summary unconditionally: say what you read, and that it is no longer actionable. That closes
the last silent path in a procedure whose whole purpose is that a review cannot be mistaken for silence.

`SKILL.md` was consolidated again to stay under its 20,000-byte ceiling (19,992): the same requirements, restated more
tightly where two sentences said one thing.

Refs #73
elewashy added a commit that referenced this pull request Nov 21, 2026
…ting (#236)

* fix(claude): the gate reads GitHub's timeline, not the model's formatting

First live run of the finished reviewer, on #235: it found the planted defect at 🔴 and posted a correct finding, then
wrote its summary as one flattened paragraph — `Code review. Base <sha>, Head <sha>. …`, no heading, no bold fields.
The gate parses the heading and the bold fields, so it reported `no_earlier_review` and `first review` (the #231
failure, reintroduced by a formatting difference), and no empty push could ever be skipped. Both decisions the gate
exists to make were dead, and every unit test was green.

A model's prose is not a place to keep a policy decision. Three changes, in that order:

1. The summary is parsed loosely as well as strictly: `Base <sha>` / `Head <sha>` inside a sentence is a record, and a
   comment without the heading still counts when it names both commits. A full 40-character sha is required, so a
   finding quoting a 7-character prefix cannot become a baseline.
2. `GET /issues/{n}/timeline` is the fallback: `committed` events carry every pushed sha, `reviewed` events carry the
   `commit_id` a review was submitted against. The reviewer does not write either, so no wording can break the tier.
3. A timeline-only baseline never skips: the timeline has no base commit, a moved base changes what the pull request
   means, and the answer to an unknown is a review.

Verified against the real pull request, read-only: the gate now reports `skip: no_reviewable_changes (tier: re-review)`
for #235 where it previously reported `review: no_earlier_review (tier: first review)`.

The skill now says how to post the summary — a file and `--body-file`, because a one-line `--body` string is what
arrives flattened.

Refs #73, #231, #233

* fix(claude): an unreadable timeline is unknown history, and the skill stops promising a test run

Both from the review of this pull request, which found a real defect in it.

1. `fetch_history` returned `earlier_review=False` when the summary was unrecognisable *and* the timeline call failed,
   so a transient API error re-created the #231 failure: a re-review announcing itself as a first review. It now reports
   an unreadable history, which `classify` turns into an unknown tier and a review. Pinned by a test that makes the
   timeline call fail.

2. The skill told the reviewer to "reproduce it if you can … `uv run --no-sync pytest`" and to reconstruct a parent
   with `git stash`/`git checkout`. None of those are in the workflow's allow list, and the review job has no database,
   so the instruction asked for checks the reviewer cannot make — an invitation to claim verification it did not
   perform (it did not, and said so under "Not verified"). Reading is what this job can do; the instruction now says
   that, and says a refused command is not a check.

Engine 4114 passed / 21 skipped; root 15; ruff, format, pyright, actionlint, check_evidence (105 refs) clean.

Refs #236
elewashy added a commit that referenced this pull request Nov 21, 2026
…ds (#237)

Measured on #236, not inferred: the reviewer verified its own 🟡 finding, tried to resolve the thread, and reported
`thread open: not permitted (no Contents: write)`. `resolveReviewThread` is gated on the token's **Contents: write**;
`pull_requests: write` alone is refused with `Resource not accessible by integration` (github/gh-aw#35726,
oven-sh/bun#36959 and #40468, all with live logs).

That is not cosmetic here: branch protection has `required_conversation_resolution` enabled, so an unresolved thread the
bot left behind **blocks the merge**. #236 could not be merged until a human resolved it.

So both review jobs now grant `contents: write` — one permission, for one GraphQL mutation — and the compensating
controls are what keep it narrow: `persist-credentials: false` on every checkout, `git push`/`commit`/`merge`,
`gh pr merge/close/review/edit` denied, and every REST write verb (`gh api -X …`, `gh api --method …`) denied, leaving
GraphQL as the only mutation path available to the job.

The reviewer still checks `viewerCanResolve` first and reports `not permitted` rather than assuming; if an installation
cannot grant Contents: write, the human fallback is unchanged and documented.

Refs #236, #233
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant