Skip to content

fix(claude): a review that posts nothing cannot be told from one that never ran - #234

Merged
elewashy merged 1 commit into
mainfrom
fix/review-post-visibility
Nov 21, 2026
Merged

elewashy merged 1 commit into
mainfrom
fix/review-post-visibility

Conversation

@elewashy

Copy link
Copy Markdown
Collaborator

Refs #73

Follow-up to #233, from live evidence rather than inspection.

What happened

After #233 merged, I triggered the on-demand reviewer on it (@claude review) to verify the path end to end — that path had been broken since it was written (git init leaves no remote, so git fetch origin failed with exit 128; #233 fixed it to use actions/checkout, and its first live run at 10:42Z then succeeded: 17 turns, $0.328).

It posted nothing: no summary comment, no inline comment, no review object. The pull request had already been merged. A run that reads a pull request and posts nothing cannot be distinguished from a run that never happened — the exact failure mode this integration exists to remove, in the state where a reader most needs telling.

Change

SKILL.md §7: always post the summary, whatever the state — say what you read and that it is no longer actionable, rather than leaving the reader to guess.

Consolidated elsewhere in the same file to stay under the 20,000-byte ceiling (now 19,992): same requirements, tighter wording. No review requirement was removed.

Verification

  • 4107 passed / 21 skipped (was 4106 + this PR's one new test); root suite 15; ruff, format, pyright (both projects), actionlint, document hashes, and check_evidence (97 refs, 0 unresolved) clean.
  • New test pins the rule and its rationale: test_the_review_always_posts_a_summary_whatever_the_pull_request_state.
  • Not verified: that a live review now posts on a merged PR — that needs this on main plus another on-demand run. The rule is instruction-level; its enforcement is the model's.

… never ran

Live verification of the on-demand path after #233 merged: the job now completes (its checkout was broken until
`f44ce3f`'s successor), the prompt carried `REPOSITORY`, `BASE_SHA` and `HEAD_SHA`, the model ran 17 turns for $0.328 —
and it posted nothing at all: no summary, no inline comment, no review. The pull request had already been merged, which
is exactly when a reader most needs telling what was reviewed.

The skill now requires the summary unconditionally: say what you read, and that it is no longer actionable. That closes
the last silent path in a procedure whose whole purpose is that a review cannot be mistaken for silence.

`SKILL.md` was consolidated again to stay under its 20,000-byte ceiling (19,992): the same requirements, restated more
tightly where two sentences said one thing.

Refs #73
@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Code review

Base: 18a1c7620c74d8c65f3229c2beee825510b51a2b · Head: 93197741c84ef78038cd13dc1428101f9c13b733 · Reviewed: 2 files, +43/-30 · Tier: first review (no earlier review)

What the change does

Adds an unconditional closing rule to .claude/skills/review-pr/SKILL.md §7 (always post a summary, even when the pull request is merged or closed) and tightens wording elsewhere in that file to stay under the 20,000-byte ceiling. Adds one test in custom-memory-engine/tests/test_claude_integration.py that pins the rule's three phrases.

Findings

No blocking findings.

Instruction change reviewed as the change (base SKILL.md read from origin/main, not from the PR). It does not weaken a check. Two wording trims are worth knowing about, neither blocking:

  • 🟡 §7 no longer says "One committable suggestion block, only when applying it fixes the issue completely". The remaining "Never post a committable suggestion for a multi-file change…" covers the main case, but the positive rule is gone, contrary to the PR body's "no review requirement was removed".
  • 🟡 §5.4 dropped "or live impact" (now "never assert a leak you did not trace"). The summary-shape line still says to claim no unsupported impact, so the effect is small.

Verified

  • Read the full diff against the base file. Every other change is rewording; the requirements, the §6 tier rules and the **Reachability:** rule are intact.
  • SKILL.md is 19,961 bytes by wc -c, under the 20,000 ceiling asserted at test_claude_integration.py:325. The PR body says 19,992; I did not reconcile the difference.
  • The new rule does not conflict with the existing "stop" paths in §2, §6.1 and §7; each already says to write something first.
  • The three regexes in the new test match the added wording, including the \s+ across the line break in "posts nothing cannot be told / from one that never ran" (read, not executed).

Not verified

  • I did not run the test suite (the shell blocked compound commands). CI should decide.
  • Whether a live on-demand review of a merged pull request now posts. This is instruction-level, as the PR says, and needs another run after this reaches main.
  • The new test pins wording only; it cannot show the model follows the rule. I did not run it against the parent commit.

Earlier findings

None: first review.

@elewashy
elewashy merged commit 615ad97 into main Nov 21, 2026
8 checks passed
@elewashy
elewashy deleted the fix/review-post-visibility branch November 21, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant