Repository navigation
ci: add comment-cop action to flag multi-line comments in claude PRs - #35534
Conversation
Scans the diff of claude-labeled PRs for multi-line comment blocks added in src/ that don't carry a SAFETY: marker, and leaves a review comment on each group asking for deletion. Runs entirely via the GitHub API (actions/github-script, no checkout). Review comments carry an HTML marker keyed on path + content hash so reruns on synchronize/labeled don't post duplicates.
|
Warning Review limit reached
Next review available in: 25 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
Comment |
There was a problem hiding this comment.
Beyond the inline nit: checked the pull_request_target + pull-requests: write combo — no checkout of PR head and no PR-controlled data reaches shell/eval, so no code-execution or injection surface. Also confirmed actions/github-script is SHA-pinned. Leaving the policy call (auto-nagging every 2+ line comment) and wording to a human.
Extended reasoning...
This adds a new pull_request_target workflow with write permissions — the canonical security-sensitive pattern. I verified it never checks out or executes PR code (pure API reads via pulls.listFiles / listReviewComments, then createReview), and PR-controlled strings only flow into review-comment bodies, not into any evaluated context. That rules out the usual pwn-request class. The one inline nit (dead ---/+++ guards causing line-number drift on removed lines like --i;) is narrow. Not approving because this establishes new automated policy on contributor PRs (message wording, 2-line threshold, SAFETY-only exemption), which is a maintainer decision rather than a code-correctness one.
- flag all multi-line comments including /// and //! doc comments - remove per-run comment cap - drop paths: filter so the labeled event fires reliably after auto-label - drop SAFETY: mention from the posted comment body - drop dead +++/--- guards that misfire on content like --i; - post individual review line comments instead of a review summary - auto-resolve prior comment-cop threads whose block is gone from the diff
| if: > | ||
| github.repository == 'oven-sh/bun' && | ||
| contains(github.event.pull_request.labels.*.name, 'claude') && | ||
| (github.event.action != 'labeled' || github.event.label.name == 'claude') |
There was a problem hiding this comment.
🔴 The labeled trigger — described as covering "the normal case where auto-label-claude-prs.yml applies the label after open" — never fires: that workflow calls addLabels with the default GITHUB_TOKEN, and GitHub suppresses workflow runs for events triggered by GITHUB_TOKEN. On opened the label isn't in the event payload yet, so contains(labels.*.name, 'claude') is false and the job is skipped — meaning comment-cop first runs on the next push and never runs at all on one-shot PRs merged without a follow-up commit. Gate on github.event.pull_request.user.login == 'robobun' || contains(github.event.pull_request.body, '🤖 Generated with') (matching auto-label's own predicate) so opened fires directly, or have auto-label use a PAT/app token.
Extended reasoning...
What the bug is
The workflow gates on the claude label being present in the event payload:
on:
pull_request_target:
types: [opened, synchronize, reopened, labeled]
jobs:
comment-cop:
if: >
github.repository == 'oven-sh/bun' &&
contains(github.event.pull_request.labels.*.name, 'claude') &&
(github.event.action != 'labeled' || github.event.label.name == 'claude')The PR description states the labeled trigger "covers the normal case where auto-label-claude-prs.yml applies the label after open." But auto-label-claude-prs.yml adds the label via actions/github-script@v7 with no github-token: override — it authenticates with the default GITHUB_TOKEN:
- uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
with:
script: |
github.rest.issues.addLabels({ ..., labels: ['claude'] });GitHub's documented recursion guard: "When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of workflow_dispatch and repository_dispatch, will not create a new workflow run." So the pull_request_target: labeled event emitted by that addLabels call is suppressed and never triggers comment-cop.
Step-by-step proof — the normal robobun flow
- robobun opens a PR. GitHub fires
pull_request[opened](→ auto-label) andpull_request_target[opened](→ comment-cop) from the same event snapshot. - comment-cop on
opened:github.event.pull_request.labelsis the payload snapshot captured at open time — before auto-label has run. It does not containclaude, socontains(github.event.pull_request.labels.*.name, 'claude')is false → job skipped. - auto-label on
opened:user.login == 'robobun'→ callsaddLabels(['claude'])viaGITHUB_TOKEN. GitHub records alabeledevent but does not create a workflow run for it (recursion guard). - comment-cop on
labeled: never fires. The(github.event.action != 'labeled' || github.event.label.name == 'claude')clause and thelabeledentry intypes:are dead in this flow. - Next
synchronize: on the first subsequent push, the payload snapshot now includes theclaudelabel → comment-cop finally runs.
Net: the initial diff of every robobun PR is silently unscanned. For one-shot PRs (opened → reviewed → merged with no follow-up commit — common for small Claude fixes), comment-cop never runs at all.
Why nothing else prevents it
- There is no race that saves this: even if auto-label completes before comment-cop's runner is scheduled,
github.event.pull_request.labelsis the event payload, not a live API read — it reflects labels at the instant the PR was opened. - The only path that would fire
labeledfor real is a human (or a PAT-authenticated bot) applying the label manually. That is not the "normal case" the PR description relies on. reopenedrequires a close/reopen cycle;synchronizerequires a new push. Neither covers the initial commit.
Impact
The feature's stated purpose — flagging multi-line comment blocks in Claude PRs — does not execute on the commit it most needs to scan (the initial diff). REVIEW.md: "Every line you add must be demonstrably live … manually exercise the failure path of any checker whose purpose is to fail." The primary designed trigger path here is provably dead.
Fix
Simplest: replace the label gate with the same predicate auto-label uses, so opened fires directly and the labeled dependency disappears:
if: >
github.repository == 'oven-sh/bun' &&
(github.event.pull_request.user.login == 'robobun' ||
contains(github.event.pull_request.body, '🤖 Generated with'))(then drop labeled from types:). Alternatively, keep the label gate but have auto-label-claude-prs.yml use a PAT/app token so its labeled event triggers downstream workflows, or add a workflow_run trigger on auto-label-claude-prs completion.
| seenKeys.add(key); | ||
| if (!t.isResolved && !presentKeys.has(key)) toResolve.push(t.id); |
There was a problem hiding this comment.
🟡 Auto-resolve can falsely close threads when GitHub omits patch for a large file: line 101 skips files with no f.patch, so their groups never enter presentKeys, and !presentKeys.has(key) here then resolves every comment-cop thread on that file even though the flagged blocks are still present. Consider tracking a Set of paths where f.patch was missing and skipping resolution for keys whose path prefix (key.slice(0, key.lastIndexOf(':'))) is in that set.
Extended reasoning...
What the bug is
The auto-resolve pass conflates two distinct states — "this file's diff contains no flagged comment blocks" and "this file's diff was too large for GitHub to return a patch" — and treats both as "the flagged block is gone, resolve the thread."
Code path
presentKeys (line 106) is derived from groups, which is populated only from files that survive all four filters at lines 98–101. The last of those is:
if (!f.patch) continue;GitHub's pulls.listFiles REST endpoint omits the patch field on a file entry when that file's diff is large (and truncates the file list entirely at 3000 files). When that happens, f.patch is undefined, the file is skipped, and none of its groups are added to presentKeys.
Then at line 140:
if (!t.isResolved && !presentKeys.has(key)) toResolve.push(t.id);Every existing comment-cop thread whose key isn't in presentKeys is queued for resolveReviewThread. There is no check for "the file this key belongs to was skipped for having no patch," so the thread is resolved even though the flagged comment block is still in the diff — the action just couldn't scan it.
Why nothing else prevents it
The key format is ${path}:${sha12}, so the path is recoverable from the marker, but nothing consults it. The only inputs to the resolution decision are t.isResolved and set membership in presentKeys; there is no separate record of which files were seen-but-unscannable vs. genuinely absent from the diff.
Step-by-step proof
- Push 1: a claude-labeled PR touches
src/foo.rsand adds a 3-line comment block.listFilesreturns{ filename: 'src/foo.rs', patch: '@@ … +// a\n+// b\n+// c …' }. The scanner emits one group with keysrc/foo.rs:abc123def456, adds it topresentKeys, and posts a review comment with marker<!-- comment-cop:src/foo.rs:abc123def456 -->. - Push 2: the same PR adds a large generated section elsewhere in
src/foo.rs(or the cumulative diff for that file otherwise crosses GitHub's per-file threshold). The 3-line comment block is untouched.listFilesnow returns{ filename: 'src/foo.rs', status: 'modified', additions: 4200, … }with nopatchfield. - Line 101 evaluates
!f.patch→true→continue.groupsgets nothing forsrc/foo.rs;presentKeysdoes not containsrc/foo.rs:abc123def456. - The GraphQL query returns the thread from step 1 with
isResolved: falseand body containing the marker. Line 136 extractskey = 'src/foo.rs:abc123def456'. - Line 140:
!false && !presentKeys.has('src/foo.rs:abc123def456')→true && true→ thread id pushed totoResolve. resolveReviewThreadmarks the thread resolved. The comment block is still there.
Impact
Low. This is a CI nag helper on claude-labeled PRs, not runtime code. The failure mode is a bot review thread getting collapsed prematurely — non-destructive and reversible (it can be manually unresolved), and a human reviewer still sees the comment block in the diff. The trigger — a src/ file's diff crossing GitHub's per-file patch-omission threshold on a later push after having been flagged on an earlier one — is narrow for focused AI PRs, though not impossible.
Fix
Track indeterminate paths and exclude them from auto-resolve:
const noPatch = new Set();
for (const f of files) {
if (f.status === 'removed') continue;
if (!f.filename.startsWith('src/')) continue;
if (!SRC_EXT.test(f.filename)) continue;
if (!f.patch) { noPatch.add(f.filename); continue; }
for (const g of groupsFromPatch(f.filename, f.patch)) groups.push(g);
}
…
if (!t.isResolved && !presentKeys.has(key)) {
const path = key.slice(0, key.lastIndexOf(':'));
if (!noPatch.has(path)) toResolve.push(t.id);
}Two or three lines; leaves threads on unscannable files alone rather than assuming either way.
…35534) Adds a self-contained GitHub Action that scans `claude`-labeled PRs for multi-line comment blocks added under `src/` and leaves a review line comment on each one asking for it to be deleted (and the thread resolved once done). ### Behavior - Triggers on `pull_request_target` (`opened` / `synchronize` / `reopened` / `labeled`), gated on `github.repository == 'oven-sh/bun'` and the `claude` label being present. The `labeled` trigger covers the normal case where `auto-label-claude-prs.yml` applies the label after open. - Fetches PR files via `pulls.listFiles` and parses the unified diff in `actions/github-script`. No repo checkout. - A group is **2+ consecutive added lines** that are pure comment lines (`//`, `///`, `//!`, `/* ... */`, `* ` continuation) in `.rs` / `.c` / `.cpp` / `.h` / `.ts` / `.js` and friends. Groups containing `SAFETY:` are skipped. - Posts one standalone review line comment per consecutive group via `pulls.createReviewComment` (no review summary). - Each comment carries `<!-- comment-cop:<path>:<sha12> -->` keyed on the group's content hash, so reruns (new pushes, relabels) skip groups that already have a marker. - On each run, any existing comment-cop thread whose flagged block is no longer present in the diff is auto-resolved via `resolveReviewThread`. ### Verification Dry-ran the parser against the five most recent open `claude` PRs via `gh api repos/oven-sh/bun/pulls/<n>/files`: ``` #35531 src/runtime/webcore/Blob.rs:5245-5246 #35529 src/http_jsc/websocket_client.rs:907-910 #35528 src/runtime/webcore/FileSink.rs:917-920 #35526 src/http_jsc/websocket_client.rs:2179-2182 #35532 (clean) ``` Cross-checked `#35531` line numbers against the fetched head ref; `Blob.rs:5245` is the first line of the flagged block. `actionlint` and prettier both pass. <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · build/CI scripts only; test-proof not applicable <!-- robobun:evidence:end -->
Adds a self-contained GitHub Action that scans
claude-labeled PRs for multi-line comment blocks added undersrc/and leaves a review line comment on each one asking for it to be deleted (and the thread resolved once done).Behavior
pull_request_target(opened/synchronize/reopened/labeled), gated ongithub.repository == 'oven-sh/bun'and theclaudelabel being present. Thelabeledtrigger covers the normal case whereauto-label-claude-prs.ymlapplies the label after open.pulls.listFilesand parses the unified diff inactions/github-script. No repo checkout.//,///,//!,/* ... */,*continuation) in.rs/.c/.cpp/.h/.ts/.jsand friends. Groups containingSAFETY:are skipped.pulls.createReviewComment(no review summary).<!-- comment-cop:<path>:<sha12> -->keyed on the group's content hash, so reruns (new pushes, relabels) skip groups that already have a marker.resolveReviewThread.Verification
Dry-ran the parser against the five most recent open
claudePRs viagh api repos/oven-sh/bun/pulls/<n>/files:Cross-checked
#35531line numbers against the fetched head ref;Blob.rs:5245is the first line of the flagged block.actionlintand prettier both pass.no test proof · iteration 0 · build/CI scripts only; test-proof not applicable