Skip to content

ci: add comment-cop action to flag multi-line comments in claude PRs - #35534

Merged
Jarred-Sumner merged 2 commits into
mainfrom
claude/farm/d6d2226d/comment-cop-action
Jul 25, 2026
Merged

Jarred-Sumner merged 2 commits into
mainfrom
claude/farm/d6d2226d/comment-cop-action

Conversation

@robobun

@robobun robobun commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a self-contained GitHub Action that scans claude-labeled PRs for multi-line comment blocks added under src/ and leaves a review line comment on each one asking for it to be deleted (and the thread resolved once done).

Behavior

  • Triggers on pull_request_target (opened / synchronize / reopened / labeled), gated on github.repository == 'oven-sh/bun' and the claude label being present. The labeled trigger covers the normal case where auto-label-claude-prs.yml applies the label after open.
  • Fetches PR files via pulls.listFiles and parses the unified diff in actions/github-script. No repo checkout.
  • A group is 2+ consecutive added lines that are pure comment lines (//, ///, //!, /* ... */, * continuation) in .rs / .c / .cpp / .h / .ts / .js and friends. Groups containing SAFETY: are skipped.
  • Posts one standalone review line comment per consecutive group via pulls.createReviewComment (no review summary).
  • Each comment carries <!-- comment-cop:<path>:<sha12> --> keyed on the group's content hash, so reruns (new pushes, relabels) skip groups that already have a marker.
  • On each run, any existing comment-cop thread whose flagged block is no longer present in the diff is auto-resolved via resolveReviewThread.

Verification

Dry-ran the parser against the five most recent open claude PRs via gh api repos/oven-sh/bun/pulls/<n>/files:

#35531  src/runtime/webcore/Blob.rs:5245-5246
#35529  src/http_jsc/websocket_client.rs:907-910
#35528  src/runtime/webcore/FileSink.rs:917-920
#35526  src/http_jsc/websocket_client.rs:2179-2182
#35532  (clean)

Cross-checked #35531 line numbers against the fetched head ref; Blob.rs:5245 is the first line of the flagged block. actionlint and prettier both pass.


no test proof · iteration 0 · build/CI scripts only; test-proof not applicable

Scans the diff of claude-labeled PRs for multi-line comment blocks
added in src/ that don't carry a SAFETY: marker, and leaves a review
comment on each group asking for deletion.

Runs entirely via the GitHub API (actions/github-script, no checkout).
Review comments carry an HTML marker keyed on path + content hash so
reruns on synchronize/labeled don't post duplicates.
@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 25 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f391de1f-d47a-4b56-95d9-59fa4bd6cd96

📥 Commits

Reviewing files that changed from the base of the PR and between 762cefb and b545253.

📒 Files selected for processing (1)
  • .github/workflows/comment-cop.yml

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 8:10 PM PT - Jul 24th, 2026

@robobun, your commit b545253 is building: #80083

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline nit: checked the pull_request_target + pull-requests: write combo — no checkout of PR head and no PR-controlled data reaches shell/eval, so no code-execution or injection surface. Also confirmed actions/github-script is SHA-pinned. Leaving the policy call (auto-nagging every 2+ line comment) and wording to a human.

Extended reasoning...

This adds a new pull_request_target workflow with write permissions — the canonical security-sensitive pattern. I verified it never checks out or executes PR code (pure API reads via pulls.listFiles / listReviewComments, then createReview), and PR-controlled strings only flow into review-comment bodies, not into any evaluated context. That rules out the usual pwn-request class. The one inline nit (dead ---/+++ guards causing line-number drift on removed lines like --i;) is narrow. Not approving because this establishes new automated policy on contributor PRs (message wording, 2-line threshold, SAFETY-only exemption), which is a maintainer decision rather than a code-correctness one.

Comment thread .github/workflows/comment-cop.yml Resolved
- flag all multi-line comments including /// and //! doc comments
- remove per-run comment cap
- drop paths: filter so the labeled event fires reliably after auto-label
- drop SAFETY: mention from the posted comment body
- drop dead +++/--- guards that misfire on content like --i;
- post individual review line comments instead of a review summary
- auto-resolve prior comment-cop threads whose block is gone from the diff
@Jarred-Sumner
Jarred-Sumner merged commit fdac5e7 into main Jul 25, 2026
4 of 7 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the claude/farm/d6d2226d/comment-cop-action branch July 25, 2026 03:13
Comment on lines +17 to +20
if: >
github.repository == 'oven-sh/bun' &&
contains(github.event.pull_request.labels.*.name, 'claude') &&
(github.event.action != 'labeled' || github.event.label.name == 'claude')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The labeled trigger — described as covering "the normal case where auto-label-claude-prs.yml applies the label after open" — never fires: that workflow calls addLabels with the default GITHUB_TOKEN, and GitHub suppresses workflow runs for events triggered by GITHUB_TOKEN. On opened the label isn't in the event payload yet, so contains(labels.*.name, 'claude') is false and the job is skipped — meaning comment-cop first runs on the next push and never runs at all on one-shot PRs merged without a follow-up commit. Gate on github.event.pull_request.user.login == 'robobun' || contains(github.event.pull_request.body, '🤖 Generated with') (matching auto-label's own predicate) so opened fires directly, or have auto-label use a PAT/app token.

Extended reasoning...

What the bug is

The workflow gates on the claude label being present in the event payload:

on:
  pull_request_target:
    types: [opened, synchronize, reopened, labeled]
jobs:
  comment-cop:
    if: >
      github.repository == 'oven-sh/bun' &&
      contains(github.event.pull_request.labels.*.name, 'claude') &&
      (github.event.action != 'labeled' || github.event.label.name == 'claude')

The PR description states the labeled trigger "covers the normal case where auto-label-claude-prs.yml applies the label after open." But auto-label-claude-prs.yml adds the label via actions/github-script@v7 with no github-token: override — it authenticates with the default GITHUB_TOKEN:

- uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
  with:
    script: |
      github.rest.issues.addLabels({ ..., labels: ['claude'] });

GitHub's documented recursion guard: "When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of workflow_dispatch and repository_dispatch, will not create a new workflow run." So the pull_request_target: labeled event emitted by that addLabels call is suppressed and never triggers comment-cop.

Step-by-step proof — the normal robobun flow

  1. robobun opens a PR. GitHub fires pull_request[opened] (→ auto-label) and pull_request_target[opened] (→ comment-cop) from the same event snapshot.
  2. comment-cop on opened: github.event.pull_request.labels is the payload snapshot captured at open time — before auto-label has run. It does not contain claude, so contains(github.event.pull_request.labels.*.name, 'claude') is false → job skipped.
  3. auto-label on opened: user.login == 'robobun' → calls addLabels(['claude']) via GITHUB_TOKEN. GitHub records a labeled event but does not create a workflow run for it (recursion guard).
  4. comment-cop on labeled: never fires. The (github.event.action != 'labeled' || github.event.label.name == 'claude') clause and the labeled entry in types: are dead in this flow.
  5. Next synchronize: on the first subsequent push, the payload snapshot now includes the claude label → comment-cop finally runs.

Net: the initial diff of every robobun PR is silently unscanned. For one-shot PRs (opened → reviewed → merged with no follow-up commit — common for small Claude fixes), comment-cop never runs at all.

Why nothing else prevents it

  • There is no race that saves this: even if auto-label completes before comment-cop's runner is scheduled, github.event.pull_request.labels is the event payload, not a live API read — it reflects labels at the instant the PR was opened.
  • The only path that would fire labeled for real is a human (or a PAT-authenticated bot) applying the label manually. That is not the "normal case" the PR description relies on.
  • reopened requires a close/reopen cycle; synchronize requires a new push. Neither covers the initial commit.

Impact

The feature's stated purpose — flagging multi-line comment blocks in Claude PRs — does not execute on the commit it most needs to scan (the initial diff). REVIEW.md: "Every line you add must be demonstrably live … manually exercise the failure path of any checker whose purpose is to fail." The primary designed trigger path here is provably dead.

Fix

Simplest: replace the label gate with the same predicate auto-label uses, so opened fires directly and the labeled dependency disappears:

if: >
  github.repository == 'oven-sh/bun' &&
  (github.event.pull_request.user.login == 'robobun' ||
   contains(github.event.pull_request.body, '🤖 Generated with'))

(then drop labeled from types:). Alternatively, keep the label gate but have auto-label-claude-prs.yml use a PAT/app token so its labeled event triggers downstream workflows, or add a workflow_run trigger on auto-label-claude-prs completion.

Comment on lines +139 to +140
seenKeys.add(key);
if (!t.isResolved && !presentKeys.has(key)) toResolve.push(t.id);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Auto-resolve can falsely close threads when GitHub omits patch for a large file: line 101 skips files with no f.patch, so their groups never enter presentKeys, and !presentKeys.has(key) here then resolves every comment-cop thread on that file even though the flagged blocks are still present. Consider tracking a Set of paths where f.patch was missing and skipping resolution for keys whose path prefix (key.slice(0, key.lastIndexOf(':'))) is in that set.

Extended reasoning...

What the bug is

The auto-resolve pass conflates two distinct states — "this file's diff contains no flagged comment blocks" and "this file's diff was too large for GitHub to return a patch" — and treats both as "the flagged block is gone, resolve the thread."

Code path

presentKeys (line 106) is derived from groups, which is populated only from files that survive all four filters at lines 98–101. The last of those is:

if (!f.patch) continue;

GitHub's pulls.listFiles REST endpoint omits the patch field on a file entry when that file's diff is large (and truncates the file list entirely at 3000 files). When that happens, f.patch is undefined, the file is skipped, and none of its groups are added to presentKeys.

Then at line 140:

if (!t.isResolved && !presentKeys.has(key)) toResolve.push(t.id);

Every existing comment-cop thread whose key isn't in presentKeys is queued for resolveReviewThread. There is no check for "the file this key belongs to was skipped for having no patch," so the thread is resolved even though the flagged comment block is still in the diff — the action just couldn't scan it.

Why nothing else prevents it

The key format is ${path}:${sha12}, so the path is recoverable from the marker, but nothing consults it. The only inputs to the resolution decision are t.isResolved and set membership in presentKeys; there is no separate record of which files were seen-but-unscannable vs. genuinely absent from the diff.

Step-by-step proof

  1. Push 1: a claude-labeled PR touches src/foo.rs and adds a 3-line comment block. listFiles returns { filename: 'src/foo.rs', patch: '@@ … +// a\n+// b\n+// c …' }. The scanner emits one group with key src/foo.rs:abc123def456, adds it to presentKeys, and posts a review comment with marker <!-- comment-cop:src/foo.rs:abc123def456 -->.
  2. Push 2: the same PR adds a large generated section elsewhere in src/foo.rs (or the cumulative diff for that file otherwise crosses GitHub's per-file threshold). The 3-line comment block is untouched. listFiles now returns { filename: 'src/foo.rs', status: 'modified', additions: 4200, … } with no patch field.
  3. Line 101 evaluates !f.patch → true → continue. groups gets nothing for src/foo.rs; presentKeys does not contain src/foo.rs:abc123def456.
  4. The GraphQL query returns the thread from step 1 with isResolved: false and body containing the marker. Line 136 extracts key = 'src/foo.rs:abc123def456'.
  5. Line 140: !false && !presentKeys.has('src/foo.rs:abc123def456') → true && true → thread id pushed to toResolve.
  6. resolveReviewThread marks the thread resolved. The comment block is still there.

Impact

Low. This is a CI nag helper on claude-labeled PRs, not runtime code. The failure mode is a bot review thread getting collapsed prematurely — non-destructive and reversible (it can be manually unresolved), and a human reviewer still sees the comment block in the diff. The trigger — a src/ file's diff crossing GitHub's per-file patch-omission threshold on a later push after having been flagged on an earlier one — is narrow for focused AI PRs, though not impossible.

Fix

Track indeterminate paths and exclude them from auto-resolve:

const noPatch = new Set();
for (const f of files) {
  if (f.status === 'removed') continue;
  if (!f.filename.startsWith('src/')) continue;
  if (!SRC_EXT.test(f.filename)) continue;
  if (!f.patch) { noPatch.add(f.filename); continue; }
  for (const g of groupsFromPatch(f.filename, f.patch)) groups.push(g);
}
…
if (!t.isResolved && !presentKeys.has(key)) {
  const path = key.slice(0, key.lastIndexOf(':'));
  if (!noPatch.has(path)) toResolve.push(t.id);
}

Two or three lines; leaves threads on unscannable files alone rather than assuming either way.

Jarred-Sumner pushed a commit that referenced this pull request Jul 27, 2026
…35534)

Adds a self-contained GitHub Action that scans `claude`-labeled PRs for
multi-line comment blocks added under `src/` and leaves a review line
comment on each one asking for it to be deleted (and the thread resolved
once done).

### Behavior

- Triggers on `pull_request_target` (`opened` / `synchronize` /
`reopened` / `labeled`), gated on `github.repository == 'oven-sh/bun'`
and the `claude` label being present. The `labeled` trigger covers the
normal case where `auto-label-claude-prs.yml` applies the label after
open.
- Fetches PR files via `pulls.listFiles` and parses the unified diff in
`actions/github-script`. No repo checkout.
- A group is **2+ consecutive added lines** that are pure comment lines
(`//`, `///`, `//!`, `/* ... */`, `* ` continuation) in `.rs` / `.c` /
`.cpp` / `.h` / `.ts` / `.js` and friends. Groups containing `SAFETY:`
are skipped.
- Posts one standalone review line comment per consecutive group via
`pulls.createReviewComment` (no review summary).
- Each comment carries `<!-- comment-cop:<path>:<sha12> -->` keyed on
the group's content hash, so reruns (new pushes, relabels) skip groups
that already have a marker.
- On each run, any existing comment-cop thread whose flagged block is no
longer present in the diff is auto-resolved via `resolveReviewThread`.

### Verification

Dry-ran the parser against the five most recent open `claude` PRs via
`gh api repos/oven-sh/bun/pulls/<n>/files`:

```
#35531  src/runtime/webcore/Blob.rs:5245-5246
#35529  src/http_jsc/websocket_client.rs:907-910
#35528  src/runtime/webcore/FileSink.rs:917-920
#35526  src/http_jsc/websocket_client.rs:2179-2182
#35532  (clean)
```

Cross-checked `#35531` line numbers against the fetched head ref;
`Blob.rs:5245` is the first line of the flagged block. `actionlint` and
prettier both pass.

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · build/CI scripts only; test-proof not
applicable

<!-- robobun:evidence:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants