Repository navigation
ci: a label-driven Jibril release gate covering every recorded pnpm failure mode - #43
Conversation
…able-workflow form
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds a Garnet Jibril release gate with release selection, sensor reproduction, baseline timing, evidence verification, artifact recording, tracking comments, and a failure ledger covering 25 failure modes. ChangesGarnet release gate
Estimated code review effort: 4 (Complex) | ~60 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant GateWorkflow
participant ReusableGateJob
participant GarnetControlPlane
participant VerifyScript
participant GitHubAPI
GateWorkflow->>ReusableGateJob: dispatch selected Jibril tag
ReusableGateJob->>ReusableGateJob: run baseline and sensor workload
ReusableGateJob->>GarnetControlPlane: produce profile and status evidence
ReusableGateJob->>VerifyScript: pass job outputs and baseline timing
VerifyScript->>GarnetControlPlane: verify profile and attribution
VerifyScript->>GitHubAPI: verify comments, workflows, and review threads
VerifyScript->>ReusableGateJob: return verdict and records
ReusableGateJob->>GitHubAPI: upload artifacts and update tracking comment
Suggested reviewers: ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…nt and permalink The gate used to stop at the systemd unit. It now runs a second job after the sensor's post step and checks the whole evidence chain: profile uploaded to the control plane for this run and attempt, the workload destination attributed to the workload step, the Runtime Review App comment final for the PR head, and the profile-selected public permalink answering 200. Each run writes one record (base, head, merge and profile SHAs, comparison scope, action SHA, requested and resolved Jibril version) to the step summary, an artifact and a per-tag comment that is updated in place instead of duplicated. A pull_request trigger scoped to the gate files runs the chain in PR context, the only context where the App comment exists. The caller grants pull-requests: write to the called workflow; the workload step is non-fatal so a registry outage reads as an untested leg rather than a failed gate.
|
Execution Profiles recorded for 1 job, triggered by
|
Jibril release gate ·
|
| leg | result |
|---|---|
| L1 startup | FAIL — status file absent — sensor never reached main loop |
| L2 profile | absent (control plane HTTP 403) · 0 egress peers, 0 destinations |
| L3 workload | untested |
| L4 App comment | pending · #43 (comment) |
| L5 permalink | html untested · api untested |
| L6 integration shape | action 3d47f4a9; sensor version left to the action; explicit token |
| L7 integration run | untested: no TS CI run found for head 0a0750a |
| L8 reviewers | zizmor: 0 finding(s) at the regular persona, 0 pedantic, on 2 touched workflow file(s) · unresolved bot threads on .github/: 0 |
| L9 timing | sensor start 12s · workload 1s with sensor vs 0s without · overhead 13s (budget 120s) |
| L10 coverage | 1 of 6 CI cells instrumented: ubuntu/node 24 · not instrumented: ubuntu/node 22, ubuntu/node 26, windows 1/3/node 22, windows 2/3/node 22, windows 3/3/node 22 · Linux x86_64 only; macOS, Windows, fork PRs and Dependabot runs record nothing |
| record | value |
|---|---|
| jibril | requested v2.17.0-rc.9 · resolved unknown |
| action | garnet-org/action@e546567a72e4fede11ec39d6e9f75b539adef22c |
| run | 34288923777 attempt 1 · blacksmith-01m21jtfr76yr1xk6cg2vtbsff-8vcpu · pull_request |
| workflow ref seen by sensor | absent |
| base → head | 0d0bbd1 → 0a0750a |
| merge sha at run | 6665a7d · now 6665a7d · profile recorded none |
| comparison in comment | n/a |
| synthetic merge | no |
Failing legs:
- L1 startup: status file absent — sensor never reached main loop
- L2 profile: absent (control plane HTTP 403) after 600s (run 34288923777)
- L4 app comment: pending after 600s for head 0a0750a · no profile reached the control plane, so the placeholder has nothing to finalize from
Disclosed, not failing:
- L6 shape: release.yml keeps 2 Garnet step(s) on macOS jobs; the sensor never records there (CodeRabbit and Qodo flagged this on fork PRs)
- L6 shape: ci.yml passes the token explicitly, so Dependabot-actor runs get an empty value and record nothing (upstream pull 13270)
Legs map to past failures in .github/garnet-gate/FAILURE_LEDGER.md. Full record in the run's garnet-release-gate artifact.
…record When no profile exists the permalink leg is never probed; show it as untested instead of a curl 000 code, and print the recorded profile SHA as none instead of an empty code span.
… pnpm failure mode The gate now runs on any pull request labelled garnet-release-testing and reports on that pull request. The verifier grows from startup, profile, workload, App comment and permalink to ten legs: integration shape as pnpm reviewers read it, pnpm's own TS CI run on the candidate, zizmor and open bot review threads on touched workflow files, sensor overhead against a no-sensor baseline, and the instrumented share of the CI matrix. Each leg maps to a row in .github/garnet-gate/FAILURE_LEDGER.md, which lists every Garnet failure seen on pnpm so far and the check that catches it. Disclosures that a machine cannot decide (macOS steps, Dependabot runs, fork pull requests) print under the verdict instead of failing it.
Jibril release gate ·
|
| leg | result |
|---|---|
| L1 startup | PASS — sensor active, status ok, github steps resolution ok |
| L2 profile | uploaded after 1s · 01a08476-307d-7456-9c96-4bcf02108314 · 2 egress peers, 2 destinations |
| L3 workload | destination present, but attributed to steps ["0. Start the clock"] instead of "Workload with egress" |
| L4 App comment | final after 0s · #43 (comment) |
| L5 permalink | html 200 · api 200 · https://app.garnet.ai/public/runs/34311733947?profile=01a08476-307d-7456-9c96-4bcf02108314 |
| L6 integration shape | action 3d47f4a9; sensor version left to the action; explicit token |
| L7 integration run | TS CI run 34311733945 finished success with no profile (no profile is bound to that run): the job was green, the sensor recorded nothing |
| L8 reviewers | zizmor: 0 finding(s) at the regular persona, 0 pedantic, on 2 touched workflow file(s) · unresolved bot threads on .github/: 5 |
| L9 timing | sensor start 16s · workload 0s with sensor vs 0s without · overhead 16s (budget 120s) · profile visible 1s after the job · comment final 0s after the profile |
| L10 coverage | 1 of 6 CI cells instrumented: ubuntu/node 24 · not instrumented: ubuntu/node 22, ubuntu/node 26, windows 1/3/node 22, windows 2/3/node 22, windows 3/3/node 22 · Linux x86_64 only; macOS, Windows, fork PRs and Dependabot runs record nothing |
| record | value |
|---|---|
| jibril | requested v2.17.0-rc.10 · resolved v2.17.0-rc.10 |
| action | garnet-org/action@e546567a72e4fede11ec39d6e9f75b539adef22c |
| run | 34311733947 attempt 1 · blacksmith-01m226xjw08tkpf4gx4yytjnhp-8vcpu · pull_request |
| workflow ref seen by sensor | garnet-labs/pnpm/.github/workflows/garnet-jibril-release-gate.yml@refs/pull/43/merge |
| base → head | 0d0bbd1 → a4abb18 |
| merge sha at run | 1a06ba6 · now 1a06ba6 · profile recorded 1a06ba6 (refs/pull/43/merge) |
| comparison in comment | immediate parent → head (previous f76a862) |
| synthetic merge | yes · sensor recorded merge commit 1a06ba6, PR head is a4abb18 |
Failing legs:
- L3 workload: destination present, but attributed to steps ["0. Start the clock"] instead of "Workload with egress"
- L7 integration run: TS CI run 34311733945 finished success with no profile (no profile is bound to that run): the job was green, the sensor recorded nothing
- L8 reviewers: 5 unresolved bot review thread(s) on .github/ paths
- qodo-code-review on .github/workflows/garnet-jibril-release-gate.yml: ci: a label-driven Jibril release gate covering every recorded pnpm failure mode #43 (comment)
- qodo-code-review on .github/scripts/garnet-release-gate-verify.sh: ci: a label-driven Jibril release gate covering every recorded pnpm failure mode #43 (comment)
- qodo-code-review on .github/scripts/garnet-release-gate-verify.sh: ci: a label-driven Jibril release gate covering every recorded pnpm failure mode #43 (comment)
- qodo-code-review on .github/scripts/garnet-release-gate-verify.sh: ci: a label-driven Jibril release gate covering every recorded pnpm failure mode #43 (comment)
- qodo-code-review on .github/workflows/garnet-jibril-release-gate-job.yml: ci: a label-driven Jibril release gate covering every recorded pnpm failure mode #43 (comment)
Disclosed, not failing:
- L6 shape: release.yml keeps 2 Garnet step(s) on macOS jobs; the sensor never records there (CodeRabbit and Qodo flagged this on fork PRs)
- L6 shape: ci.yml passes the token explicitly, so Dependabot-actor runs get an empty value and record nothing (upstream pull 13270)
Legs map to past failures in .github/garnet-gate/FAILURE_LEDGER.md. Full record in the run's garnet-release-gate artifact.
…ob list workflow runs Jibril resolves GITHUB_JOB against the caller workflow file. With the caller job named gate and the callee job named reproduce, v2.17.0-rc.10 reported github.steps.status=degraded and attributed the workload egress to an unknown step (run 34309875655). pnpm's ci.yml calls test.yml with matching job ids, so the gate now does the same. The verify job's token had no actions: read, so listing the PR head's TS CI run failed and the integration leg read the error as an absent run. The leg now fails on a listing error and reports an absent run only on an empty list. Ledger rows F21 and F22 record both.
|
/agentic_review |
|
@coderabbitai review |
Code Review by Qodo
1. The gate rejects its own workload evidence
|
| export MARKER="<!-- garnet:jibril-release-gate $tag -->" | ||
| existing="$(gh api "repos/$GH_REPO/issues/$report_pr/comments" --paginate \ | ||
| | jq -r '.[] | select((.body // "") | contains($ENV.MARKER)) | .id' | head -n1)" | ||
| [ -z "$existing" ] || run_gate=false |
There was a problem hiding this comment.
2. Anyone can suppress scheduled gates 🐞 Bug ⛨ Security
detect treats any issue comment containing the predictable per-tag marker as proof that the release was already gated, without validating the comment author. A public commenter can post that marker before the scheduled run and force run_gate=false, while the updater can also select the spoofed comment as its update target.
Agent Prompt
## Issue description
Scheduled gates trust a marker in any PR comment, allowing an untrusted commenter to suppress a release run or become the updater's selected comment.
## Issue Context
Both marker lookup sites must validate that the comment belongs to the trusted gate identity, not merely that its body contains the marker.
## Fix Focus Areas
- .github/workflows/garnet-jibril-release-gate.yml[74-77]
- .github/workflows/garnet-jibril-release-gate-job.yml[232-238]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| added_lines="$(gh api "repos/$repo/compare/$PR_BASE_SHA...$PR_HEAD_SHA" --paginate 2>/dev/null \ | ||
| | jq -r '.files[]? | select(.filename | startswith(".github/workflows/")) | "\(.filename)\t\(.patch // "")"' \ | ||
| | awk -F'\t' '{f=$1; n=split($2, L, "\n"); for(i=1;i<=n;i++) if (L[i] ~ /^\+/) print f "\t" substr(L[i],2)}')" |
There was a problem hiding this comment.
3. Large diffs bypass permission checks 🐞 Bug ⛨ Security
L6 derives added workflow lines from the compare API's optional patch field and silently substitutes an empty string when that field is absent or the request fails. A sufficiently large comparison or unavailable API can therefore hide newly added token permissions and let the integration-shape leg pass without examining them.
Agent Prompt
## Issue description
The permission gate silently loses workflow additions when the compare API fails, truncates files, or omits a patch.
## Issue Context
The repository is already checked out, so use a complete local diff or explicitly fail L6 whenever the remote comparison cannot be proven complete.
## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[230-245]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| run="$(jq -c --arg n "$INTEGRATION_WORKFLOW_NAME" '[.workflow_runs[] | select(.name == $n)] | sort_by(.run_attempt) | last // empty' <<<"$runs_json")" | ||
| [ -n "$run" ] || break |
There was a problem hiding this comment.
4. Missing ci runs still pass the gate 🐞 Bug ≡ Correctness
L7 immediately breaks its polling loop when no matching TS CI run exists and never calls leg_fail. When the run is absent rather than merely still starting, the final verdict can remain PASS even though the required integration execution was never observed.
Agent Prompt
## Issue description
L7 treats an absent integration workflow run as non-failing and exits the advertised polling loop immediately.
## Issue Context
Continue polling while no matching run is registered, then append an L7 failure if the deadline expires without one.
## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[256-286]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
GitHub sets GITHUB_ACTION to a step's id when one is present. Jibril derives the step value from the workflow file as __run_N, so a run step with an id records on step <unknown> even when step discovery is ok. pnpm's install and test steps carry no id; the gate now matches them and logs GITHUB_ACTION for the record. The workload outcome reaches the assert step through a file. L7 names the control plane's 403 for what it is on this path: no profile bound to the run.
|
/agentic_review |
|
@coderabbitai review |
|
Code review by qodo was updated up to the latest commit f76a862 |
…tion With no id on the workload step the egress recorded on the job's first run step. GitHub numbers only id-less run steps (GITHUB_ACTION=__run for the third run step of the job); Jibril numbers every run step, so each id-bearing run step ahead of the workload shifts attribution one step back. pnpm's test job has one such step before its test step. The gate keeps that shape and the L3 leg names the expected step next to the recorded one.
|
/agentic_review |
|
@coderabbitai review |
| # step with an id records as `<unknown>` (ledger F23). The two | ||
| # id-bearing run steps above stay: pnpm's job has one before its test | ||
| # step, and Jibril counts them where GitHub does not (ledger F25). |
There was a problem hiding this comment.
1. The gate rejects its own workload evidence 🐞 Bug ≡ Correctness
The reproduction adds two id-bearing run steps before Workload with egress, while the verifier requires the registry destination to be attributed to that workload name. Because Jibril numbers id-less run steps differently from GitHub and the recorded failure shows this shape shifts attribution to an earlier step, the gate can fail L3 on its deliberately reproduced workload even when the sensor records the egress.
Agent Prompt
## Issue description
The reproduction places two id-bearing `run` steps before the id-less workload step, while the verifier requires attribution to `Workload with egress`. Jibril's step numbering can therefore attribute the registry egress to an earlier step and make L3 fail for a healthy reproduction.
## Issue Context
The focused change documents that the reproduced shape should match pnpm's test job, which has one id-bearing run step before the workload. The verifier treats any attribution other than the configured workload name as a failure.
## Fix Focus Areas
- .github/workflows/garnet-jibril-release-gate-job.yml[71-107]
- .github/scripts/garnet-release-gate-verify.sh[115-130]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
PR Code Suggestions ✨Warning
No code suggestions found for the PR. |
| run="$(jq -c --arg n "$INTEGRATION_WORKFLOW_NAME" '[.workflow_runs[] | select(.name == $n)] | sort_by(.run_attempt) | last // empty' <<<"$runs_json")" | ||
| [ -n "$run" ] || break | ||
| integration_run_id="$(jq -r '.id' <<<"$run")" | ||
| integration_conclusion="$(jq -r '.conclusion // ""' <<<"$run")" | ||
| [ "$(jq -r '.status' <<<"$run")" = completed ] && break |
There was a problem hiding this comment.
🔴 Missing integration runs pass
When no matching run exists or it remains incomplete, integration_state stays untested without adding a failure. The gate can report PASS without exercising L7.
Prompt for agents
Update L7 in .github/scripts/garnet-release-gate-verify.sh so absence of a matching TS CI run, or failure to complete before INTEGRATION_POLL_SECONDS, adds an L7 failure. Continue polling when no run has appeared instead of breaking immediately. Preserve the separate error for inability to list runs, and only fetch a profile after a selected run completes.
Was this helpful? React with 👍 or 👎 to provide feedback.
| added_lines="$(gh api "repos/$repo/compare/$PR_BASE_SHA...$PR_HEAD_SHA" --paginate 2>/dev/null \ | ||
| | jq -r '.files[]? | select(.filename | startswith(".github/workflows/")) | "\(.filename)\t\(.patch // "")"' \ | ||
| | awk -F'\t' '{f=$1; n=split($2, L, "\n"); for(i=1;i<=n;i++) if (L[i] ~ /^\+/) print f "\t" substr(L[i],2)}')" |
There was a problem hiding this comment.
🔴 Permission additions evade validation
For multi-line patches, awk inspects only the first line because later lines lack the filename field. Forbidden permissions never reach added_lines, so L6 can accept them.
Prompt for agents
Rewrite the L6 compare parsing in .github/scripts/garnet-release-gate-verify.sh so every added patch line remains associated with its file. The GitHub compare API returns each patch as one string containing embedded newlines, so a filename plus tab prefix applies only to the first rendered line. Parse each file and its patch structurally, emit every '+' line with the filename, and distinguish API or parsing failures from an empty diff.
Was this helpful? React with 👍 or 👎 to provide feedback.
| code="$(fetch_profile "$integration_run_id")" | ||
| if [ "$code" = 200 ]; then | ||
| integration_profile_id="$(jq -r '.id' "/tmp/profile-$integration_run_id.json")" | ||
| integration_state="profile \`$integration_profile_id\` from $INTEGRATION_WORKFLOW_NAME run $integration_run_id ($integration_conclusion)" |
There was a problem hiding this comment.
🔴 Old profiles satisfy reruns
On a rerun, fetch_profile can return the previous attempt's profile, which L7 accepts without checking its attempt or job. A failed current attempt can satisfy the integration leg.
Prompt for agents
Validate the L7 profile against the selected TS CI run's current run_attempt and the expected instrumented job, as L2 already does for the gate profile. Treat a profile from another attempt or job as absent for the current integration run, and poll if current-attempt ingestion can lag.
Was this helpful? React with 👍 or 👎 to provide feedback.
| action_refs="$(grep -hoE 'garnet-org/action@[^ ]+' "${integration_files[@]}" 2>/dev/null | sort -u)" | ||
| unpinned="$(grep -vE '^garnet-org/action@[0-9a-f]{40}$' <<<"$action_refs" | sed '/^$/d')" | ||
| if [ -n "$unpinned" ]; then | ||
| shape_fail=true; shape_notes+=("action ref not a full SHA: $(tr '\n' ' ' <<<"$unpinned")") | ||
| else | ||
| shape_notes+=("action \`${action_refs#garnet-org/action@}\`") |
There was a problem hiding this comment.
🔴 Removed monitoring passes validation
When all monitoring actions are removed, action_refs and unpinned are empty. L6 treats that state as a valid full-SHA pin and can pass the broken integration.
Prompt for agents
Make L6 fail when no garnet-org/action reference is found in the required integration workflows. Keep that missing-action state distinct from the existing unpinned-reference failure, and ensure the expected workflow placement is validated rather than accepting one unrelated remaining reference.
Was this helpful? React with 👍 or 👎 to provide feedback.
| threads="$(gh api graphql -f query=' | ||
| query($owner:String!,$name:String!,$number:Int!){ | ||
| repository(owner:$owner,name:$name){ pullRequest(number:$number){ | ||
| reviewThreads(first:100){ nodes{ isResolved path comments(first:1){ nodes{ author{ login } url } } } } } } }' \ | ||
| -f owner="${repo%/*}" -f name="${repo#*/}" -F number="$PR_NUMBER" 2>/dev/null \ | ||
| | jq -c '[.data.repository.pullRequest.reviewThreads.nodes[]? | ||
| | select(.isResolved == false) | ||
| | select(.path | startswith(".github/")) | ||
| | select((.comments.nodes[0].author.login // "") | test("coderabbit|qodo|greptile|devin|\\[bot\\]"; "i")) | ||
| | {path, author: .comments.nodes[0].author.login, url: .comments.nodes[0].url}]' || echo '[]')" |
| # No `id:` here, as on pnpm's install and test steps. GitHub sets | ||
| # GITHUB_ACTION to the step id when one is present; Jibril derives | ||
| # `__run_N` from the workflow file and attributes by that value, so a | ||
| # step with an id records as `<unknown>` (ledger F23). The two | ||
| # id-bearing run steps above stay: pnpm's job has one before its test | ||
| # step, and Jibril counts them where GitHub does not (ledger F25). | ||
| - name: Workload with egress |
There was a problem hiding this comment.
🔍 Workload attribution remains unresolved
Runtime evidence (Garnet, head a4abb18): Runner.Worker → bash → node reached registry.npmjs.org under “Start the clock” in Garnet Jibril Release Gate/reproduce. The recorded head still misses the intended workload step.
Was this helpful? React with 👍 or 👎 to provide feedback.
| break | ||
| fi | ||
| run="$(jq -c --arg n "$INTEGRATION_WORKFLOW_NAME" '[.workflow_runs[] | select(.name == $n)] | sort_by(.run_attempt) | last // empty' <<<"$runs_json")" | ||
| [ -n "$run" ] || break |
There was a problem hiding this comment.
Integration check can pass untested
When the first workflow-run query finds no matching TS CI run, this line exits the loop immediately. L7 remains marked as “untested,” but no failure is recorded, so the gate can report PASS without verifying the required integration run. This is especially relevant when the gate starts before the parallel TS CI run has been registered, because the intended 25-minute polling window is bypassed after the first empty response.
| count_findings() { jq 'if type == "array" then map(select(.ignored | not)) | length else 0 end' "$1" 2>/dev/null || echo 0; } | ||
| # shellcheck disable=SC2086 | ||
| zizmor --format json-v1 --no-exit-codes --offline $touched > /tmp/zizmor.json 2>/tmp/zizmor.err || true | ||
| zizmor_findings="$(count_findings /tmp/zizmor.json)" | ||
| # shellcheck disable=SC2086 | ||
| zizmor --format json-v1 --no-exit-codes --offline --persona pedantic $touched > /tmp/zizmor-pedantic.json 2>/dev/null || true | ||
| zizmor_pedantic="$(count_findings /tmp/zizmor-pedantic.json)" | ||
| zizmor_state="$zizmor_findings finding(s) at the regular persona, $zizmor_pedantic pedantic, on $(wc -l <<<"$touched") touched workflow file(s)" | ||
| if [ "$zizmor_findings" -gt 0 ]; then | ||
| leg_fail "L8 reviewers: zizmor $zizmor_state" | ||
| jq -r '.[] | select(.ignored | not) | " - \(.ident) (\(.determinations.severity)): \(.locations[0].symbolic.key.Local.path // "")"' /tmp/zizmor.json 2>/dev/null | ||
| fi | ||
| else | ||
| zizmor_state="zizmor not installed" | ||
| fi | ||
| else | ||
| zizmor_state="no workflow files touched" | ||
| fi | ||
| threads="$(gh api graphql -f query=' | ||
| query($owner:String!,$name:String!,$number:Int!){ | ||
| repository(owner:$owner,name:$name){ pullRequest(number:$number){ | ||
| reviewThreads(first:100){ nodes{ isResolved path comments(first:1){ nodes{ author{ login } url } } } } } } }' \ | ||
| -f owner="${repo%/*}" -f name="${repo#*/}" -F number="$PR_NUMBER" 2>/dev/null \ | ||
| | jq -c '[.data.repository.pullRequest.reviewThreads.nodes[]? | ||
| | select(.isResolved == false) | ||
| | select(.path | startswith(".github/")) | ||
| | select((.comments.nodes[0].author.login // "") | test("coderabbit|qodo|greptile|devin|\\[bot\\]"; "i")) | ||
| | {path, author: .comments.nodes[0].author.login, url: .comments.nodes[0].url}]' || echo '[]')" |
There was a problem hiding this comment.
L8 treats failures of both required reviewer checks as clean results. Invalid or missing zizmor output becomes zero findings, while a failed GraphQL review-thread query becomes an empty thread list. A scanner crash, API outage, or query error can therefore let the release gate report PASS without checking for workflow findings or unresolved review threads.
Summary
One gate for pnpm's Garnet health, on this fork only. It runs on every pull request labelled
garnet-release-testing(and on a schedule for each new Jibril tag), reproduces the exact shape that has kept upstream dark since 2026-09-02 (a reusable workflow called with the$/self-repository form, on theblacksmith-8vcpu-ubuntu-2404runner class), and passes only when the whole evidence chain lands. A green job is not a pass.Ten legs, evaluated by
.github/scripts/garnet-release-gate-verify.sh:jibrilunit active,/var/run/jibril/jibril.status.jsonwith.status = okand GitHub step discovery okregistry.npmjs.orgin the profile's egress, attributed to the stepWorkload with egressgarnet-runtime-review[bot]comment bound to the PR head, final marker present. If pending, the leg names the cause it can see: no profile, or a merge commit GitHub regenerated after the sensor recordedci.yml/test.yml/release.ymlas pnpm reviewers read them: full-SHA action pin, explicitapi_token, noid-token: write,secrets: inheritor newpull-requests: writeadded by the PR. macOS steps and the Dependabot empty-token case are printed as disclosures.github/Every leg maps to a row in
.github/garnet-gate/FAILURE_LEDGER.md: every Garnet failure seen on pnpm so far, where it was seen, and the check that catches it. A new failure gets a ledger row and a check in the same PR.Each run writes a JSON record (base, head, merge-at-run and current merge SHA, action SHA, requested and resolved Jibril version, profile id, comparison scope, timing, coverage, every leg's state) as an artifact, renders it as the Step Summary, and keeps one comment per tag on the PR, updated in place.
Standing rule this PR encodes: every pull request on this fork that tests Garnet product or release work for pnpm carries
garnet-release-testing. Unlabelled test PRs have not been accepted. The label is on every open Garnet test PR now.The caller job in
garnet-jibril-release-gate.ymlis namedreproduce, the same id as the job it calls, becauseci.yml'stestcallstest.yml'stest. Jibril resolvesGITHUB_JOBagainst the caller file; with differing ids the status file reports step discovery degraded and the egress lands on an unknown step (ledger F21).What this PR does not do: touch
ci.ymlor any upstream-facing workflow, addid-token: write, change permissions beyond the gate's ownpull-requests: writefor its comment andactions: readfor listing the head's TS CI run, or claim coverage for macOS, Windows, fork PRs or Dependabot runs.Expected diagnostics:
uses: $/.github/workflows/garnet-jibril-release-gate-job.ymlis deliberate. It is the production failure shape. actionlint and Greptile flag it; GitHub accepts it, and upstreamci.ymlruns this form on every PR.blacksmith-8vcpu-ubuntu-2404label; that is the runner class upstream uses for the Garnet-enabled cell.Live results on this PR, one comment per tag:
v2.17.0-rc.9: FAIL on L1, L2, L4. The sensor exits on the$/reference; no profile, comment pending.v2.17.0-rc.10(run 34309875655): the sensor stays active, the profile uploads, the App comment finalizes, the permalink answers 200, overhead 15 s. FAIL on L1 (steps.status=degraded,job reproduce not found) and L3 (egress on step<unknown>), both caused by the caller job idgatein the gate itself, and L7 read "no TS CI run" because the verify token had noactions: read. This push fixes the gate for both; the next labelled run is the reading that counts.The verifier was also run locally against a known good fork run (
33660546962) → PASS.Related: garnet-org/jibril#754 (the fix), garnet-org/action#147 (
api_token-first auth and start-failure disclosure).Squash Commit Body
Checklist
v2.17.0-rc.10.Link to Devin session: https://app.devin.ai/sessions/3ab2e708c38e478dad8aa472a81c8fcc
Open in Devin Desktop: https://app.devin.ai/desktop/session/3ab2e708c38e478dad8aa472a81c8fcc?variant=devin
Requested by: @jadoonf
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Runtime evidence (Garnet)
Kernel-recorded execution record for head
a4abb18faa6447ed6c22b7fb3a8285eae3efae08, mirrored verbatim fromthe sticky Garnet Runtime Review comment on this PR so reviewers that read only
the description ground in the same bytes. Facts only. Judgment stays with the
reviewer. Cite grounded findings as:
Execution record (verbatim mirror)
Execution Profiles recorded for 1 job, triggered by
a4abb18Garnet Jibril Release Gate/reproduce↗ · 2 destinationsView this job's Execution Profile in Garnet →
💡 How to read this
Runner.Worker ← process on a path └─ npm └─ node ← process that acted └─ ○ npmjs[.]org ← observed actionfollow a path downward to see what ran and what it did — each path to an observed action is an execution chain
names on the path = processes · ○ = observed action · (…) = context
+ only in the current record · − only in the previous record · runner background = the runner's infrastructure, not your workflow
Workload behaviour (Garnet)
Garnet Jibril Release Gate/reproduce· head1a06ba6· 3 workload chains · 1 runner-platform chainsWorkload behaviour vs
ba67148: +0 −8 destinations−
github[.]com−
codeload.github[.]com−
nodejs[.]org−
node-registry.bit[.]cloud−
pnpm[.]io−
release-assets.githubusercontent[.]com−
unofficial-builds.nodejs[.]org−
npm.jsr[.]iorunner platform: +0 −20 destinations · not workload behaviour
3 workload chains · 2 destinations at head, 8 gone
localhostnode,sudoregistry.npmjs[.]orgnodecodeload.github[.]comnodegithub[.]comgit-remote-http,nodenode-registry.bit[.]cloudnodenodejs[.]orgnodenpm.jsr[.]ionodepnpm[.]ionoderelease-assets.githubusercontent[.]comnodeunofficial-builds.nodejs[.]orgnoderunner platform · 1 chains · 1 destinations at head, 20 gone · no recorded workflow step
localhostsudo140.82.112.22nginx140.82.113.21nginx140.82.114.21nginx169.254.169.123chronyd185.125.190.56chronyd185.125.190.57chronyd185.125.190.58chronyd192.168.127.1blacksmithd,sudo198.137.202.56chronyd23.143.196.204chronyd23.186.168.128chronyd5.161.94.12chronyd8.8.8.8nginx,systemd-resolve91.189.91.157chronydapi.snapcraft[.]iosnapdcloudfront.cdn.snapcraftcontent[.]comsnapdnodejs[.]orgpnpmproductionresultssa15.blob.core[.]windowsnoderegistry.npmjs[.]orgpnpmunofficial-builds.nodejs[.]orgpnpmrecorded at the kernel by Garnet · profile view →
Summary by CodeRabbit
Greptile Summary
This PR adds a label-driven and scheduled Jibril release gate, a ten-leg verifier, persisted evidence artifacts and comments, and a ledger mapping recorded pnpm/Garnet failures to checks.
a4abb18confirms the gate’sreproducejob uploaded profile01a08476-307d-7456-9c96-4bcf02108314and reachedregistry.npmjs.org; the captured chain attributed that traffic to “Start the clock,” providing relevant evidence for the attribution failure mode this gate is designed to detect.Confidence Score: 3/5
The PR is not yet safe to merge because required integration-run and reviewer checks can fail open while the release gate reports PASS.
L7 exits its polling loop on the first empty workflow-run response and records no failure, while L8 maps scanner and review-thread query failures to clean results; either behavior breaks the gate’s core contract that PASS represents a complete evidence chain. All previous review findings are no longer outstanding: three were fixed and resolved, and the
$/reference finding was correctly withdrawn after live verification.Files Needing Attention: .github/scripts/garnet-release-gate-verify.sh
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD T[Schedule, dispatch, or labeled PR] --> D[Detect release tag] D --> B[Baseline workload] D --> R[Instrumented reproduce job] B --> V[Ten-leg verifier] R --> V V --> E[JSON artifact and step summary] V --> C[Per-tag PR comment] V -->|All required evidence established| P[PASS] V -->|Any failing leg| F[FAIL]Reviews (2): Last reviewed commit: "ci: record the step numbering skew behin..." | Re-trigger Greptile