Skip to content

ci(garnet): decide upstream drift, warning noise and the credential-less path in the gate - #48

Merged
jadoonf merged 1 commit into
mainfrom
devin/1788969836-gate-v2-pnpm-pillars
Sep 9, 2026
Merged

jadoonf merged 1 commit into
mainfrom
devin/1788969836-gate-v2-pnpm-pillars

Conversation

@jadoonf

@jadoonf jadoonf commented Sep 9, 2026 •

Copy link
Copy Markdown

Summary

This is fork-only infrastructure: it changes the Garnet/Jibril release gate that decides whether a Garnet action release or a Jibril repin is safe to take into pnpm/pnpm. No pnpm CLI code changes.

Before this PR the gate had ten legs and every one of them read only what the fork's own run produced. A run could be green while pnpm's real workflows had drifted away from the shape being reproduced, while the action step emitted warnings, and while nobody had ever exercised the credential-less path a fork PR or Dependabot takes. Four legs were added and the existing ones now decide rather than describe.

New legs:

  • L11 upstream drift. Fetches pnpm/pnpm's ci.yml, test.yml, release.yml and update-latest.yml at main on every run and compares the reproduced job against them through .github/scripts/garnet-gate-shape.py: reusable-call form ($/ vs ./), the runner of the garnet: true cell, the action ref, jibril_version, the explicit api_token, job permissions, and whether an id-bearing run: step sits before the workload. Two differences are declared deliberate (the candidate action ref and the release under test); anything else fails. The upstream SHA goes into the record.
  • L12 warning noise. Pulls the job log, isolates the Garnet runtime monitoring step, counts ::warning::/::error::, and fails any warning on a token-carrying run. The disclosed fail-open warning (F2) is named separately from an unexpected one.
  • L13 credential-less run. A ubuntu-24.04 job runs the action with api_token: '' and a read-only GITHUB_TOKEN. It must exit 0, print one info line naming the missing token, emit no warning and no error. It is labelled a simulation, and the comment states that there is still no server-side evidence for real Dependabot or fork PRs.
  • L14 release/tag instrumentation. Reports every Garnet step in upstream's release and tag workflows with its runner, whether it can produce a Linux sensor profile, and why. macOS is a disclosure; a Garnet step that is inert on a Linux runner fails.

Changed legs:

  • L3 now runs against upstream's actual step order. The reproduce job carries an id-bearing run: step, then an id-less one, then the workload, exactly as pnpm's test job does, so the attribution the gate measures is the attribution pnpm gets. When the egress lands one step early the leg says so by name: Jibril step-numbering skew, garnet-org/jibril parseStepsList (F25).
  • L6 grew into the full posture leg: no id-token: write in the production-shaped job, no secrets: inherit, no permission additions beyond what the gate itself needs, explicit api_token, full 40-char SHA pins, and — read from the action source at the pinned SHA — the version an empty jibril_version resolves to there. A floating default cannot hide behind a pinned action.
  • L9 splits timing into sensor start, workload overhead and post-step flush (30s / 120s / 60s), each measured from the job timeline, because the maintainer objection is that a smoke job must not add meaningful CI time.
  • L4 requires exactly one App comment, updated in place, bound to the current head/merge SHA, carrying the recorded workload step name in double quotes and at least one destination, with the PR-body evidence mirror on the same head.
  • L8 classifies every zizmor finding (regular and --pedantic) through .github/garnet-gate/scanner-dispositions.yml as fixed / disclosed / intentional-diagnostic / garnet-side and fails an unclassified one; actionlint deliberate warnings are whitelisted by message rather than by switching the rule off; unresolved bot review threads under .github/** fail.
  • L10 lists every upstream matrix cell, asserts local and upstream instrumented counts agree, and marks no_bad_egress_domain vacuous — not a proof of anything — until pnpm has deny rules in the control plane.

.github/garnet-gate/FAILURE_LEDGER.md gains F26–F34 (upstream drift, warning noise, credential-less behaviour, release/tag inertness, flush timing, duplicate/mis-bound comment, vacuous assertion, unclassified finding, mirror/head mismatch) and every pre-existing row now names the leg that covers it.

The $/ caller form and the blacksmith-8vcpu-ubuntu-2404 runner are kept on purpose: they are the two shapes pnpm actually uses and the ones that broke Jibril, so the gate reproduces them and classifies the scanner noise they cause instead of hiding it.

Squash Commit Body

Four new legs read what a pnpm maintainer would otherwise check by hand, and
the existing legs decide instead of report.

L11 compares the reproduced job against pnpm/pnpm's ci.yml, test.yml,
release.yml and update-latest.yml at main on every run: reusable-call form,
instrumented runner, action ref, sensor version, explicit api_token, job
permissions and the run-step order around the workload. L12 fails any warning
the Garnet action step emits on a token-carrying run. L13 runs the action with
an empty api_token on ubuntu-24.04 and requires a clean green job. L14 reports
whether each Garnet step in the release and tag workflows can record at all.

The reproduce job now mirrors upstream's step order, so L3 measures the
attribution skew pnpm gets and names it: Jibril step numbering skew,
garnet-org/jibril parseStepsList. L6 reads the action source at the pinned sha
to learn what an empty jibril_version resolves to there. L9 splits timing into
sensor start, workload overhead and post-step flush. L4 requires one in-place,
head-bound App comment matching the PR-body mirror. L8 classifies every
scanner finding through a written disposition file. L10 marks
no_bad_egress_domain vacuous.

Ledger rows F26 to F34 cover the new legs.

Checklist

  • New features are implemented only in the Rust pnpm v12 CLI. Bug fixes
    are implemented in every affected version. (No CLI code changed; CI only.)
  • Added a changeset if this PR changes any published package. (No published package changes.)
  • Added or updated tests. (The gate is the test; validated with bash -n, shellcheck, actionlint, zizmor regular and --pedantic, plus a local verifier dry run.)
  • Updated the documentation if needed. (Failure ledger updated.)

Link to Devin session: https://app.devin.ai/sessions/a29aac85f1f142a6ab5006dc8657db1c
Open in Devin Desktop: https://app.devin.ai/desktop/session/a29aac85f1f142a6ab5006dc8657db1c?variant=devin
Requested by: @jadoonf


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by CodeRabbit

  • New Features

    • Expanded release-gate verification to cover upstream workflow drift, warning noise, credential-less runs, release/tag workflows, timing, and evidence consistency.
    • Added validation that scanner findings have documented dispositions.
    • Added credential-less simulation to confirm workflows skip safely when tokens are unavailable.
    • Improved verification reports with clearer per-check results, disclosures, and evidence details.
  • Bug Fixes

    • Improved workflow output handling and comment validation.
    • Added checks to detect missing or incorrectly classified monitoring steps.

Runtime evidence (Garnet)

Kernel-recorded execution record for head d40afff36dcd786622523701996cdc2cff381940, mirrored verbatim from
the sticky Garnet Runtime Review comment on this PR so reviewers that read only
the description ground in the same bytes. Facts only. Judgment stays with the
reviewer. Cite grounded findings as:

Runtime evidence (Garnet, head d40afff): <execution chain> → <destination> (<workflow>/<job>) —

Execution record (verbatim mirror)

Execution Profiles recorded for 1 job, triggered by d40afff

3 destinations
recorded at the kernel by Garnet · 2026-09-09 16:08 UTC

Garnet Jibril Release Gate / reproduce ↗ · 3 destinations
Runner.Worker
└─ bash
   └─ curl (step: "Stop the action clock")
      ├─ ○ localhost (dns resolver)
      └─ ○ registry.npmjs[.]org

systemd
└─ systemd-resolved
   └─ ○ 8.8.8.8

View this job's Execution Profile in Garnet →


💡 How to read this
Runner.Worker          ← process on a path
└─ npm
   └─ node             ← process that acted
      └─ ○ npmjs[.]org ← observed action

follow a path downward to see what ran and what it did — each path to an observed action is an execution chain

names on the path = processes · ○ = observed action · (…) = context

Greptile Summary

This change expands the release evidence checks, but several checks can still certify incomplete or inaccurate results. Broad workflow permissions, mutable sensor versions, attribution drift, incomplete mirrored evidence, unmeasured flush timing, and invalid scanner dispositions can all be accepted. The duplicate diagnostic-line concern is non-blocking, but should be corrected to keep the credential-less output contract precise.

Confidence Score: 0/5

Not safe to merge: the release validation can approve inaccurate security posture, instrumentation, evidence, timing, and scanner-disposition results.

Multiple independently reproduced blocking failures affect both the workflow-shape validator and the release verification script, including one that weakens credential-posture reporting.

Files Needing Attention: .github/scripts/garnet-gate-shape.py; .github/scripts/garnet-release-gate-verify.sh

Security Review

A valid scalar permissions: write-all is interpreted as no permissions, allowing a broadly credentialed workflow to pass checks intended to reject OIDC and pull-request write access.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex produced a proof for a posted P1 finding anchored by the Scalar permissions reproduction fixture.
  • T-Rex validated the P1 finding through the latest sensor-version path, including the explicit latest acceptance and the default latest rejection outputs.
  • T-Rex evaluated the L11 shape-drift fixture, comparing baseline, modified outputs, and the final L11 fixture result.
  • T-Rex exercised the L4 marker-only reproduction harness, capturing baseline and pointer-only mirror results.
  • T-Rex ran the L9 missing-flush fixture and captured its output.

View all artifacts

T-Rex Ran code and verified through T-Rex

Comments Outside Diff (6)

  1. General comment

    P1 Scalar permissions: write-all bypasses OIDC and pull-request write posture checks

    • Bug
      • permissions_of() accepts only dictionary-valued permissions. A valid workflow-level permissions: write-all parses as the string write-all, so the function returns {}. posture() consequently reports id-token: write absent and no pull-requests: write as passing even though GitHub grants those write permissions under write-all.
    • Cause
      • Lines 68-71 discard scalar permission forms by requiring isinstance(perms, dict), while lines 216 and 220 infer absence from the resulting empty dictionary.
    • Fix
      • Handle GitHub Actions scalar permissions explicitly. At minimum, preserve/recognize write-all and make both posture checks fail for it; ideally normalize all supported scalar forms (read-all, write-all, and {}) before performing scope-specific checks.

    T-Rex Ran code and verified through T-Rex

  2. General comment

    P1 Explicit jibril_version: latest bypasses the sensor pin requirement

    • Bug
      • The check at lines 230–239 treats every non-empty jibril_version value as pinned. A fixture passed jibril_version: latest and the actual posture() function reported sensor_version_pinned.ok=True. At the identical scope, an omitted input whose default resolves to latest reports False.
    • Cause
      • The condition is bool(version) or default_version not in ("", "latest", "unknown"); it validates presence rather than requiring an immutable version value when jibril_version is explicit.
    • Fix
      • Reject mutable/unknown explicit values such as latest (and ideally validate explicit versions against the project’s immutable-version format) before accepting the sensor as pinned.

    T-Rex Ran code and verified through T-Rex

  3. General comment

    P1 L11 misses added id-bearing run steps before the workload

    • Bug
      • Adding an id-bearing upstream run: step immediately before the retained id-less predecessor leaves both compared L11 booleans unchanged, so drift_count remains 0. The focused run showed the workload's GitHub token remained __run_3, but Jibril attribution moved from Immediate anonymous predecessor to the newly added Added identified setup step.
    • Cause
      • Lines 317–326 compare only whether any id-bearing run step exists before the workload and whether the immediate predecessor is id-less. They do not compare the count, order, identities, or full sequence of preceding run steps.
    • Fix
      • Compare the ordered preceding run-step shape (at minimum id-bearing step count and positions; preferably names/ids plus id-less positions) between upstream and gate, and fail L11 on additions or reordering.

    T-Rex Ran code and verified through T-Rex

  4. General comment

    P1 L4 accepts a pointer-only PR evidence mirror when the commit marker matches

    • Bug
      • The changed L4 block extracts only the first <!-- garnet:commit <40-hex> --> marker from the App comment and PR-description evidence block. In the executed pointer-only case, the PR mirror replaced the evidence content with Evidence body is replaced by a pointer: https://evidence.example/run/123 but retained the same marker. The check returned exit 0 with FAIL_COUNT=0 and MIRROR_STATE=mirrored in the description, both bound to 0123456.
    • Cause
      • Lines 300-305 compare mirror_marker and comment_marker only; they do not compare the evidence body, integrity digest, canonical serialized content, or reject a pointer in place of copied evidence.
    • Fix
      • Validate mirror content against the App comment's evidence payload, preferably using a canonical full-block digest or exact structured evidence fields in addition to the commit marker. Reject a mirror that contains only a reference/pointer rather than the required copied evidence.

    T-Rex Ran code and verified through T-Rex

  5. General comment

    P2 L13 permits multiple missing-token info lines

    • Bug
      • The stated L13 contract requires one info line, but a fixture providing two matching missing-token lines with a successful job and no warnings/errors passes the gate.
    • Cause
      • The validation condition only fails when sim_info is less than one ([ "$sim_info" -lt 1 ]) rather than requiring it to equal one.
    • Fix
      • Require exactly one matching line, for example fail unless sim_info equals 1, and report duplicate informational output distinctly.

    T-Rex Ran code and verified through T-Rex

  6. General comment

    P1 L8 accepts arbitrary, missing, and historical fixed classes for live findings

    • Bug
      • The focused runtime fixture fed one non-ignored, route-matched live zizmor finding to the classifier. class: invalid-class, no class field, and class: fixed each yielded gate_unclassified=0 and gate_result=accepted. This conflicts with the dispositions file’s stated four-class contract and with fixed meaning the finding is gone.
    • Cause
      • Lines 474-479 emit any matched class value (including Python None) and count only output lines beginning with unclassified; there is no allowed-class validation and no prohibition on fixed for a currently reported finding.
    • Fix
      • Validate matched disposition classes against the documented allowlist. Treat a missing or unknown class as unclassified/error, and treat fixed as unclassified/error whenever the finding is present in either live zizmor report; only absent findings may retain a historical fixed row.

    T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "ci(garnet): decide upstream drift, warni..." | Re-trigger Greptile

Greptile also left 7 inline comments on this PR.

…ess path in the gate

The gate had ten legs and read only what the fork's own run produced. Four
more legs read what a pnpm maintainer would have to check by hand, and the
ones that existed now decide instead of report.

L11 reads pnpm/pnpm's ci.yml, test.yml, release.yml and update-latest.yml at
main on every run and fails when the reproduced shape stops matching pnpm's:
the reusable-call form, the instrumented runner, the action ref, the sensor
version, the explicit api_token, the job permissions and the run-step order
around the workload. Two differences are declared deliberate (the candidate
action ref and the release tag under test); anything else is drift.

L12 counts the annotations of the Garnet action step and fails any warning on
a token-carrying run, naming the disclosed fail-open text apart from the rest.
L13 runs the action with an empty api_token on ubuntu-24.04 and requires a
green job with one info line and no warning; it is labelled a simulation, and
the comment says no server-side evidence exists for Dependabot or fork runs.
L14 reports every Garnet step in the release and tag workflows with its runner
and whether it can record: macOS is a disclosure, inert on Linux is a failure.

The reproduce job now carries the same run-step order pnpm's test job has, an
id-bearing step then an id-less one before the workload, so the attribution
skew the gate measures is the skew pnpm gets. L3 names it: Jibril step
numbering skew, garnet-org/jibril parseStepsList.

L6 reads the action source at the pinned sha to learn what an empty
jibril_version resolves to there, so a floating default cannot hide behind a
pinned action. L9 splits timing into sensor start, workload overhead and
post-step flush, each with its own budget. L4 requires one App comment,
updated in place, bound to the current head, carrying the recorded step name
in quotes and a destination, and matching the evidence mirror in the PR body.
L8 classifies every zizmor finding through a written disposition file and
fails an unclassified one. L10 marks no_bad_egress_domain vacuous.

Ledger rows F26 to F34 cover the new legs.
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot added the garnet-release-testing Garnet product/release test PR; the Jibril release gate runs and reports here label Sep 9, 2026
@coderabbitai

coderabbitai Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: ddfc21ef-f8a5-4131-b0f6-2a6e28a842e4

📥 Commits

Reviewing files that changed from the base of the PR and between ab4a9cd and d40afff.

📒 Files selected for processing (6)
  • .github/garnet-gate/FAILURE_LEDGER.md
  • .github/garnet-gate/scanner-dispositions.yml
  • .github/scripts/garnet-gate-shape.py
  • .github/scripts/garnet-release-gate-verify.sh
  • .github/workflows/garnet-jibril-release-gate-job.yml
  • .github/workflows/garnet-jibril-release-gate.yml

📝 Walkthrough

Walkthrough

The release gate now evaluates fourteen legs. It parses upstream and local workflow shapes, classifies scanner findings, tests credential-less execution, measures sensor timing, checks release workflows, validates evidence mirrors, and renders an expanded verdict.

Changes

Release gate verification

Layer / File(s) Summary
Workflow shape analysis
.github/scripts/garnet-gate-shape.py, .github/garnet-gate/scanner-dispositions.yml
The new shape script compares local and upstream workflow facts, attribution, posture, matrix coverage, and release instrumentation. Scanner findings use disposition rules.
Workflow evidence and simulation
.github/workflows/garnet-jibril-release-gate-job.yml, .github/workflows/garnet-jibril-release-gate.yml
The workflow adds credential-less simulation, upstream workflow retrieval, actionlint installation, PyYAML setup, and grouped gate outputs.
Expanded verification legs
.github/scripts/garnet-release-gate-verify.sh
The verifier adds parsed posture checks, scanner classification, comment checks, start and flush budgets, upstream drift, warning noise, credential-less validation, and release/tag checks.
Evidence records and verdict reporting
.github/scripts/garnet-release-gate-verify.sh, .github/garnet-gate/FAILURE_LEDGER.md
The verifier records new evidence fields and renders a table-based fourteen-leg comment with failures and disclosures. The failure ledger documents F26–F34.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant GateWorkflow
  participant VerifyScript
  participant ShapeScript
  participant ScannerTools
  participant PullRequest
  GateWorkflow->>VerifyScript: provide upstream and simulation evidence
  VerifyScript->>ShapeScript: compare local and upstream workflow shapes
  ShapeScript-->>VerifyScript: return posture, drift, attribution, and release facts
  VerifyScript->>ScannerTools: scan workflows and classify findings
  ScannerTools-->>VerifyScript: return findings and dispositions
  VerifyScript->>PullRequest: validate comment and evidence mirror
  VerifyScript-->>GateWorkflow: publish verdict and record
Loading

Suggested reviewers: zkochan

✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch devin/1788969836-gate-v2-pnpm-pillars

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@garnet-runtime-review

garnet-runtime-review Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Execution Profiles recorded for 1 job, triggered by d40afff

3 destinations
recorded at the kernel by Garnet · 2026-09-09 16:08 UTC

Garnet Jibril Release Gate / reproduce ↗ · 3 destinations
Runner.Worker
└─ bash
   └─ curl (step: "Stop the action clock")
      ├─ ○ localhost (dns resolver)
      └─ ○ registry.npmjs[.]org

systemd
└─ systemd-resolved
   └─ ○ 8.8.8.8

View this job's Execution Profile in Garnet →


💡 How to read this
Runner.Worker          ← process on a path
└─ npm
   └─ node             ← process that acted
      └─ ○ npmjs[.]org ← observed action

follow a path downward to see what ran and what it did — each path to an observed action is an execution chain

names on the path = processes · ○ = observed action · (…) = context

@jadoonf
jadoonf marked this pull request as ready for review September 9, 2026 16:08
@jadoonf
jadoonf merged commit b1a9db5 into main Sep 9, 2026
26 of 30 checks passed
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Expand Garnet gate with upstream-aware acceptance checks

✨ Enhancement 🧪 Tests ⚙️ Configuration changes 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Expands Garnet release gate from ten to fourteen machine-decided acceptance checks.
• Detects upstream drift, warning noise, credential-less failures, and inert release
 instrumentation.
• Strengthens scanner, timing, attribution, comment, and evidence-record validation.
Diagram

graph TD
  A["Gate Trigger"] --> B["Gate Jobs"] --> C["Upstream Fetch"] --> D["Shape Analyzer"] --> E["14-Leg Verifier"] --> F["Gate Record"]
  E --> G["GitHub APIs"]
  E --> H["Garnet API"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Declarative workflow policy engine
  • ➕ Could express posture and drift rules as independently testable policies.
  • ➕ Would reduce embedded Python and shell classification logic.
  • ➖ Adds another toolchain and policy language to the gate.
  • ➖ Jibril-specific step attribution and log checks still require custom code.
2. Versioned upstream workflow snapshot
  • ➕ Makes reruns deterministic and independent of upstream availability.
  • ➕ Allows drift updates to receive normal code review.
  • ➖ Detects drift only after a separate refresh process runs.
  • ➖ Weakens the core guarantee that every gate execution reflects pnpm main.

Recommendation: Keep the live upstream fetch and focused Python shape analyzer: they directly support the gate's purpose of validating current pnpm behavior. Consider extracting stable posture checks into a policy engine only if the rule set grows substantially; retain custom logic for Jibril attribution, annotations, and runtime evidence.

Files changed (6) +1025 / -118

Enhancement (2) +861 / -103
garnet-gate-shape.pyAnalyze local and upstream Garnet workflow shapes +371/-0

Analyze local and upstream Garnet workflow shapes

• Adds a PyYAML-based analyzer that compares gate and upstream workflow call forms, runners, action pins, sensor versions, tokens, permissions, and run-step ordering. It also predicts Jibril attribution, reports matrix coverage and release instrumentation, and emits one JSON record for the verifier.

.github/scripts/garnet-gate-shape.py

garnet-release-gate-verify.shExpand verification into a fourteen-leg acceptance gate +490/-103

Expand verification into a fourteen-leg acceptance gate

• Adds upstream drift, warning-noise, credential-less simulation, and release/tag instrumentation legs while strengthening posture, scanners, timing, attribution, comment, mirror, and coverage checks. The verifier now records richer structured evidence and emits a compact mobile-friendly verdict comment with failures and disclosures separated.

.github/scripts/garnet-release-gate-verify.sh

Refactor (1) +5 / -3
garnet-jibril-release-gate.ymlConsolidate gate selector outputs +5/-3

Consolidate gate selector outputs

• Groups the selected tag, execution decision, and report pull-request outputs into one append operation. This preserves behavior while making output publication atomic and easier to audit.

.github/workflows/garnet-jibril-release-gate.yml

Documentation (1) +18 / -7
FAILURE_LEDGER.mdDocument the fourteen-leg gate and newly covered failures +18/-7

Document the fourteen-leg gate and newly covered failures

• Updates existing failure mappings and records upstream drift, warning noise, credential-less behavior, release instrumentation, timing, comment correctness, vacuous coverage, scanner dispositions, and evidence-mirror consistency. It also links the gate to the new shape analyzer and scanner disposition registry.

.github/garnet-gate/FAILURE_LEDGER.md

Other (2) +141 / -5
scanner-dispositions.ymlRegister accepted and intentional scanner findings +54/-0

Register accepted and intentional scanner findings

• Adds structured dispositions for known zizmor findings and intentional actionlint diagnostics. Unlisted findings can now fail the reviewer leg instead of relying on review-thread context.

.github/garnet-gate/scanner-dispositions.yml

garnet-jibril-release-gate-job.ymlRun upstream-aware and credential-less gate scenarios +87/-5

Run upstream-aware and credential-less gate scenarios

• Aligns reproduce-step ordering with pnpm's test workflow and adds an Ubuntu credential-less simulation. The verify job now installs actionlint and PyYAML, fetches current upstream workflows, and passes upstream and simulation evidence into the expanded verifier.

.github/workflows/garnet-jibril-release-gate-job.yml

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 8 potential issues.

Devin Review

Comment thread .github/scripts/garnet-gate-shape.py Resolved
Comment thread .github/scripts/garnet-gate-shape.py Resolved
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

/agentic_review

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

@coderabbitai review

@qodo-code-review

qodo-code-review Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (12) 📘 Rule violations (4) 📜 Skill insights (0)

Grey Divider


Action required

1. A compromised package can alter pull requests 🐞 Bug ⛨ Security ⭐ New
Description
The new PyYAML fallback installs an unpinned, unhashed package from the public package index in the
verify job. If that package release or its distribution path is compromised when a runner lacks
PyYAML, its install code receives the job’s automatically supplied token, which has `pull-requests:
write` and can modify the tracking pull request’s gate comment.
Code

.github/workflows/garnet-jibril-release-gate-job.yml[R259-260]

+      - name: Ensure PyYAML
+        run: python3 -c 'import yaml' || pip3 install --quiet --break-system-packages pyyaml
Relevance

●●● Strong

Recent workflow-security feedback was accepted when preventing excessive secret exposure; unpinned
package installation similarly warrants fixing.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The added fallback invokes pip3 install without a version or hash, while the same verify job is
granted pull-requests: write specifically to upsert a gate comment.

.github/workflows/garnet-jibril-release-gate-job.yml[232-235]
.github/workflows/garnet-jibril-release-gate-job.yml[259-260]
.github/workflows/garnet-jibril-release-gate-job.yml[317-332]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PyYAML fallback executes an unpinned package installation in a job with pull-request write authority.

## Issue Context
Use a version-pinned, hash-verified artifact or a prebuilt trusted environment; do not let an index’s current PyYAML release become executable workflow code.

## Fix Focus Areas
- .github/workflows/garnet-jibril-release-gate-job.yml[259-260]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Reviewers can trust altered evidence 🐞 Bug ⛨ Security ⭐ New
Description
L4 treats the PR-body mirror as identical to the trusted App comment whenever their first commit
markers match, without comparing the mirrored payload. If a PR author truncates or changes the
mirror while retaining the same head marker, the gate passes and reviewers consume content that is
not the verbatim trusted record.
Code

.github/scripts/garnet-release-gate-verify.sh[R305-309]

+    elif [ "$mirror_marker" != "$comment_marker" ]; then
+      mirror_state="mirror at ${mirror_marker:-no sha}, comment at ${comment_marker:-no sha}"
+      leg_fail "L4 app comment: evidence mirror and comment disagree — $mirror_state (F34)"
+    else
+      mirror_state="mirrored in the description, both bound to ${PR_HEAD_SHA:0:7}"
Relevance

●● Moderate

Evidence-integrity hardening is accepted, but no close precedent specifically requires comparing
mirrored payloads.

PR-#28
PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Static inference: the verifier extracts both bodies but compares only mirror_marker and
comment_marker; repository review policy defines the description section as a verbatim mirror of
the trusted Runtime Review comment.

.github/scripts/garnet-release-gate-verify.sh[296-310]
REVIEW.md[3-7]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L4 validates only that the evidence mirror and trusted App comment contain the same commit marker. A changed or truncated mirror can therefore pass despite no longer being a verbatim copy.

## Issue Context
Normalize the comment and the content inside the mirror delimiters, then compare their complete payloads. Continue validating the head-bound commit marker separately.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[296-310]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Action post failures escape the gate 🐞 Bug ☼ Reliability ⭐ New
Description
L12 and L13 pass only the main Run garnet-org/action@... slice and steps.sensor.outcome to
their checks, while step_log stops at the next regular run: group and the separately named post
step is considered only for timing. When Garnet's later upload or cleanup emits warnings or errors
or fails after a successful main invocation, both the trusted-token and credential-less legs can
pass without checking the action's complete lifecycle.
Code

.github/scripts/garnet-release-gate-verify.sh[R648-650]

+    action_log="$(step_log "$reproduce_log" "garnet-org/action@")"
+    warning_count="$(grep -c '##\[warning\]' <<<"$action_log" || true)"
+    error_count="$(grep -c '##\[error\]' <<<"$action_log" || true)"
Relevance

●● Moderate

Lifecycle coverage is a plausible reliability gap, but historical evidence does not clearly
establish acceptance for this scope.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Both new verification legs call step_log only for the main action group, and the parser stops that
slice at the next Run group, which the workflow places immediately after Garnet. The verifier then
relies on the main steps.sensor.outcome, while the simulation exports no post-step result;
meanwhile, L9's timeline lookup explicitly recognizes Post Garnet runtime monitoring as a distinct
later step for flush timing, confirming that post-action logs and status are outside the slices used
by L12 and L13.

.github/scripts/garnet-release-gate-verify.sh[640-665]
.github/scripts/garnet-release-gate-verify.sh[667-688]
.github/scripts/garnet-release-gate-verify.sh[560-575]
.github/workflows/garnet-jibril-release-gate-job.yml[200-221]
.github/scripts/garnet-release-gate-verify.sh[126-130]
.github/scripts/garnet-release-gate-verify.sh[648-656]
.github/scripts/garnet-release-gate-verify.sh[562-572]
.github/workflows/garnet-jibril-release-gate-job.yml[75-95]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The trusted-token warning check and credential-less check inspect only Garnet's main invocation. Update them to validate annotations and failures from both the main action and its separate post-action upload or cleanup hook.

## Issue Context
`step_log` ends the main action's log slice at the next ordinary `run:` group, so later post-step warnings and errors are omitted. The post step is part of the action lifecycle and is already recognized by L9 as measurable flush time, but the simulation exports only `steps.sensor.outcome`; read the post-step conclusion and log slice for both jobs, include its warnings and errors in L12/L13, and fail when the post hook does not complete successfully.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[126-130]
- .github/scripts/garnet-release-gate-verify.sh[560-575]
- .github/scripts/garnet-release-gate-verify.sh[562-572]
- .github/scripts/garnet-release-gate-verify.sh[640-665]
- .github/scripts/garnet-release-gate-verify.sh[648-656]
- .github/scripts/garnet-release-gate-verify.sh[667-688]
- .github/workflows/garnet-jibril-release-gate-job.yml[200-221]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View high (4)
4. Broad job permissions evade the gate 🐞 Bug ⛨ Security
Description
Static inference: permissions_of ignores job-level scalar permissions such as write-all and
falls back to the workflow-level mapping. If upstream adds permissions: write-all to the
instrumented job while retaining its current workflow-level contents: read, both the posture and
drift checks see the narrower inherited mapping and approve a job whose token has broad write
access.
Code

.github/scripts/garnet-gate-shape.py[R68-72]

+    for source in (job, doc or {}):
+        perms = source.get("permissions")
+        if isinstance(perms, dict) and perms:
+            return perms
+    return {}
Relevance

●●● Strong

Specific scalar permission handling can cause a security gate false negative; fail-closed permission
parsing matches recent gate hardening.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new parser returns only non-empty dictionaries, while the reusable workflow currently supplies
workflow-level contents: read; therefore a scalar job override is discarded and replaced by that
narrower mapping.

.github/scripts/garnet-gate-shape.py[66-72]
.github/scripts/garnet-gate-shape.py[312-316]
.github/workflows/test.yml[31-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The shape analyzer ignores scalar job-level permissions such as `read-all` and `write-all`, incorrectly falling back to workflow permissions and allowing broad job permissions to evade L6 and L11.

## Issue Context
GitHub job-level permissions override workflow-level permissions, including scalar forms. Preserve that override and normalize scalar values so posture checks can reject dangerous access.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[66-72]
- .github/scripts/garnet-gate-shape.py[207-221]
- .github/scripts/garnet-gate-shape.py[312-316]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. Upstream attribution drift can still pass the gate 🐞 Bug ≡ Correctness
Description
The L11 comparisons reduce the complete ordered pre-workload run: step layout to whether any
id-bearing step exists and whether the immediate predecessor is id-less, even though attribution()
depends on step names, identifiers, counts, and order. Upstream can add, remove, or reorder earlier
steps while preserving both booleans, changing Jibril's recorded __run_N attribution without
reaching the drift failure.
Code

.github/scripts/garnet-gate-shape.py[R317-325]

+    compare(
+        "id-bearing run step before the workload",
+        bool(upstream_shape["id_bearing_before"]),
+        bool(gate_shape["id_bearing_before"]),
+    )
+    compare(
+        "id-less run step immediately before the workload",
+        upstream_shape["immediately_before_has_id"] is False,
+        gate_shape["immediately_before_has_id"] is False,
Relevance

●●● Strong

Boolean-only drift checks miss meaningful ordered-step attribution changes; upstream-shape
verification is the PR’s stated intent.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The parser retains the exact ordered run_steps, including lists of id-bearing and id-less steps,
and calculates detailed attribution from that ordering because id-bearing steps affect Jibril
numbering. However, L11 determines drift using only bool(id_bearing_before) and the immediate
predecessor's id state; the resulting attribution is emitted into the record but never compared, so
distinct layouts can be treated as equivalent.

.github/scripts/garnet-gate-shape.py[75-116]
.github/scripts/garnet-gate-shape.py[120-135]
.github/scripts/garnet-gate-shape.py[317-346]
.github/workflows/test.yml[118-180]
.github/scripts/garnet-gate-shape.py[317-326]
.github/workflows/garnet-jibril-release-gate-job.yml[97-105]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L11 currently reduces the upstream and reproduced run-step layouts to two boolean properties: whether any earlier id-bearing step exists and whether the immediate predecessor is id-less. Update the drift check so changes to the number, identity, or order of earlier `run:` steps cannot pass when they alter Jibril's `__run_N` attribution.

## Issue Context
The parser already emits ordered `run_steps`, retains the relevant ordered names through `id_bearing_before`, and calculates attribution for both the upstream workflow and the gate reproduction. Compare a normalized ordered sequence that includes each step's id-bearing state rather than reducing the sequence to booleans. Current pnpm's reusable test job has an id-bearing `Determine test scope` step followed by id-less validation and test steps, while the gate reproduction includes additional id-bearing clock steps and deliberately models this ordering because id-bearing steps affect attribution.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[84-135]
- .github/scripts/garnet-gate-shape.py[317-326]
- .github/workflows/garnet-jibril-release-gate-job.yml[71-127]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


6. Unsafe workflow changes can clear the scanner gate 🐞 Bug ⛨ Security
Description
L8 discards both zizmor and actionlint exit statuses, while count_findings maps invalid or
non-array zizmor output to zero and the actionlint classifier counts only diagnostic-shaped lines
from potentially empty output. A scanner crash, malformed or unreadable report or disposition file,
failed analysis, or classifier exception can therefore yield no failure reason and zero unexpected
findings, allowing the scanner leg to approve workflow changes without actually inspecting them.
Code

.github/scripts/garnet-release-gate-verify.sh[R434-437]

+  zizmor --format json-v1 --no-exit-codes --offline "${scan_files[@]}" > /tmp/zizmor.json 2>/tmp/zizmor.err || true
+  zizmor_findings="$(count_findings /tmp/zizmor.json)"
+  zizmor --format json-v1 --no-exit-codes --offline --persona pedantic "${scan_files[@]}" > /tmp/zizmor-pedantic.json 2>/dev/null || true
+  zizmor_pedantic="$(count_findings /tmp/zizmor-pedantic.json)"
Relevance

●●● Strong

Suppressed scanner failures and malformed-output-as-zero behavior create a direct fail-open security
bypass.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The scanner commands use unconditional error suppression, after which the zizmor parser treats any
report that is not a JSON array as zero and the actionlint path fails only when its classifier
output has a positive diagnostic-line count. No later L8 check distinguishes successful empty scans
from execution, input, or classifier failures; by contrast, the zizmor classifier directly above
preserves and checks its exit status, demonstrating the intended fail-closed pattern that the
actionlint path lacks.

.github/scripts/garnet-release-gate-verify.sh[432-437]
.github/scripts/garnet-release-gate-verify.sh[477-493]
.github/scripts/garnet-release-gate-verify.sh[497-517]
.github/scripts/garnet-release-gate-verify.sh[440-484]
.github/scripts/garnet-release-gate-verify.sh[497-521]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L8 treats scanner invocation, parsing, and classification failures as empty reports: the `zizmor` and `actionlint` paths suppress nonzero statuses, invalid zizmor JSON is counted as zero findings, and actionlint classifier failures can leave no counted diagnostic lines. Update the gate so scanner infrastructure failures cannot pass as successful scans.

## Issue Context
Preserve the expected nonzero statuses that scanners use to report ordinary findings so diagnostics can still be collected and classified. Separately detect scanner execution or setup failures, malformed or unreadable reports, unreadable disposition files, and classifier exceptions, and ensure these conditions call `leg_fail` rather than being recorded as zero findings; the existing zizmor classifier status handling provides the intended fail-closed pattern for classification.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[432-437]
- .github/scripts/garnet-release-gate-verify.sh[497-521]
- .github/workflows/garnet-jibril-release-gate-job.yml[246-260]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


7. Mutable sensor releases pass as pinned 🐞 Bug ⛨ Security
Description
Static inference: posture accepts every non-empty jibril_version, including the mutable value
latest, as a pinned sensor release. Because sensor-version differences are deliberately exempted
from L11 drift, an upstream or gate step can select latest and still clear the security posture
intended to keep the root eBPF binary immutable under a fixed action SHA.
Code

.github/scripts/garnet-gate-shape.py[R234-238]

+    add(
+        "sensor version pinned",
+        bool(version) or default_version not in ("", "latest", "unknown"),
+        f"jibril_version: {version}" if version
+        else f"action default at {ref[:7]} resolves to {default_version or 'unknown'}",
Relevance

●●● Strong

Accepting latest as pinned directly defeats the stated immutable sensor-version security
requirement.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The acceptance expression is bool(version) or ..., so any non-empty value bypasses the explicit
rejection list. The drift comparison always marks a differing sensor version deliberate, leaving
posture as the only pinning enforcement.

.github/scripts/garnet-gate-shape.py[230-239]
.github/scripts/garnet-gate-shape.py[300-306]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The posture check treats any explicit sensor-version string as pinned, allowing `latest` and other mutable selectors to pass.

## Issue Context
Validate explicit inputs against the immutable release format expected by Jibril, and reject `latest`, empty, unknown, expressions that do not resolve to the tested release, and other floating selectors.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[230-239]
- .github/scripts/garnet-gate-shape.py[300-306]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

8. One helper doc omits its contract 📘 Rule violation ⚙ Maintainability ⭐ New
Description
posture documents its role as producing reviewer-facing lines but does not specify the accepted
workflow shape, inherited-permission behavior, or guarantees of the returned finding records. When
its callers or result consumers change, they cannot tell which missing values are supported or which
checks the result always contains.
Code

.github/scripts/garnet-gate-shape.py[R207-208]

+def posture(job: dict, name: str, doc: dict | None = None, default_version: str = "") -> list[dict]:
+    """The lines a pnpm reviewer and their scanners read on an instrumented job."""
Relevance

●●● Strong

Documentation contract omissions are typically accepted, especially for new helpers with explicit
behavioral outputs.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 2757165 requires documentation to state preconditions and postconditions rather than
merely narrating implementation. The added posture docstring only describes the lines it produces
and omits the function's behavioral contract.

Rule 2757165: Document preconditions and postconditions instead of narrating implementation details
.github/scripts/garnet-gate-shape.py[207-208]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The `posture` docstring narrates the helper's role without documenting its input expectations, inheritance behavior, and output guarantees.

## Issue Context
Callers need to know how absent jobs, permissions, action steps, and default versions affect the returned finding records.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[207-208]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


9. A warning comment preserves history 📘 Rule violation ⚙ Maintainability ⭐ New
Description
The L12 comment records that the fail-open message is the only warning pnpm has previously shown
instead of describing only the current annotation policy. When warning behavior changes, this
incident-focused statement can become stale and mislead maintainers about why the gate rejects
annotations.
Code

.github/scripts/garnet-release-gate-verify.sh[R640-643]

+# --- L12 warning noise ---------------------------------------------------------
+# On a run that has a token, an annotation from the action step is a defect: the
+# only warning pnpm has ever been shown there is the fail-open one of F2, and a
+# reader who learns to skip warnings skips that one too (F27).
Relevance

●●● Strong

Removing historical rationale from source comments is a straightforward maintainability fix with
little implementation risk.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 2757166 prohibits source comments that narrate prior behavior instead of current
constraints. The added comment says what warning pnpm has historically been shown and ties the
implementation rationale to that past observation.

Rule 2757166: Avoid historical or refactor-focused commentary in source comments
.github/scripts/garnet-release-gate-verify.sh[640-643]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The L12 source comment embeds historical warning behavior rather than limiting itself to the current gate constraint and rationale.

## Issue Context
Historical incident details belong in the failure ledger or commit history; the source comment should explain why token-path annotations currently fail the gate.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[640-643]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


10. Repeated skip messages still pass 🐞 Bug ≡ Correctness ⭐ New
Description
L13 only rejects sim_info below one, even though the new leg's contract requires exactly one
missing-token information line. Any action release that repeats the skip notice two or more times
therefore clears the credential-less gate and reintroduces the output noise this leg is meant to
prevent.
Code

.github/scripts/garnet-release-gate-verify.sh[R682-683]

+  elif [ "$sim_info" -lt 1 ]; then
+    leg_fail "L13 credential-less: the action skipped silently; a credential-less run has to say why (F28)"
Relevance

●●● Strong

The requirement says exactly one line, while the current condition allows duplicates; this is a
deterministic correctness fix.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Static inference: the leg documents a one-line requirement and counts matching lines, but its
condition fails only when the count is less than one.

.github/scripts/garnet-release-gate-verify.sh[667-684]
.github/garnet-gate/FAILURE_LEDGER.md[62-62]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The credential-less leg accepts any positive number of missing-token messages instead of exactly one.

## Issue Context
Change the assertion to require `sim_info == 1` and report the observed count when it differs.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[667-684]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (6)
11. A workflow comment preserves history 📘 Rule violation ⚙ Maintainability
Description
dependabot_sim's comment embeds the historical upstream pull number and its old log text even
though the adjacent lines already state the current empty-token behavior. When that incident or
message becomes stale, a later maintainer can mistake release history for a live workflow constraint
while changing the simulation.
Code

.github/workflows/garnet-jibril-release-gate-job.yml[R188-191]

+  # What a Dependabot-actor or fork-head run gets: no secret reaches the job,
+  # so the action receives an empty api_token. pnpm sees this path on every
+  # Dependabot PR (upstream pull 13270 logged `Input 'api_token' is required`),
+  # and the only acceptable behaviour there is a clean skip. A simulation, not
Relevance

●●● Strong

The comment explicitly embeds historical incident details rather than only current behavior and
constraints.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 2757166 prohibits source comments that preserve change history instead of describing
only current behavior and constraints. The cited workflow comment records what upstream pull 13270
previously logged before stating the current clean-skip requirement.

Rule 2757166: Avoid historical or refactor-focused commentary in source comments
.github/workflows/garnet-jibril-release-gate-job.yml[188-193]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Remove the historical pull reference and old log quotation from the workflow comment while retaining the current credential-less behavior and simulation limitation.

## Issue Context
Incident history belongs in the failure ledger or commit history; the workflow comment should explain only the current operational contract.

## Fix Focus Areas
- .github/workflows/garnet-jibril-release-gate-job.yml[188-193]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


12. Release instrumentation can be omitted from the gate 🐞 Bug ⛨ Security
Description
release_report() calls action_step(job) only once even though that helper returns at its first
Garnet match, then labels recognized Linux runners as producing a profile from the runner name
without checking credentials or disabling conditions, despite L14 claiming to report every Garnet
step. When a release job contains a later action step, lacks api_token, is conditionally disabled,
or otherwise cannot execute, the generated report still appears covered and the verifier only
rejects reported entries with unknown runners, allowing inert instrumentation to pass L14.
Code

.github/scripts/garnet-gate-shape.py[R181-185]

+        doc = load(path)
+        for job_name, job in jobs(doc).items():
+            step = action_step(job)
+            if step is None:
+                continue
Relevance

●●● Strong

Only inspecting the first step and assuming Linux success creates clear instrumentation false
positives in a security gate.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
action_step() stops at the first matching action, and release_report() invokes it only once per
job, so subsequent Garnet steps never enter the generated report. For the one returned step,
release_report() unconditionally maps runners containing ubuntu or linux to `produces_profile:
yes`; the verifier trusts this generated list and only fails entries with unknown runners, even
though existing release instrumentation shows that an explicit API token is part of the required
step configuration.

.github/scripts/garnet-gate-shape.py[55-59]
.github/scripts/garnet-gate-shape.py[173-204]
.github/scripts/garnet-release-gate-verify.sh[691-708]
.github/workflows/update-latest.yml[193-201]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L14 examines only the first `garnet-org/action` step in each release or tag job and infers successful instrumentation from a recognized Linux runner name. As a result, additional, credential-less, conditionally disabled, or otherwise inert release steps can escape validation while the job is reported as producing a profile.

## Issue Context
`action_step()` returns immediately after its first match and is also used by posture checks, while `release_report()` calls it only once per job. Enumerate every matching action step, preserve a per-step identity in the release report, and validate the prerequisites needed to produce a profile, including credentials and disabling conditions, rather than assigning `produces_profile: yes` from the runner alone.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[55-59]
- .github/scripts/garnet-gate-shape.py[173-204]
- .github/scripts/garnet-gate-shape.py[207-240]
- .github/scripts/garnet-release-gate-verify.sh[691-708]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


13. Releases can pass without flush timing 🐞 Bug ➹ Performance
Description
When the job API has no matching post-action timeline entry, L9 appends `post-step flush not
measured without calling leg_fail`. Because the final verdict depends solely on accumulated
failures, a missing post step or unreadable flush duration can pass despite L9 claiming to budget
post-step latency.
Code

.github/scripts/garnet-release-gate-verify.sh[R568-575]

+if [ -n "$flush_seconds" ]; then
+  flush_seconds="${flush_seconds%.*}"
+  timing_state="$timing_state · post-step flush ${flush_seconds}s (budget ${FLUSH_BUDGET_SECONDS}s)"
+  [ "$flush_seconds" -le "$FLUSH_BUDGET_SECONDS" ] \
+    || leg_fail "L9 timing: post-step flush ${flush_seconds}s exceeds ${FLUSH_BUDGET_SECONDS}s on the light cell (F30)"
+else
+  timing_state="$timing_state · post-step flush not measured"
+fi
Relevance

●●● Strong

Missing flush duration bypasses an explicitly budgeted timing check; required evidence should fail
closed.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The only branch that enforces the flush budget is entered when a duration exists; the
missing-duration branch merely records text. The final verdict is PASS whenever no leg added a
failure reason.

.github/scripts/garnet-release-gate-verify.sh[562-575]
.github/scripts/garnet-release-gate-verify.sh[711-713]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L9 does not fail when the post-action flush duration cannot be obtained. This allows a PASS verdict without the new flush-budget evidence.

## Issue Context
The flush is read from the GitHub job timeline by an exact post-step name. Treat absent or non-numeric timing as an L9 failure, and retain a clear diagnostic that distinguishes an absent post step from an API/read failure.

## Fix Focus Areas
- .github/scripts/garnet-release-gate-verify.sh[562-575]
- .github/scripts/garnet-release-gate-verify.sh[711-713]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


14. Attribution docs hide the contract 📘 Rule violation ⚙ Maintainability
Description
attribution documents GitHub and Jibril's internal numbering algorithm but omits the accepted
job and workload_name shape and the {"found": false} missing-workload result. When either
workflow structure changes, callers cannot determine from the function contract which result fields
are guaranteed or how the missing case must be handled.
Code

.github/scripts/garnet-gate-shape.py[R87-90]

+    GitHub sets GITHUB_ACTION to the step id when there is one and numbers only
+    the id-less run steps (__run, __run_2, ...). Jibril's parseStepsList never
+    reads `id:` and numbers every run step, so an id-bearing run step earlier in
+    the job shifts every later id-less step back by one (ledger F25).
Relevance

●●● Strong

The docstring omits required input and output contracts; this is a straightforward maintainability
improvement aligned with contract-focused review.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 2757165 requires documentation to state preconditions and postconditions instead of
predominantly narrating implementation details. The cited docstring explains the two numbering
algorithms and their skew but does not document the function's input assumptions or its reduced
result shape when no workload matches.

Rule 2757165: Document preconditions and postconditions instead of narrating implementation details
.github/scripts/garnet-gate-shape.py[84-103]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Replace the implementation-focused `attribution` docstring with a contract describing input expectations, output guarantees, and the missing-workload case.

## Issue Context
The function returns two distinct result shapes depending on whether the workload is found, but its documentation currently narrates the numbering implementation instead of explaining that contract.

## Fix Focus Areas
- .github/scripts/garnet-gate-shape.py[84-103]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


15. Invalid scanner classes still pass 🐞 Bug ⚙ Maintainability
Description
Static inference: the zizmor classifier treats any matching disposition row as classified by copying
its class without validating the documented four-value vocabulary. A typo, missing class, or
unsupported value in the new ledger therefore suppresses the finding instead of failing L8.
Code

.github/scripts/garnet-release-gate-verify.sh[R465-475]

+        match = next(
+            (
+                row for row in rows
+                if row.get("ident") == key[0]
+                and row.get("path") == path
+                and ("route" not in row or row.get("route") == route)
+            ),
+            None,
+        )
+        state = match.get("class") if match else "unclassified"
+        print(f"{state}\t{key[0]} at {path}{' ' + route if route else ''}")
Relevance

●●● Strong

Unvalidated disposition classes can suppress scanner findings, contradicting the documented
vocabulary and fail-closed gate design.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new disposition file states that exactly four classes are allowed, but the classifier considers
a finding unclassified only when no row matches and never checks the matched class value.

.github/garnet-gate/scanner-dispositions.yml[3-15]
.github/scripts/garnet-release-gate-verify.sh[465-479]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The scanner classifier accepts any class value on a matching disposition row, contrary to the disposition file's four-class contract.

## Issue Context
Validate the complete YAML schema before matching findings. Reject missing or unknown classes, malformed rows, duplicates, and invalid optional routes with a failing L8 result.

## Fix Focus Areas
- .github/garnet-gate/scanner-dispositions.yml[3-15]
- .github/scripts/garnet-release-gate-verify.sh[443-475]
- .github/scripts/garnet-release-gate-verify.sh[477-488]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


16. Sensor overhead is measured against unrelated work 🐞 Bug ➹ Performance
Description
L9 subtracts the baseline's two network operations from REPRODUCE_WORKLOAD_SECONDS, but that
duration starts before the new validation step and ends before the assertion step's status-file
polling and diagnostics complete. The resulting value includes gate bookkeeping that the baseline
never runs, so releases can be rejected for apparent workload overhead unrelated to the sensor.
Code

.github/scripts/garnet-release-gate-verify.sh[R552-556]

if [ -n "$action_seconds" ] && [ -n "${REPRODUCE_WORKLOAD_SECONDS:-}" ] && [ -n "${BASELINE_WORKLOAD_SECONDS:-}" ]; then
-  overhead_seconds=$(( action_seconds + REPRODUCE_WORKLOAD_SECONDS - BASELINE_WORKLOAD_SECONDS ))
-  timing_state="sensor start ${action_seconds}s · workload ${REPRODUCE_WORKLOAD_SECONDS}s with sensor vs ${BASELINE_WORKLOAD_SECONDS}s without · overhead ${overhead_seconds}s (budget ${OVERHEAD_BUDGET_SECONDS}s)"
-  [ "$overhead_seconds" -le "$OVERHEAD_BUDGET_SECONDS" ] || leg_fail "L9 timing: overhead ${overhead_seconds}s exceeds ${OVERHEAD_BUDGET_SECONDS}s"
+  overhead_seconds=$(( REPRODUCE_WORKLOAD_SECONDS - BASELINE_WORKLOAD_SECONDS ))
+  timing_state="sensor start ${action_seconds}s (budget ${START_BUDGET_SECONDS}s) · workload ${REPRODUCE_WORKLOAD_SECONDS}s with sensor vs ${BASELINE_WORKLOAD_SECONDS}s without, overhead ${overhead_seconds}s (budget ${OVERHEAD_BUDGET_SECONDS}s)"
+  [ "$action_seconds" -le "$START_BUDGET_SECONDS" ] || leg_fail "L9 timing: sensor start ${action_seconds}s exceeds ${START_BUDGET_SECONDS}s"
+  [ "$overhead_seconds" -le "$OVERHEAD_BUDGET_SECONDS" ] || leg_fail "L9 timing: workload overhead ${overhead_seconds}s exceeds ${OVERHEAD_BUDGET_SECONDS}s"
Relevance

●● Moderate

Timing boundaries may be semantically mismatched, but the intended workload-overhead redesign makes
acceptance uncertain.

PR-#43

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The baseline measures only curl and node fetch. The reproduce clock begins before validation, while
the assertion records the end around the status-check path, and L9 directly subtracts those unlike
measurements.

.github/workflows/garnet-jibril-release-gate-job.yml[38-43]
.github/workflows/garnet-jibril-release-gate-job.yml[87-107]
.github/workflows/garnet-jibril-release-gate-job.yml[129-175]
.github/scripts/garnet-release-gate-verify.sh[552-556]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
L9's workload-overhead subtraction uses different timed regions for the reproduce and baseline jobs. The reproduce duration contains validation and assertion/polling work that is absent from the baseline.

## Issue Context
The workload clock is written immediately after the action and the assertion captures its end before performing status polling. Time only the two egress operations in reproduce, or make the baseline execute the same surrounding work before enforcing a budget.

## Fix Focus Areas
- .github/workflows/garnet-jibril-release-gate-job.yml[87-107]
- .github/workflows/garnet-jibril-release-gate-job.yml[129-175]
- .github/scripts/garnet-release-gate-verify.sh[552-556]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 36 rules
✅ REVIEW.md
Review mode: 🧠 Deep: This is a large, security-sensitive CI gate redesign spanning multiple independent scripts and workflows, with numerous subtle validation and fail-open paths that materially benefit from redundant review passes.
ⓘ  3 issues published inline · 6 in summary

Grey Divider

Tip of the day
💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread .github/scripts/garnet-gate-shape.py Resolved
Comment thread .github/scripts/garnet-gate-shape.py Resolved
Comment thread .github/scripts/garnet-gate-shape.py Resolved
@qodo-code-review

Copy link
Copy Markdown

PR Code Suggestions ✨

Warning

/improve is deprecated. Use /agentic_review instead (removal date not yet scheduled).

No code suggestions found for the PR.

@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit d40afff

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

Jibril release gate: FAIL

v2.17.0-rc.10 on garnet-org/action@e546567: 7 failing check(s) below, each named with its ledger row.

leg what it found
PASS L1 startup PASS — sensor active, status ok, github steps resolution ok →
PASS L2 profile uploaded after 0s · 01a086ed-29f7-719f-853a-b06652dea8fe · 3 destinations →
FAIL L3 workload destination present, but attributed to steps ["2. Stop the action clock"] instead of "Workload … →
FAIL L4 App comment final after 0s · missing the recorded step name in quotes; a destination · absent from the pull… →
PASS L5 permalink html 200 · api 200 →
FAIL L6 posture upstream test: sensor version pinned — action default at 3d47f4a resolves to latest
FAIL L7 pnpm's own run TS CI run 34374699364 finished success with no profile (no profile is bound to that run): the j…
FAIL L8 scanners 0 regular, 0 pedantic finding(s) over 2 workflow file(s); 0 without a written class · actionlin…
PASS L9 timing sensor start 15s (budget 30s) · workload 0s with sensor vs 1s without, overhead -1s (budget 120…
PASS L10 coverage 1 of 6 CI cells instrumented: ubuntu/node 24 · not instrumented: ubuntu/node 22, ubuntu/node 26…
PASS L11 upstream drift pnpm/pnpm at d3ab827 · 0 difference(s), 2 declared deliberate →
PASS L12 warning noise 0 warning(s), 0 error(s) from the action step; 0 are the disclosed fail-open text (F2) →
FAIL L13 credential-less (simulation) outcome success · 0 warning(s), 0 error(s), 0 info line(s) naming the missing token →
PASS L14 release and tag release.yml verify-ts-release on macos-latest: profile no (macos runner: the sensor is Linu…

Failing legs

Disclosed, not failing

  • the sensor is Linux x86_64 eBPF: macOS and Windows cells record nothing, and no change to this rollout makes them record
  • fork pull requests and Dependabot-actor runs get no secret, so they produce no server-side evidence; L13 is a simulation of that path, not evidence for it
  • the action fails open: a sensor that never starts leaves a green job (F2), which is why the gate reads the status file and the profile rather than the job conclusion
  • no_bad_egress_domain has been attention-only in every pnpm profile and has never had a deny rule to fail against: it is not a proof of anything until pnpm has deny rules in the control plane
  • L13 is a simulation on this repository's own event with a read-only token, not a Dependabot run: it shows what the action does without a token, and no server-side evidence exists for Dependabot or fork pull requests (F11, F12)
  • release.yml runs a Garnet step in verify-ts-release on macos-latest and records nothing: macos runner: the sensor is Linux-only eBPF (F13)
  • release.yml runs a Garnet step in release on macos-latest and records nothing: macos runner: the sensor is Linux-only eBPF (F13)

Record

field value
jibril requested v2.17.0-rc.10 · resolved v2.17.0-rc.10
action garnet-org/action@e546567a72e4 · empty jibril_version there resolves to v2.16.0
run 34374702332 attempt 1 · blacksmith-01m23dgf4c4rjvgy29d7k46g4n-8vcpu · pull_request
pnpm at main d3ab827 · action 3d47f4a9004f → latest
workflow ref seen by sensor garnet-labs/pnpm/.github/workflows/garnet-jibril-release-gate.yml@refs/pull/48/merge
base → head ab4a9cd → d40afff
merge sha at run 6d147a5 · now b1a9db5 · profile 6d147a5
comparison no previous (first record on this PR)
synthetic merge yes · sensor recorded merge commit 6d147a5, PR head is d40afff
permalink https://app.garnet.ai/public/runs/34374702332?profile=01a086ed-29f7-719f-853a-b06652dea8fe

Legs map to past failures in .github/garnet-gate/FAILURE_LEDGER.md. Full record in the run's garnet-release-gate artifact.

Comment thread .github/scripts/garnet-gate-shape.py Resolved
Comment thread .github/scripts/garnet-gate-shape.py Resolved
Comment thread .github/scripts/garnet-gate-shape.py Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

garnet-release-testing Garnet product/release test PR; the Jibril release gate runs and reports here

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant