Repository navigation
ci(garnet): decide upstream drift, warning noise and the credential-less path in the gate - #48
Conversation
…ess path in the gate The gate had ten legs and read only what the fork's own run produced. Four more legs read what a pnpm maintainer would have to check by hand, and the ones that existed now decide instead of report. L11 reads pnpm/pnpm's ci.yml, test.yml, release.yml and update-latest.yml at main on every run and fails when the reproduced shape stops matching pnpm's: the reusable-call form, the instrumented runner, the action ref, the sensor version, the explicit api_token, the job permissions and the run-step order around the workload. Two differences are declared deliberate (the candidate action ref and the release tag under test); anything else is drift. L12 counts the annotations of the Garnet action step and fails any warning on a token-carrying run, naming the disclosed fail-open text apart from the rest. L13 runs the action with an empty api_token on ubuntu-24.04 and requires a green job with one info line and no warning; it is labelled a simulation, and the comment says no server-side evidence exists for Dependabot or fork runs. L14 reports every Garnet step in the release and tag workflows with its runner and whether it can record: macOS is a disclosure, inert on Linux is a failure. The reproduce job now carries the same run-step order pnpm's test job has, an id-bearing step then an id-less one before the workload, so the attribution skew the gate measures is the skew pnpm gets. L3 names it: Jibril step numbering skew, garnet-org/jibril parseStepsList. L6 reads the action source at the pinned sha to learn what an empty jibril_version resolves to there, so a floating default cannot hide behind a pinned action. L9 splits timing into sensor start, workload overhead and post-step flush, each with its own budget. L4 requires one App comment, updated in place, bound to the current head, carrying the recorded step name in quotes and a destination, and matching the evidence mirror in the PR body. L8 classifies every zizmor finding through a written disposition file and fails an unclassified one. L10 marks no_bad_egress_domain vacuous. Ledger rows F26 to F34 cover the new legs.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe release gate now evaluates fourteen legs. It parses upstream and local workflow shapes, classifies scanner findings, tests credential-less execution, measures sensor timing, checks release workflows, validates evidence mirrors, and renders an expanded verdict. ChangesRelease gate verification
Estimated code review effort: 4 (Complex) | ~60 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant GateWorkflow
participant VerifyScript
participant ShapeScript
participant ScannerTools
participant PullRequest
GateWorkflow->>VerifyScript: provide upstream and simulation evidence
VerifyScript->>ShapeScript: compare local and upstream workflow shapes
ShapeScript-->>VerifyScript: return posture, drift, attribution, and release facts
VerifyScript->>ScannerTools: scan workflows and classify findings
ScannerTools-->>VerifyScript: return findings and dispositions
VerifyScript->>PullRequest: validate comment and evidence mirror
VerifyScript-->>GateWorkflow: publish verdict and record
Suggested reviewers: ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Execution Profiles recorded for 1 job, triggered by
|
PR Summary by QodoExpand Garnet gate with upstream-aware acceptance checks
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
|
/agentic_review |
|
@coderabbitai review |
Code Review by Qodo
1. A compromised package can alter pull requests
|
PR Code Suggestions ✨Warning
No code suggestions found for the PR. |
|
Code review by qodo was updated up to the latest commit d40afff |
Jibril release gate: FAIL
Failing legs
Disclosed, not failing
Record
Legs map to past failures in |
Summary
This is fork-only infrastructure: it changes the Garnet/Jibril release gate that decides whether a Garnet action release or a Jibril repin is safe to take into
pnpm/pnpm. No pnpm CLI code changes.Before this PR the gate had ten legs and every one of them read only what the fork's own run produced. A run could be green while pnpm's real workflows had drifted away from the shape being reproduced, while the action step emitted warnings, and while nobody had ever exercised the credential-less path a fork PR or Dependabot takes. Four legs were added and the existing ones now decide rather than describe.
New legs:
pnpm/pnpm'sci.yml,test.yml,release.ymlandupdate-latest.ymlatmainon every run and compares the reproduced job against them through.github/scripts/garnet-gate-shape.py: reusable-call form ($/vs./), the runner of thegarnet: truecell, the action ref,jibril_version, the explicitapi_token, job permissions, and whether an id-bearingrun:step sits before the workload. Two differences are declared deliberate (the candidate action ref and the release under test); anything else fails. The upstream SHA goes into the record.Garnet runtime monitoringstep, counts::warning::/::error::, and fails any warning on a token-carrying run. The disclosed fail-open warning (F2) is named separately from an unexpected one.ubuntu-24.04job runs the action withapi_token: ''and a read-onlyGITHUB_TOKEN. It must exit 0, print one info line naming the missing token, emit no warning and no error. It is labelled a simulation, and the comment states that there is still no server-side evidence for real Dependabot or fork PRs.Changed legs:
run:step, then an id-less one, then the workload, exactly as pnpm's test job does, so the attribution the gate measures is the attribution pnpm gets. When the egress lands one step early the leg says so by name:Jibril step-numbering skew, garnet-org/jibril parseStepsList(F25).id-token: writein the production-shaped job, nosecrets: inherit, no permission additions beyond what the gate itself needs, explicitapi_token, full 40-char SHA pins, and — read from the action source at the pinned SHA — the version an emptyjibril_versionresolves to there. A floating default cannot hide behind a pinned action.zizmorfinding (regular and--pedantic) through.github/garnet-gate/scanner-dispositions.ymlas fixed / disclosed / intentional-diagnostic / garnet-side and fails an unclassified one;actionlintdeliberate warnings are whitelisted by message rather than by switching the rule off; unresolved bot review threads under.github/**fail.no_bad_egress_domainvacuous — not a proof of anything — until pnpm has deny rules in the control plane..github/garnet-gate/FAILURE_LEDGER.mdgains F26–F34 (upstream drift, warning noise, credential-less behaviour, release/tag inertness, flush timing, duplicate/mis-bound comment, vacuous assertion, unclassified finding, mirror/head mismatch) and every pre-existing row now names the leg that covers it.The
$/caller form and theblacksmith-8vcpu-ubuntu-2404runner are kept on purpose: they are the two shapes pnpm actually uses and the ones that broke Jibril, so the gate reproduces them and classifies the scanner noise they cause instead of hiding it.Squash Commit Body
Checklist
are implemented in every affected version. (No CLI code changed; CI only.)
bash -n,shellcheck,actionlint,zizmorregular and--pedantic, plus a local verifier dry run.)Link to Devin session: https://app.devin.ai/sessions/a29aac85f1f142a6ab5006dc8657db1c
Open in Devin Desktop: https://app.devin.ai/desktop/session/a29aac85f1f142a6ab5006dc8657db1c?variant=devin
Requested by: @jadoonf
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit
New Features
Bug Fixes
Runtime evidence (Garnet)
Kernel-recorded execution record for head
d40afff36dcd786622523701996cdc2cff381940, mirrored verbatim fromthe sticky Garnet Runtime Review comment on this PR so reviewers that read only
the description ground in the same bytes. Facts only. Judgment stays with the
reviewer. Cite grounded findings as:
Execution record (verbatim mirror)
Execution Profiles recorded for 1 job, triggered by
d40afffGarnet Jibril Release Gate/reproduce↗ · 3 destinationsRunner.Worker └─ bash └─ curl (step: "Stop the action clock") ├─ ○ localhost (dns resolver) └─ ○ registry.npmjs[.]org systemd └─ systemd-resolved └─ ○ 8.8.8.8View this job's Execution Profile in Garnet →
💡 How to read this
Runner.Worker ← process on a path └─ npm └─ node ← process that acted └─ ○ npmjs[.]org ← observed actionfollow a path downward to see what ran and what it did — each path to an observed action is an execution chain
names on the path = processes · ○ = observed action · (…) = context
Greptile Summary
This change expands the release evidence checks, but several checks can still certify incomplete or inaccurate results. Broad workflow permissions, mutable sensor versions, attribution drift, incomplete mirrored evidence, unmeasured flush timing, and invalid scanner dispositions can all be accepted. The duplicate diagnostic-line concern is non-blocking, but should be corrected to keep the credential-less output contract precise.
Confidence Score: 0/5
Not safe to merge: the release validation can approve inaccurate security posture, instrumentation, evidence, timing, and scanner-disposition results.
Multiple independently reproduced blocking failures affect both the workflow-shape validator and the release verification script, including one that weakens credential-posture reporting.
Files Needing Attention: .github/scripts/garnet-gate-shape.py; .github/scripts/garnet-release-gate-verify.sh
Security Review
A valid scalar
permissions: write-allis interpreted as no permissions, allowing a broadly credentialed workflow to pass checks intended to reject OIDC and pull-request write access.What T-Rex did
Comments Outside Diff (6)
General comment
permissions_of()accepts only dictionary-valued permissions. A valid workflow-levelpermissions: write-allparses as the stringwrite-all, so the function returns{}.posture()consequently reportsid-token: write absentandno pull-requests: writeas passing even though GitHub grants those write permissions underwrite-all.isinstance(perms, dict), while lines 216 and 220 infer absence from the resulting empty dictionary.write-alland make both posture checks fail for it; ideally normalize all supported scalar forms (read-all,write-all, and{}) before performing scope-specific checks.General comment
jibril_versionvalue as pinned. A fixture passedjibril_version: latestand the actualposture()function reportedsensor_version_pinned.ok=True. At the identical scope, an omitted input whose default resolves tolatestreportsFalse.bool(version) or default_version not in ("", "latest", "unknown"); it validates presence rather than requiring an immutable version value whenjibril_versionis explicit.latest(and ideally validate explicit versions against the project’s immutable-version format) before accepting the sensor as pinned.General comment
run:step immediately before the retained id-less predecessor leaves both compared L11 booleans unchanged, sodrift_countremains 0. The focused run showed the workload's GitHub token remained__run_3, but Jibril attribution moved fromImmediate anonymous predecessorto the newly addedAdded identified setupstep.General comment
<!-- garnet:commit <40-hex> -->marker from the App comment and PR-description evidence block. In the executed pointer-only case, the PR mirror replaced the evidence content withEvidence body is replaced by a pointer: https://evidence.example/run/123but retained the same marker. The check returned exit 0 withFAIL_COUNT=0andMIRROR_STATE=mirrored in the description, both bound to 0123456.mirror_markerandcomment_markeronly; they do not compare the evidence body, integrity digest, canonical serialized content, or reject a pointer in place of copied evidence.General comment
sim_infois less than one ([ "$sim_info" -lt 1 ]) rather than requiring it to equal one.sim_infoequals1, and report duplicate informational output distinctly.General comment
class: invalid-class, noclassfield, andclass: fixedeach yieldedgate_unclassified=0andgate_result=accepted. This conflicts with the dispositions file’s stated four-class contract and withfixedmeaning the finding is gone.classvalue (including PythonNone) and count only output lines beginning withunclassified; there is no allowed-class validation and no prohibition onfixedfor a currently reported finding.fixedas unclassified/error whenever the finding is present in either live zizmor report; only absent findings may retain a historicalfixedrow.Reviews (1): Last reviewed commit: "ci(garnet): decide upstream drift, warni..." | Re-trigger Greptile