Repository navigation
Check the security mutation registry for drift on every PR - #553
Merged
Merged
Conversation
The registry in security-mutations.json pins each target function's source marker and SHA-256, its mutation anchors and its selected tests. Until now it was only loaded by the nightly security-mutations workflow, so a pull request that edited one of the registered functions passed PR CI and broke the next scheduled campaign, after the change had merged. Add a --validate-registry mode to scripts/security_mutations.py. It runs the campaign's own manifest loader against the checked-out tree and fails closed on the same drift categories (function scope and hash, replacement anchors, test name rules, stale exceptions), but never compiles, discovers tests or applies a mutant. Its report says registry_valid with zero counts so it cannot pass for a mutation score, and it cannot be combined with --calibrate. The manifest containment check is shared with the campaign so both reject the same paths. Run the check, and the workflow contract tests that assert it exists, in the existing required supply-chain job. The required-check and image-promotion lists stay unchanged, and the full campaign remains scheduled and manual only. Document the pull-request reconciliation rule and what the check cannot see: renamed tests and new survivors still need the compiled binary and the scheduled campaign.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Antnation
added a commit
that referenced
this pull request
Oct 15, 2026
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #436
Why
security-mutations.jsonpins each target function's start marker and SHA-256, its mutation anchors and its selected tests. Only the nightlysecurity-mutations.ymlcampaign ever loaded it, so a PR that edited one of the four registered functions passed PR CI and broke the next scheduled run after merge.What
scripts/security_mutations.py --validate-registry: runs the campaign's ownload_manifestagainst the checked-out tree, without a cargo build, test discovery or mutation. It fails closed on the same categories the campaign would hit before compiling:function_scope_drift,function_hash_drift,invalid_replacement, test-name syntax, count and duplication,stale_exception, version or schema, unsafe paths andmanifest_outside_repository.reason: registry_validandsource_state: checkout_registry_only, with zero counts and no results, so it cannot be mistaken for a mutation score.--calibrate(mutually exclusive argparse group).repository_manifest), so the campaign and the PR check accept and reject the same registry paths. Campaign behaviour is unchanged.CI: two steps added to the existing required
supply-chainjob, which already checks out the repo, sets up the pinned Python and installs PyYAML:python scripts/test_security_mutations_workflows.py, so removing or weakening the new step fails the same PR.python scripts/security_mutations.py --validate-registry.No job is added, so
promote-image.needsand the required-check list stay the same. The full campaign and the controller tests, which include a real rustc calibration, remain scheduled and manual only.Tests:
test_security_mutations.py: the current registry passes the drift check, and the check never starts a process (subprocess.Popenis patched to fail). The report carries identity but no score. It fails closed on edited, renamed, duplicated, CRLF and removed target source, and on unreconciled registry entries (version, duplicate or invalid test names, missing anchor, stale exception). It rejects a registry outside the repository and refuses--calibratealongside.test_security_mutations_workflows.py: the step exists insupply-chain, which has noif,needsorcontinue-on-errorand stays inpromote-image.needs. The step follows tool setup and the PyYAML install. The onlysecurity_mutationscommands anywhere inci.ymlare the contract test and--validate-registry, so the campaign never runs on PRs. The scheduled controller command never uses--validate-registry.Docs:
docs/testing/security-mutations.mdgains a "Pull-request registry check" section. It covers the PR-time reconciliation rule, how the function digest is computed (with a snippet that reproduces the registered hash), the CRLF caveat, and what the check cannot see: renamed tests, survivors, unviable mutants and timeouts still need the compiled binary and the scheduled campaign.security-coverage.mdis updated so it no longer says no PR requirement changed.Verification
The gateway crate cannot build locally (Node pin), and this Windows checkout uses CRLF, which changes function hashes. So I exported LF trees with
git -c core.autocrlf=false archiveand ran them with a local-only shim that stubs the Linux-onlyresourceimport:--validate-registrypasses on main's tree and on this branch: 4 targets, 8 registered mutants,registry_valid.path_prefix_matches) fails withfunction_hash_drift, and renaming it fails withfunction_scope_drift. Restoring it passes again. The doc snippet reproduces the registered digest.SIGKILL, chmod, real rustc), and they fail identically on main's tree.ci.ymlvariants all fail the workflow contracts: step removed,continue-on-error, conditional step, and the campaign added to PR CI.test_build_tools.py,build_tools.py check,test_security_corpus_workflows.pyandtest_image_publication.pypass.