Skip to content

Check the security mutation registry for drift on every PR - #553

Merged
Antnation merged 1 commit into
mainfrom
ci/436-mutation-registry-drift
Oct 15, 2026
Merged

Antnation merged 1 commit into
mainfrom
ci/436-mutation-registry-drift

Conversation

@Antnation

Copy link
Copy Markdown
Contributor

Part of #436

Why

security-mutations.json pins each target function's start marker and SHA-256, its mutation anchors and its selected tests. Only the nightly security-mutations.yml campaign ever loaded it, so a PR that edited one of the four registered functions passed PR CI and broke the next scheduled run after merge.

What

  • scripts/security_mutations.py --validate-registry: runs the campaign's own load_manifest against the checked-out tree, without a cargo build, test discovery or mutation. It fails closed on the same categories the campaign would hit before compiling: function_scope_drift, function_hash_drift, invalid_replacement, test-name syntax, count and duplication, stale_exception, version or schema, unsafe paths and manifest_outside_repository.

    • It reports reason: registry_valid and source_state: checkout_registry_only, with zero counts and no results, so it cannot be mistaken for a mutation score.
    • It cannot be combined with --calibrate (mutually exclusive argparse group).
    • The manifest containment check is now a shared helper (repository_manifest), so the campaign and the PR check accept and reject the same registry paths. Campaign behaviour is unchanged.
  • CI: two steps added to the existing required supply-chain job, which already checks out the repo, sets up the pinned Python and installs PyYAML:

    1. python scripts/test_security_mutations_workflows.py, so removing or weakening the new step fails the same PR.
    2. python scripts/security_mutations.py --validate-registry.

    No job is added, so promote-image.needs and the required-check list stay the same. The full campaign and the controller tests, which include a real rustc calibration, remain scheduled and manual only.

  • Tests:

    • test_security_mutations.py: the current registry passes the drift check, and the check never starts a process (subprocess.Popen is patched to fail). The report carries identity but no score. It fails closed on edited, renamed, duplicated, CRLF and removed target source, and on unreconciled registry entries (version, duplicate or invalid test names, missing anchor, stale exception). It rejects a registry outside the repository and refuses --calibrate alongside.
    • test_security_mutations_workflows.py: the step exists in supply-chain, which has no if, needs or continue-on-error and stays in promote-image.needs. The step follows tool setup and the PyYAML install. The only security_mutations commands anywhere in ci.yml are the contract test and --validate-registry, so the campaign never runs on PRs. The scheduled controller command never uses --validate-registry.
  • Docs: docs/testing/security-mutations.md gains a "Pull-request registry check" section. It covers the PR-time reconciliation rule, how the function digest is computed (with a snippet that reproduces the registered hash), the CRLF caveat, and what the check cannot see: renamed tests, survivors, unviable mutants and timeouts still need the compiled binary and the scheduled campaign. security-coverage.md is updated so it no longer says no PR requirement changed.

Verification

The gateway crate cannot build locally (Node pin), and this Windows checkout uses CRLF, which changes function hashes. So I exported LF trees with git -c core.autocrlf=false archive and ran them with a local-only shim that stubs the Linux-only resource import:

  • --validate-registry passes on main's tree and on this branch: 4 targets, 8 registered mutants, registry_valid.
  • On main's tree, replaying the issue scenario (editing path_prefix_matches) fails with function_hash_drift, and renaming it fails with function_scope_drift. Restoring it passes again. The doc snippet reproduces the registered digest.
  • Both test files were run with unittest. All new tests pass. The only failures are the same 10 Linux-only tests (rlimits, fifo, symlinks, SIGKILL, chmod, real rustc), and they fail identically on main's tree.
  • Weakened ci.yml variants all fail the workflow contracts: step removed, continue-on-error, conditional step, and the campaign added to PR CI.
  • test_build_tools.py, build_tools.py check, test_security_corpus_workflows.py and test_image_publication.py pass.

The registry in security-mutations.json pins each target function's
source marker and SHA-256, its mutation anchors and its selected tests.
Until now it was only loaded by the nightly security-mutations workflow,
so a pull request that edited one of the registered functions passed PR
CI and broke the next scheduled campaign, after the change had merged.

Add a --validate-registry mode to scripts/security_mutations.py. It runs
the campaign's own manifest loader against the checked-out tree and
fails closed on the same drift categories (function scope and hash,
replacement anchors, test name rules, stale exceptions), but never
compiles, discovers tests or applies a mutant. Its report says
registry_valid with zero counts so it cannot pass for a mutation score,
and it cannot be combined with --calibrate. The manifest containment
check is shared with the campaign so both reject the same paths.

Run the check, and the workflow contract tests that assert it exists,
in the existing required supply-chain job. The required-check and
image-promotion lists stay unchanged, and the full campaign remains
scheduled and manual only. Document the pull-request reconciliation
rule and what the check cannot see: renamed tests and new survivors
still need the compiled binary and the scheduled campaign.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T01:28:27.530321Z ffc885b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Antnation
Antnation merged commit 37146db into main Oct 15, 2026
31 of 35 checks passed
Antnation added a commit that referenced this pull request Oct 15, 2026
@Antnation
Antnation deleted the ci/436-mutation-registry-drift branch October 15, 2026 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant