Repository navigation
feat(scripts): measure what a scenario suite can actually detect - #34
Merged
Merged
Conversation
A suite is normally judged by how many cases it holds, which says nothing about what it would notice if the policy changed. This measures that directly. A policy over a finite label domain induces a finite partition of its subject space, here the twelve trust x action cells. A mutant is fully described by the decision it returns for each cell, so two mutants agreeing on every cell are equivalent by construction. Equivalence is therefore decided rather than estimated, which is the usual obstacle to mutation testing access-control policies. Measured on the shipped policy: 38 single-edit mutants, 16 provably equivalent (42.1%), 22 observable. suite cases cells killed score missing default-scenarios.json 5 5/12 14/22 63.6% none adversarial-scenarios.json 25 3/12 8/22 36.4% allow coverage-matrix-scenarios.json 12 12/12 22/22 100.0% none The five-case suite catches nearly twice what the twenty-five-case suite does. Case count is not coverage, and the missing-decision column says why: the adversarial suite expects no allow, so it cannot distinguish a policy that permits a cell from one that denies it. Concretely, changing default_decision from deny to allow -- a policy permitting everything not explicitly denied -- passes it 25/25 and is caught by the five-case suite. One test began as a stronger claim than the data supports. Asserting that every adversarial-suite survivor loosens the permitted set failed on widen_trust[TW-002+trusted], which moves a cell from deny to require_approval without touching what is allowed. It now asserts the property that actually holds: a survivor differs from the reference only on cells the suite never witnesses, since the reference satisfies every expectation and a change to a witnessed cell would break one. Detection power is bounded by cell coverage. Co-authored-by: Ahmed Alameri <AhmedSamiAlameri@users.noreply.github.com> Co-authored-by: Fahad Sadek <fahadALjazzeri@users.noreply.github.com> Co-authored-by: Omar Alraas <omaralraas@users.noreply.github.com> Co-authored-by: Abdulrahman Rezki <104144822+AbdulrahmanRezki@users.noreply.github.com>
omaralraas
enabled auto-merge (squash)
September 4, 2026 18:02
omaralraas
disabled auto-merge
September 4, 2026 18:02
omaralraas
requested review from
omaralraas
and removed request for
omaralraas
September 4, 2026 18:03
omaralraas
approved these changes
Sep 4, 2026
MohammadThabetHassan
added a commit
that referenced
this pull request
Sep 4, 2026
main merged #34, which landed scripts/policy_mutation.py from this line of work under a different commit, so git saw an add/add conflict against the copy here. The two versions are identical apart from the fragment guard added on this branch; main carries no line this branch lacks. Resolved to the branch copy, keeping the guard, and the full gate passes on the merged tree.
MohammadThabetHassan
added a commit
that referenced
this pull request
Oct 4, 2026
…velopment faults scripts/rego_real_faults_study.py runs the protocol fixed in docs/REAL_FAULTS_PROTOCOL_REGO.md: for each behaviour fix in the Gatekeeper and Config Validator histories, the module at the fix's parent (the policy in use) and at the fix, each with its own commit's libraries; both suites' inputs and settings; the exact study's witness space from the two versions together, its completeness check, and OPA's decisions; the policy's own quotient as the Rego payoff study builds it; and each strategy's exposure probability in closed form, the settings' misses multiplied. Membership is the paper's procedure, run on the corpus as it stood at each commit. Config Validator passes the asset at input.asset and the Constraint at input.constraint, and decides with deny; the analysis reads a renamed copy of the syntax tree, while OPA decides on the module as written. Only the fixtures a test reads are loaded: one malformed fixture elsewhere in a 2019 tree stopped the whole suite, which is how the Config Validator development fault first came back with no inputs. The two development faults the protocol names (Gatekeeper #2, Config Validator #34) are the only ones run so far.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The instrument behind the coverage-matrix finding, made reproducible.
What it does
A policy over a finite label domain induces a finite partition of its subject space — the twelve trust x action cells. A mutant is fully described by the decision it returns for each cell, so two mutants that agree on every cell are equivalent by construction. Equivalence is decided, not estimated, which is the usual obstacle to mutation-testing access-control policies.
Measured on the shipped policy
38 single-edit mutants, 16 provably equivalent (42.1%), 22 observable.
default-scenarios.jsonadversarial-scenarios.jsoncoverage-matrix-scenarios.jsonThe five-case suite catches nearly twice what the twenty-five-case suite does. Case count is not coverage, and the last column says why.
The sharpest instance: changing
default_decisionfromdenytoallow— a policy that permits everything not explicitly denied — passes the adversarial suite 25/25, and is caught by the five-case suite. That is the dangerous direction, and the OWASP/ATLAS-mapped suite is blind to it.One test I had to weaken
I first asserted that every adversarial-suite survivor loosens the permitted set. It failed on
widen_trust[TW-002+trusted], which moves a cell fromdenytorequire_approvalwithout changing what is allowed.It now asserts what actually holds: a survivor differs from the reference only on cells the suite never witnesses. The reference satisfies every expectation, so a change to a witnessed cell would break one — detection power is bounded by cell coverage, not case count. That is the provable claim, and it is stronger than the one I started with.
ruff, mypy, reality_check and the full suite pass; coverage 95.09%.