Skip to content

feat(scripts): measure what a scenario suite can actually detect - #34

Merged
MohammadThabetHassan merged 1 commit into
mainfrom
feat/policy-mutation-analysis
Sep 4, 2026
Merged

MohammadThabetHassan merged 1 commit into
mainfrom
feat/policy-mutation-analysis

Conversation

@MohammadThabetHassan

Copy link
Copy Markdown
Owner

The instrument behind the coverage-matrix finding, made reproducible.

What it does

A policy over a finite label domain induces a finite partition of its subject space — the twelve trust x action cells. A mutant is fully described by the decision it returns for each cell, so two mutants that agree on every cell are equivalent by construction. Equivalence is decided, not estimated, which is the usual obstacle to mutation-testing access-control policies.

Measured on the shipped policy

38 single-edit mutants, 16 provably equivalent (42.1%), 22 observable.

Suite Cases Cells Killed Score Missing decision
default-scenarios.json 5 5/12 14/22 63.6% none
adversarial-scenarios.json 25 3/12 8/22 36.4% allow
coverage-matrix-scenarios.json 12 12/12 22/22 100.0% none

The five-case suite catches nearly twice what the twenty-five-case suite does. Case count is not coverage, and the last column says why.

The sharpest instance: changing default_decision from deny to allow — a policy that permits everything not explicitly denied — passes the adversarial suite 25/25, and is caught by the five-case suite. That is the dangerous direction, and the OWASP/ATLAS-mapped suite is blind to it.

One test I had to weaken

I first asserted that every adversarial-suite survivor loosens the permitted set. It failed on widen_trust[TW-002+trusted], which moves a cell from deny to require_approval without changing what is allowed.

It now asserts what actually holds: a survivor differs from the reference only on cells the suite never witnesses. The reference satisfies every expectation, so a change to a witnessed cell would break one — detection power is bounded by cell coverage, not case count. That is the provable claim, and it is stronger than the one I started with.

ruff, mypy, reality_check and the full suite pass; coverage 95.09%.

A suite is normally judged by how many cases it holds, which says nothing about
what it would notice if the policy changed. This measures that directly.

A policy over a finite label domain induces a finite partition of its subject
space, here the twelve trust x action cells. A mutant is fully described by the
decision it returns for each cell, so two mutants agreeing on every cell are
equivalent by construction. Equivalence is therefore decided rather than
estimated, which is the usual obstacle to mutation testing access-control
policies.

Measured on the shipped policy: 38 single-edit mutants, 16 provably equivalent
(42.1%), 22 observable.

  suite                         cases  cells  killed   score  missing
  default-scenarios.json            5   5/12   14/22   63.6%  none
  adversarial-scenarios.json       25   3/12    8/22   36.4%  allow
  coverage-matrix-scenarios.json   12  12/12   22/22  100.0%  none

The five-case suite catches nearly twice what the twenty-five-case suite does.
Case count is not coverage, and the missing-decision column says why: the
adversarial suite expects no allow, so it cannot distinguish a policy that
permits a cell from one that denies it. Concretely, changing default_decision
from deny to allow -- a policy permitting everything not explicitly denied --
passes it 25/25 and is caught by the five-case suite.

One test began as a stronger claim than the data supports. Asserting that every
adversarial-suite survivor loosens the permitted set failed on
widen_trust[TW-002+trusted], which moves a cell from deny to require_approval
without touching what is allowed. It now asserts the property that actually
holds: a survivor differs from the reference only on cells the suite never
witnesses, since the reference satisfies every expectation and a change to a
witnessed cell would break one. Detection power is bounded by cell coverage.

Co-authored-by: Ahmed Alameri <AhmedSamiAlameri@users.noreply.github.com>
Co-authored-by: Fahad Sadek <fahadALjazzeri@users.noreply.github.com>
Co-authored-by: Omar Alraas <omaralraas@users.noreply.github.com>
Co-authored-by: Abdulrahman Rezki <104144822+AbdulrahmanRezki@users.noreply.github.com>
@omaralraas
omaralraas enabled auto-merge (squash) September 4, 2026 18:02
@omaralraas
omaralraas disabled auto-merge September 4, 2026 18:02
@omaralraas
omaralraas requested review from omaralraas and removed request for omaralraas September 4, 2026 18:03
@MohammadThabetHassan
MohammadThabetHassan merged commit fcd32d0 into main Sep 4, 2026
14 checks passed
@MohammadThabetHassan
MohammadThabetHassan deleted the feat/policy-mutation-analysis branch September 4, 2026 18:41
MohammadThabetHassan added a commit that referenced this pull request Sep 4, 2026
main merged #34, which landed scripts/policy_mutation.py from this line of work
under a different commit, so git saw an add/add conflict against the copy here.
The two versions are identical apart from the fragment guard added on this branch;
main carries no line this branch lacks. Resolved to the branch copy, keeping the
guard, and the full gate passes on the merged tree.
MohammadThabetHassan added a commit that referenced this pull request Oct 4, 2026
…velopment faults

scripts/rego_real_faults_study.py runs the protocol fixed in docs/REAL_FAULTS_PROTOCOL_REGO.md:
for each behaviour fix in the Gatekeeper and Config Validator histories, the module at the fix's
parent (the policy in use) and at the fix, each with its own commit's libraries; both suites'
inputs and settings; the exact study's witness space from the two versions together, its
completeness check, and OPA's decisions; the policy's own quotient as the Rego payoff study
builds it; and each strategy's exposure probability in closed form, the settings' misses
multiplied. Membership is the paper's procedure, run on the corpus as it stood at each commit.

Config Validator passes the asset at input.asset and the Constraint at input.constraint, and
decides with deny; the analysis reads a renamed copy of the syntax tree, while OPA decides on
the module as written. Only the fixtures a test reads are loaded: one malformed fixture
elsewhere in a 2019 tree stopped the whole suite, which is how the Config Validator
development fault first came back with no inputs.

The two development faults the protocol names (Gatekeeper #2, Config Validator #34) are the
only ones run so far.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants