Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
efd635e
check_manuscript: read supplement.tex beside the manuscript, re-ancho…
MohammadThabetHassan Oct 3, 2026
da7c852
check_manuscript: accept a decomposition stated in both the paper and…
MohammadThabetHassan Oct 3, 2026
25f87a6
check_manuscript: check every copy of each data figure, and the graph…
MohammadThabetHassan Oct 3, 2026
23f67c0
research: pre-register the exact adequacy study of real Rego suites
MohammadThabetHassan Oct 4, 2026
7ee5619
research: pre-register the extension of the Rego exact study to polic…
MohammadThabetHassan Oct 4, 2026
eed29f3
fix(research): spawn the Rego study's workers instead of forking them
MohammadThabetHassan Oct 4, 2026
add0be8
research: the Rego suites' survivors decided exactly, and the stillbo…
MohammadThabetHassan Oct 4, 2026
1f0cf5f
check_manuscript: pin the exact Rego study as the paper now states it
MohammadThabetHassan Oct 4, 2026
b4ffef6
check_manuscript: pin the realistic-gap count as the threats section …
MohammadThabetHassan Oct 4, 2026
c15b16e
check_manuscript: pin the raw score over the mutants that compile; ac…
MohammadThabetHassan Oct 4, 2026
ab6ca02
research: pre-register the suite-strategy study on real Rego policy
MohammadThabetHassan Oct 4, 2026
6de4f9a
research: the suite-strategy study on real Rego policy, results
MohammadThabetHassan Oct 4, 2026
3ba2913
check_manuscript: pin the suite-strategy study on real Rego, and the …
MohammadThabetHassan Oct 4, 2026
1d15272
research: test the Rego suite-strategy study's closed forms and its q…
MohammadThabetHassan Oct 4, 2026
6e175c3
check_manuscript: the exact-adequacy protocol is named without an ord…
MohammadThabetHassan Oct 4, 2026
82b155d
check_manuscript: wrap the renamed protocol pin to the line limit
MohammadThabetHassan Oct 4, 2026
efb440a
research: the exact study's secondary population, the policy schemas,…
MohammadThabetHassan Oct 4, 2026
98c4941
check_manuscript: pin the policy schemas as the paper reports them
MohammadThabetHassan Oct 4, 2026
42350d5
check_manuscript: pin the claims as a critical review left them, and …
MohammadThabetHassan Oct 4, 2026
0e7c7fb
research: pre-register a study of real faults in two Rego libraries' …
MohammadThabetHassan Oct 4, 2026
cf7301b
research: the real-faults study's instrument, developed on its two de…
MohammadThabetHassan Oct 4, 2026
60c287f
research: the real-faults study, results
MohammadThabetHassan Oct 4, 2026
eac9f26
reproduce: an image for the paper's measurements, its fast checks in …
MohammadThabetHassan Oct 4, 2026
97fb9f4
check_manuscript: pin the real-faults study wherever the paper states it
MohammadThabetHassan Oct 4, 2026
f267772
reproduce: the pre-registration check lives in scripts/, and the imag…
MohammadThabetHassan Oct 4, 2026
04a49b4
research: pre-register a SymCC check of the Cedar exact study's equiv…
MohammadThabetHassan Oct 4, 2026
b4cbfee
research: the SymCC check's instrument, developed on its development …
MohammadThabetHassan Oct 4, 2026
1119460
reproduce: a README for the image, the SymCC tools in it, and the gap…
MohammadThabetHassan Oct 4, 2026
05bbaa3
research: the SymCC check reads a datetime, and says when a counterex…
MohammadThabetHassan Oct 4, 2026
b318ea9
research: SymCC refutes none of the Cedar study's 173 equivalence ver…
MohammadThabetHassan Oct 4, 2026
fdde1fb
research: pre-register a comparison with Xu et al.'s XACML coverage c…
MohammadThabetHassan Oct 4, 2026
4bb020d
research: the XACML comparison's instrument, developed on a synthetic…
MohammadThabetHassan Oct 4, 2026
09f401f
research: the XACML comparison gives XPA's PTT mutants the Target the…
MohammadThabetHassan Oct 4, 2026
6c97bd8
reproduce: the image can re-run the comparison with Xu et al.'s criteria
MohammadThabetHassan Oct 4, 2026
8d629c6
research: Xu et al.'s criteria and the quotient, on Xu et al.'s own b…
MohammadThabetHassan Oct 4, 2026
925b7b1
check_manuscript: pin the comparison with Xu et al.'s criteria wherev…
MohammadThabetHassan Oct 4, 2026
af6d1be
check_manuscript: the Xu et al. sentence compares like with like
MohammadThabetHassan Oct 4, 2026
e9afeab
research: the exclusions crossed with every reason they carry
MohammadThabetHassan Oct 4, 2026
057d36b
research: the second review's questions, answered from artifacts alre…
MohammadThabetHassan Oct 4, 2026
05182b0
check_manuscript: pin the text the second review produced
MohammadThabetHassan Oct 4, 2026
8928e08
reproduce: no SIGPIPE race when printing tool versions
MohammadThabetHassan Oct 4, 2026
cba0152
check_manuscript: the proxy's median size is checked on all three pop…
MohammadThabetHassan Oct 4, 2026
267c9c1
docs: protocol for established criteria against the quotient at match…
MohammadThabetHassan Oct 4, 2026
14318d7
research: the instrument for established criteria against the quotient
MohammadThabetHassan Oct 4, 2026
20a1123
docs: protocol for certifying the inside verdicts call by call
MohammadThabetHassan Oct 4, 2026
e50eb04
research: the instrument that certifies inside verdicts call by call
MohammadThabetHassan Oct 4, 2026
0d3cbb6
research: established criteria against the quotient, results
MohammadThabetHassan Oct 4, 2026
bfb9901
guard: pin the established-criteria study wherever the paper states it
MohammadThabetHassan Oct 4, 2026
8fc7740
research: the census reads what the protocol says it reads
MohammadThabetHassan Oct 4, 2026
48423b2
research: inside verdicts certified call by call, results
MohammadThabetHassan Oct 4, 2026
257f179
guard: pin the census wherever the paper states it
MohammadThabetHassan Oct 4, 2026
167195c
guard: the census sentences say Cedar's 22 were certified by design
MohammadThabetHassan Oct 4, 2026
3793243
research: the Rego instruments run where there is no /dev/stdin
MohammadThabetHassan Oct 4, 2026
0b98050
docs: protocol for Rego equivalence under every Constraint the librar…
MohammadThabetHassan Oct 4, 2026
0659a66
research: the instrument for Rego equivalence under every shipped Con…
MohammadThabetHassan Oct 4, 2026
eb0f11e
research: the reproduction gate compares each setting's cells, whatev…
MohammadThabetHassan Oct 4, 2026
f5e677a
research: Rego equivalence under every Constraint the library ships, …
MohammadThabetHassan Oct 4, 2026
14edd62
guard: pin the shipped-Constraints study wherever the paper states it
MohammadThabetHassan Oct 4, 2026
06a5886
guard: the 26 are equivalents in modules that read parameters
MohammadThabetHassan Oct 4, 2026
5e66b1f
docs: protocol for the payoff over subsumption-minimal mutants
MohammadThabetHassan Oct 4, 2026
e226d87
research: the instrument for the payoff over subsumption-minimal mutants
MohammadThabetHassan Oct 4, 2026
7e4d828
research: decision coverage and the decision proxy, kept apart
MohammadThabetHassan Oct 4, 2026
aee8cc5
research: the minimal-mutant study's note says what the correction ch…
MohammadThabetHassan Oct 4, 2026
ac72ad5
research: the payoff over subsumption-minimal mutants, results
MohammadThabetHassan Oct 4, 2026
9305894
guard: pin the minimal-mutant study, and scope the XACML criteria rows
MohammadThabetHassan Oct 4, 2026
51ab273
reproduce: the minimal-mutant study in the artifact map and the verif…
MohammadThabetHassan Oct 4, 2026
d9d2394
docs: tighten the README, the site's front pages and the 0.3.1 changelog
MohammadThabetHassan Oct 4, 2026
c51346f
docs: protocol for the exact adequacy of the Kyverno library's suites
MohammadThabetHassan Oct 4, 2026
b7616ff
research: the instrument for the exact adequacy of the Kyverno librar…
MohammadThabetHassan Oct 4, 2026
d95e08b
research: the exact adequacy of the Kyverno library's suites, results
MohammadThabetHassan Oct 4, 2026
3a66494
guard: pin the Kyverno exact study; reproduction map and verify tests
MohammadThabetHassan Oct 4, 2026
34b1bb3
guard: follow the manuscript's copy-edit
MohammadThabetHassan Oct 5, 2026
c0eb988
guard: let the abstract's strategy figures open a sentence
MohammadThabetHassan Oct 5, 2026
a5beb63
guard: accept "column" in the taxonomy-shift pin
MohammadThabetHassan Oct 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/reproduce.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Reproduction image

# Builds the image the paper's measurements can be reproduced in (reproduce/Dockerfile) and runs
# its fast checks: the tool versions the artifacts record, the pre-registration check, the
# study scripts' tests and the solver check of the witness construction. The hours-long studies
# and the corpus re-measurement are run by hand; reproduce/README.md gives their commands.

on:
pull_request:
paths:
- "reproduce/**"
- "scripts/**"
- "docs/*.json"
- "docs/*PROTOCOL*.md"
- "pyproject.toml"
- ".github/workflows/reproduce.yml"
workflow_dispatch:

permissions:
contents: read

jobs:
image:
name: Build and verify the reproduction image
runs-on: ubuntu-latest
steps:
- name: Check out source
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Build the image
run: docker build -f reproduce/Dockerfile -t trustweave-reproduce .

- name: Run the fast checks inside it
run: docker run --rm trustweave-reproduce verify
31 changes: 31 additions & 0 deletions .zenodo.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"title": "TrustWeave: instruments, artifacts and reproduction image for exact mutation adequacy of access-control policies",
"upload_type": "software",
"description": "The measurement instruments, the JSON artifacts every reported figure is derived from, the pre-registration protocols and the reproduction image (reproduce/Dockerfile) behind TrustWeave's study of exact mutation adequacy for access-control policies. reproduce/README.md maps each measurement to the script and artifact that produced it.",
"creators": [
{"name": "Hassan, Mohammad Thabet", "affiliation": "Canadian University Dubai"},
{"name": "Alameri, Ahmed Sami", "affiliation": "Canadian University Dubai"},
{"name": "Sadek, Fahad", "affiliation": "Canadian University Dubai"},
{"name": "Alraas, Omar", "affiliation": "Canadian University Dubai"},
{"name": "Rezki, Abdulrahman", "affiliation": "Canadian University Dubai"},
{"name": "AbuSerrieh, Lobna", "affiliation": "Canadian University Dubai"}
],
"license": "Apache-2.0",
"keywords": [
"mutation testing",
"equivalent mutants",
"access control",
"policy testing",
"test adequacy",
"Rego",
"Cedar",
"XACML"
],
"related_identifiers": [
{
"identifier": "https://github.com/MohammadThabetHassan/trustweave",
"relation": "isSupplementTo",
"scheme": "url"
}
]
}
42 changes: 29 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,25 @@ All notable changes to TrustWeave are documented in this file. The project follo

## [0.3.1] - Unreleased release candidate

### Highlights

- `trustweave discover` reads local Python source, lists the tools an agent can reach, proposes
an action class for each with its evidence, and reports what the manifest leaves undeclared
(`trustweave.dev/code-discovery/v1alpha1`, ten `TW-CODE-*` rules).
- Policy review finds rules shadowed by a combination of earlier rules, reports declined cover
searches and rules switched off by undeclared controls, and is published as `v1alpha2`; risk
identities move to `trustweave/fingerprint/v4`.
- Thirty-one defects confirmed by a full audit of the branch are fixed, each with a regression
case, among them Markdown injection in reviewer-facing reports, false negatives in chain
analysis, and a failing test-results file edited to read "passed" that still attested and
verified.
- Research instruments for the exact mutation-adequacy study: protocols fixed before each run,
the measurement scripts and artifacts, engine oracles, Lean proofs of the core results and a
reproduction guide, `docs/REPRODUCING_THE_STUDY.md`.
- The README and documentation are reorganized around a verified quickstart.

The entries below give the detail.

### Added

- `scripts/exact_evaluation_study.py` runs two studies that use exact equivalence as ground
Expand Down Expand Up @@ -323,7 +342,7 @@ All notable changes to TrustWeave are documented in this file. The project follo
shell invocation reached that way as `read`; a wildcard import inside a function was
likewise dropped instead of forcing a refusal. Names now resolve through every
enclosing function, innermost first, and a node inside a nested function is judged in
that function's scope. A multi-agent audit of the branch caught both before it was
that function's scope. A pre-merge audit of the branch caught both before it was
pushed; both are regression cases now.
- A rule naming more than 10,000 combinations of its single-valued fields is not
enumerated for a collective cover, and the artifact used to say `reachable: true`
Expand All @@ -336,15 +355,12 @@ All notable changes to TrustWeave are documented in this file. The project follo
the source does not decide is refused. This is the one case of the review's broader
point, that unresolved behaviour is silently benign, that came with a reproducible
snippet; the general audit it asks for is not done here.
- A multi-agent audit of this branch (about 600 separately prompted reviewer and verifier agents
over the code, the research harness and both manuscripts, every error claim adversarially
verified before it was accepted) confirmed thirty-five errors. Thirty-one are fixed below,
each reproduced from the audit's own probe before it was touched and each kept as a
regression case; the four that are not (the new manuscript's missing revision package and
its section 8 prose, which have no source on this machine, and the Kyverno mutation rerun,
which needs a `kyverno` binary) are recorded in the audit report rather than papered over.
Almost all of them had one shape: a confident, clear answer where the evidence supported
none.
- A full audit of this branch, covering the code, the research harness and both manuscripts,
with every reported error reproduced before it was accepted, confirmed thirty-five errors.
Thirty-one are fixed below, each reproduced from the audit's own probe before it was touched
and each kept as a regression case. The other four concern manuscript material kept outside
the repository and the Kyverno mutation rerun, which is listed under Changed. Almost all of
them had one shape: a confident, clear answer where the evidence supported none.

**Discovery analyzer.**
- A callee the module cannot name is refused rather than read as benign. `read` at high
Expand Down Expand Up @@ -443,8 +459,8 @@ All notable changes to TrustWeave are documented in this file. The project follo
file, mirrored as `maxItems` in both copies of the manifest schema.
- The `REVIEW_REQUIRED` placeholder is refused wherever a manifest string field begins with
it, naming the field. The documentation said the discovery draft failed validation because
the parser rejected the placeholder; it failed for other reasons, and a lazy-reviewer loop
applying one minimal fix per error reached a passing scan with eight placeholders left.
the parser rejected the placeholder; it failed for other reasons, and a reviewer applying
one minimal fix per error could reach a passing scan with eight placeholders left.

**Review commands, evidence and I/O.**
- Markdown forgery in every reviewer-facing artifact. Declared text was interpolated into
Expand Down Expand Up @@ -727,7 +743,7 @@ All notable changes to TrustWeave are documented in this file. The project follo
selects a symbol while `eval` runs whatever it is handed: arbitrary code execution is
what `sensitive` means here, and it is knowable without reading the code. A constant
expression is left alone and `getattr` keeps its refusal.
- Benchmark accuracy rises from 0.806 to 0.987 over the session on a benchmark that grew
- Benchmark accuracy rises from 0.806 to 0.987 over this release on a benchmark that grew
from 72 cases to 75, the answer rate on decidable cases from 0.860 to 1.000, and
precision when answering from 0.918 to 0.9833. Accuracy on labels the two annotators
agree about is 1.000; the single remaining failure is the one case where they disagree.
Expand Down
Loading
Loading