Skip to content

Commit 120dc3a

Browse files
committed
test(protect-mcp): add test/ fixtures and round-trip verification
Follow-up to wshobson#484 closing the test-plan commitment. Adds a plugins/protect-mcp/test/ directory with: - Six deterministic fixtures covering PreToolUse (allow + deny paths on Read / Bash safe / Bash destructive / Write) and PostToolUse (receipt signing input) - A Cedar test policy exercising both permit and forbid semantics - An expected receipt-schema.json (JSON Schema draft-07) pinned to draft-farley-acta-signed-receipts required fields - run-tests.sh: full round-trip, requires node >= 18 and python3. Eight tests covering evaluate (permit/forbid exit codes), sign (receipt file produced), schema conformance, verify (valid + tamper detection). - verify-fixtures.sh: static fixture validation, python3 only, safe to run in sandboxed CI without network access. - README.md explaining the layout, how to run, and the exit-code convention (including 77 = autotools "skip" for missing tools). The critical regression guard is test 8: flipping the `decision` field in a signed receipt MUST invalidate the Ed25519 signature, so `@veritasacta/verify` MUST exit 1. This locks in the tamper-detection property that the plugin claims. No changes to the plugin itself. No new runtime dependencies. No changes to marketplace.json or hooks.json.
1 parent 87b81e9 commit 120dc3a

10 files changed

Lines changed: 448 additions & 0 deletions

plugins/protect-mcp/test/README.md

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# protect-mcp test fixtures
2+
3+
Round-trip tests for the `protect-mcp` plugin's `PreToolUse` and `PostToolUse`
4+
hooks. Exercises the full evaluate → sign → verify loop against deterministic
5+
fixtures, including the tamper-detection path.
6+
7+
## Layout
8+
9+
```
10+
test/
11+
├── fixtures/
12+
│ ├── test-policy.cedar # Cedar policy used by all tests
13+
│ ├── pretool-allow-read.json # Read should be permitted
14+
│ ├── pretool-allow-bash-safe.json # Bash "git status" should be permitted
15+
│ ├── pretool-deny-bash-destructive.json # Bash "rm -rf /" should be denied
16+
│ ├── pretool-deny-write.json # Write should be denied
17+
│ └── posttool-signing-input.json # Input for receipt signing
18+
├── expected/
19+
│ └── receipt-schema.json # Expected receipt shape (JSON Schema)
20+
├── run-tests.sh # Full round-trip (requires node / npx)
21+
└── verify-fixtures.sh # Static validation (python3 only)
22+
```
23+
24+
## Running
25+
26+
### Full round-trip (local development)
27+
28+
```bash
29+
./run-tests.sh
30+
```
31+
32+
Requires `node` (>= 18), `npx`, and `python3`. Fetches `protect-mcp` and
33+
`@veritasacta/verify` from npm on first run. Runs eight tests:
34+
35+
| # | Scenario | Expected exit |
36+
|---|----------|----------------|
37+
| 1 | `PreToolUse` on `Read` | 0 (permit) |
38+
| 2 | `PreToolUse` on `Bash git status` | 0 (permit) |
39+
| 3 | `PreToolUse` on `Bash rm -rf /` | 2 (forbid) |
40+
| 4 | `PreToolUse` on `Write` | 2 (forbid) |
41+
| 5 | `PostToolUse` signing produces a receipt file | 0 (success) |
42+
| 6 | Produced receipt conforms to the schema | 0 (valid) |
43+
| 7 | `@veritasacta/verify` accepts the receipt | 0 (valid) |
44+
| 8 | Tampered receipt is rejected | 1 (tampered)|
45+
46+
Test 8 is the critical regression guard: flipping the `decision` field in a
47+
signed receipt must invalidate the Ed25519 signature, so `@veritasacta/verify`
48+
must exit 1 rather than 0.
49+
50+
### Static validation (CI-safe)
51+
52+
```bash
53+
./verify-fixtures.sh
54+
```
55+
56+
Only requires `python3`. Validates that every fixture is well-formed JSON and
57+
has the expected structure. No network calls, no npm fetches. Safe to run in
58+
sandboxed or offline CI.
59+
60+
## What the tests prove
61+
62+
- **Policy evaluation:** Cedar `permit` and `forbid` rules produce the
63+
expected exit codes (0 / 2).
64+
- **Receipt schema:** signed receipts include every required field from
65+
[`draft-farley-acta-signed-receipts`](https://datatracker.ietf.org/doc/draft-farley-acta-signed-receipts/).
66+
- **Signature integrity:** `@veritasacta/verify` validates authentic
67+
receipts and rejects tampered ones, with the documented exit codes.
68+
- **End-to-end integration:** the plugin's two hooks compose into a
69+
working allow/deny + sign + verify pipeline.
70+
71+
## Extending
72+
73+
To add a new test case:
74+
75+
1. Drop a `pretool-*.json` or `posttool-*.json` fixture into `fixtures/`
76+
2. Add a matching rule to `fixtures/test-policy.cedar` if the test needs one
77+
3. Add an assertion block to `run-tests.sh` mirroring the existing ones
78+
79+
Follow the naming convention `pretool-<allow|deny>-<scenario>.json` so the
80+
intent is obvious from `ls fixtures/`.
81+
82+
## Exit codes
83+
84+
| Script | Exit | Meaning |
85+
|--------------------|------|---------|
86+
| `run-tests.sh` | 0 | All tests passed |
87+
| `run-tests.sh` | 1 | One or more tests failed |
88+
| `run-tests.sh` | 77 | Required tool missing (skipped in CI) |
89+
| `verify-fixtures.sh` | 0 | All fixtures valid |
90+
| `verify-fixtures.sh` | 1 | Fixture malformed |
91+
| `verify-fixtures.sh` | 77 | `python3` missing (skipped) |
92+
93+
77 is the autotools convention for "skip this test" and is interpreted as a
94+
skip by most CI frameworks.
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"$id": "https://veritasacta.com/schemas/receipt-v1.json",
4+
"title": "Veritas Acta Decision Receipt v1",
5+
"description": "Expected shape of a protect-mcp-produced receipt. Mirrors draft-farley-acta-signed-receipts.",
6+
"type": "object",
7+
"required": [
8+
"receipt_id",
9+
"receipt_version",
10+
"issuer_id",
11+
"event_time",
12+
"tool_name",
13+
"decision",
14+
"public_key",
15+
"signature"
16+
],
17+
"properties": {
18+
"receipt_id": { "type": "string", "pattern": "^rec_" },
19+
"receipt_version": { "type": "string", "const": "1.0" },
20+
"issuer_id": { "type": "string" },
21+
"event_time": { "type": "string", "format": "date-time" },
22+
"tool_name": { "type": "string" },
23+
"input_hash": { "type": "string", "pattern": "^sha256:" },
24+
"decision": { "type": "string", "enum": ["allow", "deny"] },
25+
"policy_id": { "type": "string" },
26+
"policy_digest": { "type": "string", "pattern": "^sha256:" },
27+
"parent_receipt_id": { "type": ["string", "null"] },
28+
"public_key": { "type": "string", "minLength": 32 },
29+
"signature": { "type": "string", "minLength": 32 }
30+
},
31+
"additionalProperties": true
32+
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"tool_name": "Read",
3+
"tool_input": {
4+
"file_path": "./README.md"
5+
},
6+
"tool_output": {
7+
"content": "# Example project\n\nTest fixture output."
8+
},
9+
"session_id": "test-session-sign",
10+
"decision": "allow",
11+
"policy_id": "protect-mcp-test-policy"
12+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"tool_name": "Bash",
3+
"tool_input": {
4+
"command": "git status"
5+
},
6+
"session_id": "test-session-allow-bash",
7+
"context": {
8+
"command_pattern": "git"
9+
}
10+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"tool_name": "Read",
3+
"tool_input": {
4+
"file_path": "./README.md"
5+
},
6+
"session_id": "test-session-allow-read",
7+
"context": {
8+
"path_starts_with": "./"
9+
}
10+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"tool_name": "Bash",
3+
"tool_input": {
4+
"command": "rm -rf /"
5+
},
6+
"session_id": "test-session-deny-destructive",
7+
"context": {
8+
"command_pattern": "rm -rf"
9+
}
10+
}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"tool_name": "Write",
3+
"tool_input": {
4+
"file_path": "./secrets.env",
5+
"content": "dummy"
6+
},
7+
"session_id": "test-session-deny-write",
8+
"context": {
9+
"path_starts_with": "./"
10+
}
11+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// Test Cedar policy for protect-mcp hook round-trip tests.
2+
// Not a production example. See ../../agents/policy-enforcer.md for real-world policies.
3+
4+
// Allow all read-oriented tools.
5+
permit (
6+
principal,
7+
action in [Action::"Read", Action::"Glob", Action::"Grep", Action::"WebSearch"],
8+
resource
9+
);
10+
11+
// Allow safe Bash commands only.
12+
permit (
13+
principal,
14+
action == Action::"Bash",
15+
resource
16+
) when {
17+
context.command_pattern in ["git", "npm", "ls", "cat", "echo", "pwd", "test", "node"]
18+
};
19+
20+
// Explicit deny on destructive commands, even if permit would match elsewhere.
21+
// Cedar deny is authoritative.
22+
forbid (
23+
principal,
24+
action == Action::"Bash",
25+
resource
26+
) when {
27+
context.command_pattern in ["rm -rf", "dd", "mkfs", "shred", ":(){ :|:& };:"]
28+
};
29+
30+
// Writes are denied unscoped. A real policy would permit writes when
31+
// context.path_starts_with is safe. The tests exercise the unscoped
32+
// form so we can verify deny is enforced.
33+
forbid (
34+
principal,
35+
action in [Action::"Write", Action::"Edit"],
36+
resource
37+
);

0 commit comments

Comments
 (0)