Skip to content

Report unavailable and truncated policy evidence instead of zeros - #554

Merged
Antnation merged 1 commit into
mainfrom
feat/243-policy-evidence-honesty
Oct 15, 2026
Merged

Antnation merged 1 commit into
mainfrom
feat/243-policy-evidence-honesty

Conversation

@Antnation

Copy link
Copy Markdown
Contributor

Pull Request

What changed and why

Issue #243 slice 2 leftover: evidence honesty. When no audit query store is configured, the hit-count and shadow-review endpoints answered 200 with zeros and empty summaries. That looks the same as "this rule never matched" or "this shadow rule would deny nothing", so an operator could delete or promote a rule based on evidence that was never read. Cluster mode was always in this state, because both aggregates still read only the SQLite query surface.

Gateway

  • GET /v1{ADMIN_PREFIX}/policy/rules/hits and GET /v1{ADMIN_PREFIX}/policy/rules/shadow-review now answer 503 with {"error": "...", "reason": "audit_evidence_unavailable"} when query_store is None. This uses the existing {error, reason} admin error shape, through a new policy_evidence_unavailable helper in admin_responses.rs. Auth and permission checks still run first: the shadow review still returns 403 without admin:audit:read, even when no store exists.
  • Rule preview's existing 503 (no audit store) now carries the same reason. The error text is unchanged.
  • When a store is present, real zeros are unchanged and are now pinned by their own tests.
  • Rule preview has a hard scan ceiling: MAX_RULE_PREVIEW_SCAN_ROWS = 100_000 in app_state.rs, next to MAX_RULE_PREVIEW_SAMPLE_LIMIT, matching the other audit scan caps. PolicyRulePreviewResponse gains scan_truncated and complete. Both are additive. Truncation is reported only when the scan actually reaches a row past the ceiling, so a window that exactly fills the ceiling is still complete.

Admin UI

  • lib/policy.ts: fetchPolicyRuleHits and fetchPolicyRuleShadowReview return PolicyEvidence<T> (available | unavailable). Only the gateway's explicit 503 + audit_evidence_unavailable maps to unavailable. Every other failure, including other 503s, still rejects as an error.
  • RuleTable: each row's evidence reads Evidence unavailable, Not reported (the rule is missing from the counts), No recorded hits (a counted zero; this replaces "never matched"), or N hits. When evidence is unavailable, a notice explains it and the policy still loads.
  • ShadowReviewView: when evidence is unavailable, enabled shadow rules are listed from the policy with Evidence unavailable instead of zero counts. Truncated counts are labelled (scan truncated), and the banner says counts are lower bounds. A complete zero still reads 0 would-deny events.
  • RuleEditor preview: a truncated result is described as "at least N" and gets a Scan truncated badge. This is the natural consumer of the new preview fields.
  • Every state is shown in text, not by colour alone.

Docs

  • docs/configuration.md (preview, hits and shadow-review contracts) and docs/deployment/postgres.md (cluster-mode behaviour) are updated, plus a CHANGELOG entry. docs/openapi has no spec for these endpoints, so no OpenAPI changes.

Not in this PR (by design): binding shadow promotion to a reviewed revision, window or digest; the centralized privacy projection; the admin:audit:read split for preview samples.

Compatibility note: clients that relied on 200 with zeros from hits or shadow review without an audit store now get 503. This is the intended fail-closed change. The admin UI in this PR handles it.

Issue

Part of #243

Checklist

  • Tests added or updated where applicable. Rust: the unavailable 503s for hits and shadow review, real zeros with an empty store, audit-read checked before availability, preview truncation and exact-ceiling completeness, and cluster-mode 503s. Vitest: policy.test.ts, RuleTable, ShadowReviewView, RuleEditor.
  • Docs updated where applicable
  • No secrets or credentials are included in the diff
  • CI is expected to pass, except the pre-existing advisory steps (audit, admin-ui and cloudflare npm audit, connections-acceptance) tracked by the dependency PR

With no audit query store, the rule hit-count and shadow-review endpoints
answered 200 with zero counts and empty summaries. That reads exactly like
"this rule never matched" or "this shadow rule would deny nothing", so an
operator could delete a rule or promote a shadow rule to enforced deny on
evidence that was never read. Cluster mode was always in this state: both
aggregates still read only the SQLite query surface.

Both endpoints now answer 503 with a stable "audit_evidence_unavailable"
reason after their permission checks, and rule preview's existing 503 for
the same condition carries the same reason. A store that is present still
returns real zeros, which tests now pin separately.

Rule preview had no hard bound on how many observations one request could
scan. It now stops at MAX_RULE_PREVIEW_SCAN_ROWS (100,000, matching the
other audit scan ceilings) and reports scan_truncated and complete in its
response. Truncation is declared only when a row past the ceiling is
actually reached, so a window that exactly fills the ceiling stays
complete. Both fields are additive.

The admin UI maps only the gateway's explicit reason to an unavailable
evidence state; every other failure still surfaces as an error. The
rulebase shows "Evidence unavailable", "Not reported" and "No recorded
hits" as distinct text, the shadow review queue lists enabled shadow
rules from the policy with unavailable evidence instead of zero counts
and labels truncated counts, and the rule preview presents a truncated
count as a lower bound.

Shadow promotion is intentionally not yet bound to a reviewed revision,
window or digest; that remains separate work on #243.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T01:34:42.766293Z 9463e62 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9463e62fda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +250 to +253
void disableRule(item.ruleId);
}}
onPromote={() => {
void promoteRule(summary.rule_id);
void promoteRule(item.ruleId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Disable promotion when shadow evidence is unavailable

When /policy/rules/shadow-review returns the new evidence-unavailable 503, loadReview now populates items from the policy and retains write capability, so this callback still lets an operator confirm shadow → deny despite no audit evidence having been read. Previously that request failure cleared the summaries and exposed no promotion control; in cluster mode this now affects every shadow review because the aggregate is always unavailable. Keep the safe Disable action available, but gate Promote while evidence.status === 'unavailable' so an audit-store error cannot become an enforced authorization change.

AGENTS.md reference: AGENTS.md:L53-L53

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant