Skip to content

redaction, pkg/scan: when coverage is cut short, redaction paths return partial output without failing #770

Closed
#832
#813
Closed
redaction, pkg/scan: when coverage is cut short, redaction paths return partial output without failing#770
#832
#813
Labels
P0Audit tier P0: fail-open or cleartext leak; fix firstbugSomething isn't working
Milestone
v2.6.0

Description

@sectoramen
sectoramen
opened on Oct 14, 2026

Summary

When coverage is cut short (caller ctx cancelled or expired, a validator size cap, a fired --validator-budget, an unparseable file), pkg/redact.Engine.Redact and pkg/scan.RedactFile return unredacted text with err == nil, and --stdin --enable-redaction exits 0 even under --fail-on-incomplete. Measured: a pre-cancelled ctx returns Redacted identical to the input (2,000/2,000 cards and SSNs in clear); a 5.6 MiB log under context.Background() keeps 5,518/5,518 ARN lines in clear; an unparseable .pdf is copied through verbatim. Only --file discloses (NOT FULLY EXAMINED, exit 3 under the flag). A fired time budget also discards every finding the validator made (SSN=1s: 0 of 20,000; a one-line A/B keeps 7,424–13,824) and runs every check on the file two extra times (CPU 5.38–6.12 s vs 3.23–3.40 s). pkg/scan.ScanText/ScanFile ignore their ctx entirely.

Reproduction

Measured on main @ 3df48ac (v2.5.3); line references are at that commit. main has since gained dependency patch bumps and #751, #752 and #759 (changelog CI, perfguard, build tooling), none of which changes detection or redaction.

printf 'Ticket 4471: caller verified SSN %s, reply to j7.example@example.org\n' "$(printf '5%02d-%02d-%04d' 17 17 1007)" > one.txt
ferret-scan --stdin --enable-redaction --checks SSN,EMAIL --validator-budget all=1ns --fail-on-incomplete < one.txt; echo "exit=$?"
ferret-scan --file one.txt --enable-redaction --checks SSN,EMAIL --validator-budget all=1ns --fail-on-incomplete; echo "exit=$?"

--stdin: the line unchanged, a stderr warning, exit 0 (3/3 runs). --file: NOT FULLY EXAMINED … coverage cut short, exit 3.

Inputs and method

All values are synthetic:

  • Cards: the body 453201 + %09d of 100000000+(i*7919)%899999999, closed with a computed Luhn digit, written as four space-separated groups of 4.
  • SSN-shaped values: 5%02d-%02d-%04d of (10+i%80, 10+i%89, 1000+i%8999).
  • E-mail: addresses at example.org.
  • ARN: arn:aws:s3:::synthetic-bucket-q7x2/logs.

Files:

  • records.txt: 20,000 lines (2,177,780 B) of the form Record %d: customer card %s, SSN %s, contact j%d.example@example.org on file.
  • infra.log: 5,734,445 B. One e-mail line, then the ARN on every 21st line and padding otherwise (5,518 ARN lines).
  • broken.pdf: a %PDF-1.7 header and one SSN-shaped value, with no PDF structure.

CPU is user+sys (getrusage, /usr/bin/time -p), from interleaved samples at load averages 6–66. Go harnesses use replace github.com/awslabs/ferret-scan/v2 => <checkout>.

Matrix A: surfaces on cut-short inputs

CLI runs use --enable-redaction --format json --output findings.json. "Copy" means the file in --redaction-output-dir. Exit codes are shown as without / with --fail-on-incomplete.

input / trigger route exit redacted output disclosure
one.txt, no budget --file / --stdin 0 copy / stdout, both values masked —
clean file --file 0 no copy —
clean file --stdin 0 stdout = input —
one.txt, all=1ns --file 0 / 3 no copy (0 matches) NOT FULLY EXAMINED, stats.files_not_examined: 1
one.txt, all=1ns --stdin 0 / 0 stdout = input stderr WARNING only; findings.json = {"results": []}
one.txt, all=1ns, --pre-commit-mode --file 0 / 3 no copy one-line notice (124 B, worded contents unreadable); findings.json 193–194 B with stats
one.txt, all=1ns, --pre-commit-mode --stdin 0 / 0 stdout = input 0 B stderr; findings.json 0 B
broken.pdf, --checks SSN --file 0 / 3 no copy NOT FULLY EXAMINED, cannot parse
infra.log, CLOUD_RESOURCES,EMAIL --file 0 / 3 copy: e-mail masked, 5,518/5,518 ARN lines in clear NOT FULLY EXAMINED
infra.log, CLOUD_RESOURCES,EMAIL --stdin 0 / 0 stdout byte-identical to the copy stderr WARNING only; no stats
records.txt, --checks SSN, SSN=300ms --stdin 0 / 0 stdout = input, 20,000/20,000 SSNs stderr WARNING only

Detect-only --stdin already exits 3 under the flag, but in pre-commit mode it prints 0 B of disclosure too.

library call (Background() unless noted) result
RedactFile(broken.pdf, SSN) err=nil, count=0, output = input. ScanFile reports Incomplete with cause cannot parse.
RedactFile(infra.log, CLOUD_RESOURCES,EMAIL) err=nil, count=1, 5,518/5,518 ARN lines in clear
RedactFile(infra.log, CLOUD_RESOURCES) err=nil, count=0, clean-file passthrough
Engine.Redact(infra.log, CLOUD_RESOURCES) err=nil, 0 findings, Redacted == input
Engine.Redact, 2,000 records Background(): 6,000 findings (1.12–1.17 s CPU). Pre-cancelled, or a 1, 100 or 200 ms deadline: err=nil, 0 findings, FindingsByType=map[], Redacted == input. A 2 s deadline (does not fire): 6,000.
ScanText(infra.log, CLOUD_RESOURCES) err=nil, Incomplete=true, len(NotExamined)=0, reason validator declined oversize content: validator declined oversize content: cloud-resource scan skipped, …
ScanText/ScanFile, 10,000 records, pre-cancelled or a 50 ms deadline 30,000/30,000 findings, Incomplete=false, 1.64–2.07 s CPU. The ctx is ignored.

examples/lambda-redact/handler.go enables every check (:78), passes the invocation ctx (:176), and returns res.Redacted whenever err == nil (:224-234). So a firing deadline, or a request over 5 MiB, returns the original text.

Matrix C: fired budgets

Run on records.txt with --format json --confidence all --limit 0, interleaved:

--checks budget findings files_not_examined CPU
SSN none 20,000 absent 2.46–3.30 s
SSN SSN=300ms / 1s / 1.5s 0 1 1.39–1.79 / 3.58–4.32 / 5.06–6.00 s
SSN SSN=3s / 20s (does not fire) 20,000 absent 2.47–3.14 s
SSN,EMAIL none 20,000 SSN + 20,000 e-mail absent 3.23–3.40 s
SSN,EMAIL SSN=300ms / 1s e-mail only 1 3.12–3.20 / 5.38–6.12 s

With SSN=300ms and --checks SSN,EMAIL, --debug logs 3 process_content passes, 2 s and then 3 s apart, each ending in context deadline exceeded. It also logs 3 email_validator validate_content records, against 1 without a budget.

Binary A/B. The only change is at dual_path_bridge.go:962: errors.Is(r.err, execguard.ErrMatchBudgetExceeded) becomes execguard.IsCoverageCutShort(r.err) (plus dropping the then-unused errors import). Results:

budget HEAD keeps widened keeps
none 20,000 20,000 (identical one by one)
SSN=300ms 0 2,304–4,096
SSN=1s 0 7,424–13,824

The widened gate is necessary but not sufficient:

  • Lower scores. Each kept partial matches a complete-run finding on (line, text, type) (13,824/13,824 in one sample), but each 75 scores 55 and each 60 scores 40, so MEDIUM drops to LOW (12,870 and 949 of 13,824; the 5 at 100 are unchanged).
  • Missing metadata. Partials lack the bridge metadata: context_*, confidence_adjustment, original_confidence, semantic_context and validation_path.
  • Caller-ctx path still loses everything. Engine.Redact on 4,000 lines returns 0 findings at 100–300 ms deadlines (18/18 samples).
  • Cancellation leaves work running. A 50 ms deadline on 20,000 lines returns in 50–52 ms, but the abandoned validators then burn 138–204 ms of CPU over the next 4 s.

Root cause

A cut-short scan is treated as success or as "found nothing".

  • A (sinks): pkg/redact/engine.go:269-272 drops runErr; internal/core/redact.go:131 checks only UnredactedFiles, then passes through (:140-143); cmd/stdin.go:348 returns before the --fail-on-incomplete resolution (:408-418) into return 0 (:556-559); :250 drops the pre-commit warning.
  • B (ctx): pkg/scan/scan.go:286, :322 take _ context.Context; internal/core/scanner.go:415 and internal/parallel/worker_pool.go:168 root on context.Background().
  • C (partials): internal/validators/dual_path_bridge.go:961-963 keeps partials only on ErrMatchBudgetExceeded, a hand copy of execguard.IsCoverageCutShort (execguard.go:87-105); worker_pool.go:353 retries it: internal/resilience/errors.go:155, :183-198 class a deadline retryable.
Detail
  • No field to carry the loss. redact.Result and AuditRecord (types.go:221, :187) have no incomplete field, and neither does RedactFileResult (redactfile.go:41-48).
  • Comments that promise the opposite.
    • redactfile.go:54: "safe to share".
    • engine.go:293-295: "all-or-nothing".
    • engine.go:262-264: "pass a derived context".
    • validator_runner.go:60-63: callers "log the error and continue". That is safe for detection and unsafe for redaction.
  • stdin.
    • cmd/stdin.go:245-249 calls the exit code "advisory warning only".
    • The residual floor at :478-491 checks only values that were reported.
    • :369 and :585 build FormatterOptions without Stats, NotExamined or OutputToFile. File mode sets the last of these at cmd/main.go:2006.
  • No library time bound.
    • ScanConfig (scanner.go:30) has no ValidatorBudgets, and ScanText leaves ContentScanConfig.ValidatorBudgets (scanner.go:354) unset.
    • RedactFile takes no ctx.
    • core.ScanContent never fills NotExamined. Its branches (scanner.go:431-447) enumerate the family by hand, omit ErrValidatorPanicked, and prepend the reason twice.
    • internal/web/server.go:899 passes no ctx, and r.Context() appears nowhere.
    • Preprocessor.Process(filePath string) (preprocessor.go:256) takes no ctx, so extraction cannot be interrupted.
  • Partials.

Width of the class

Redaction sinks: 4 (3 fail open). pkg/ exports taking a ctx: 3, all of them (2 discard it, 1 fails open). Hand copies of the cut-short family outside execguard (non-test): 2. Non-blocking drains: 2. Retry wrappers: 1. Validators returning partials with ctx.Err(): 18. Tests feeding a sink a cut-short scan or dead ctx: 0.

Resolution

The cut-short contract (owned here; --file does this today, so both CLI paths share one rule):

  • (a) Output never exceeds --file's (a copy only when a value was masked). --stdin keeps emitting its stream, as for clean input; RedactFile stops passing a not-examined file through as clean.
  • (b) Disclosure, always: stderr (incl. the pre-commit one-liner), stats + not-examined entry, library Incomplete/NotExamined.
  • (c) Signal: CLI exit 3 under --fail-on-incomplete, else 0. Library sinks always return an error wrapping ErrIncomplete, partial result alongside (io.Reader style).

Track A, sinks (lands first; independent of B and C):

  1. Engine.Redact: a done ctx at entry or any non-nil runErr → non-nil Result (Incomplete, IncompleteReason) and ErrIncomplete.
  2. RedactFile: any of the three --fail-on-incomplete channels names the file → scan.ErrIncomplete + RedactFileResult.Coverage *RedactCoverage (nil when complete; {{U38}}).
  3. --stdin redaction: both exits via resolveIncompleteExitCode; pre-commit notice; cause into {{U33}}'s shared stdin FormatterOptions constructor.
  4. examples/lambda-redact and the README snippets check err before Redacted.

Track B, caller ctx:

  1. Thread ctx through ScanContent, ScanFile, the parallel processor and the web handler.
  2. ScanText/ScanFile: done at entry → nil, ctx.Err(); fired mid-scan → partial Result (Incomplete, NotExamined) and an error wrapping ctx.Err(). Internal guards keep (Result{Incomplete}, nil).
  3. ScanText fills NotExamined, prefixes the reason once; add scan.RedactFileContext.

Track C, partials:

  1. keepMatches uses IsCoverageCutShort.
  2. Score and annotate partials before the early send.
  3. Bounded grace drain (≤100 ms) on Done.
  4. Never retry a guard outcome (by type, before any message test).
  5. scanner.go:431-447 uses the predicate; an AST guard rejects hand copies.
Detail, compatibility
  1. Engine.Redact.
    • Return fmt.Errorf("%w: %w", ErrIncomplete, cause) for every non-nil runErr: the cut-short family, the 5-minute ceiling (engine.go:265), and hard errors whose matches RunValidators discards.
    • IncompleteReason is cause text only, never input bytes. Redacted has the findings made before the cut applied.
    • errors.Is(err, context.Canceled) and errors.Is(err, context.DeadlineExceeded) keep working for caller ctxs.
  2. RedactFile.
    • The three channels are stats.IncompleteFiles, EmptyExtractionFiles and FailedFiles, the same ones --fail-on-incomplete counts.
    • Skip the clean-file passthrough for such a file, but keep the partial copy, as --file does.
    • Wrap the taxonomy cause (coverage cut short, cannot parse).
    • Images without body text are not in EmptyExtractionFiles, so they are unaffected.
    • Carrier. RedactCoverage{Incomplete, IncompleteReason, NotExamined} (the scan.Result names) behind the pointer Coverage, nil when complete. Slice fields on RedactFileResult itself would repeat the v2.4.5 scan.Finding break ({{U38}} details 5: incompatible=1). Re-measured with -overlay at 3df48ac: the pointer shape stays Comparable() and a map-key/== consumer builds; the slice shape fails with invalid map key type and struct containing []…NotExaminedFile cannot be compared. The pointer compares by identity only on cut-short results, which also carry an error. RedactFileContext returns the same type.
  3. stdin (absorbs {{U39}} Part A).
    • Pass result.Incomplete and its reason into runStdinRedaction.
    • The shared constructor ({{U33}}) sets Stats, NotExamined and OutputToFile the way file mode does.
    • Rewrite :245-249.
    • That is the whole stdin change: it matches --file, with no stdin-only refusal.
    • If this lands before {{U33}}, the exit code and stderr parts ship alone and the stdin JSON criterion waits for it.
  4. Example and README. Map ErrIncomplete to a payload-free request_id=…: redaction incomplete error. Fix README.md :211, :220, :226, :230, :247 and :253, which discard errors.
  5. Threading.
    • ScanContent derives its 5-minute ceiling from the caller's ctx.
    • ScanFile, ProcessFiles* and the worker pool take the caller's ctx as their root. These are internal signatures.
    • Pass r.Context() at server.go:899.
  6. Caller cancellation. This follows Go convention, so callers that check only err stop. The guards keep the pkg/scan gives library consumers one bool and a prose string where the CLI has a structured not-examined taxonomy #391 detection contract. There are never fewer findings with Incomplete=false.
  7. RedactFile and budgets.
    • RedactFile becomes a context.Background() wrapper.
    • Forwarding ValidatorBudgets on TextOptions/FileOptions is optional.
    • The doc comments say that extraction cannot be interrupted.
  8. keepMatches. Drop the now-unused errors import.
  9. Scoring partials. Apply contextual confidence and validation_path before :884-886, so a kept partial scores the same 75 its complete-run twin does, not 55.
  10. Grace period.
    • Validators poll every 256 lines (ctxCheckStride).
    • The 138–204 ms of post-return CPU is the work being thrown away.
    • A validator that misses the grace period is abandoned as now.
  11. No retry for guard outcomes.
    • resilience.ClassifyError returns non-retryable for execguard.IsCoverageCutShort(err), before the net.Error and substring checks.
    • Rewording the cause ({{U35}} item 6, BudgetExceededError) therefore cannot change retry behaviour.
    • A budget is a bound, not a transient fault.
  12. AST guard. Over non-test files in internal/, cmd/ and pkg/, it fails on errors.Is(x, S) outside internal/execguard for S in {DeadlineExceeded, Canceled, ErrMatchBudgetExceeded, ErrContentTooLarge, ErrValidatorPanicked}. The one exception is a switch that only picks wording after IsCoverageCutShort has matched.

Compatibility.

  • Suppression hashes.
    • Tracks A and B change no finding's Text, line or type.
    • Track C only adds findings, on cut-short runs. Complete runs are byte-identical, per the A/B.
    • hashVersionNoConfidence (internal/suppressions/suppression.go:246, the current write formula) excludes confidence, so a kept partial matches its twin's rule.
    • Legacy v1 rules (:243) include confidence. That is why item 9 is mandatory.
    • A --generate-suppressions file from a budgeted run gains entries.
  • Behaviour.
    • Withholding output is not proposed; it would change --file, --stdin and RedactFile together. Library sinks error because a Go caller has no flag but checks err.
    • stdin redaction exits 3 instead of 0, but only on a cut-short run under the flag.
    • stdin JSON gains stats (via {{U33}}).
    • RedactFile writes no passthrough for a not-examined file.
    • Library callers get errors where they used to get (cleartext, nil).
    • An expired ctx passed to ScanText now yields an error.
  • API. All additive (list under {{U38}}): redact.ErrIncomplete, scan.ErrIncomplete, Incomplete/IncompleteReason on redact.Result and AuditRecord, scan.RedactCoverage + RedactFileResult.Coverage, scan.RedactFileContext, optional ValidatorBudgets. No exported signature changes. Of the touched structs only RedactFileResult is comparable today (the other six are not); the pointer keeps it so. The new error returns are behavioural, invisible to an API snapshot: own CHANGELOG callout. {{U39}}'s Python API must not ship before this.
  • Precision and performance.
    • No pattern, keyword or score changes.
    • Tracks A and B add no per-line cost.
    • Item 11 removes two extra passes of every check for each fired budget.

Tests

Deterministic triggers only (dead ctx, size cap, unparseable bytes, stub validator, all=1ns), never a wall-clock budget racing real work. Unit tests per sink; a cmd table test over matrix A; stub tests for items 8–11; the AST guard; an unchanged goldencorpus.

Test list
  • pkg/redact:
    • Each of a pre-cancelled ctx, a past deadline, an over-cap CLOUD_RESOURCES request and a stub hard error gives ErrIncomplete and res.Incomplete.
    • The reason and AuditRecord contain no input bytes.
    • A complete request is byte-identical to today's output.
  • Lambda example: a pre-cancelled ctx gives an error and an empty Response, and no input bytes appear in the messages.
  • pkg/scan:
    • Cancelling at entry gives nil, context.Canceled.
    • A stub mid-scan cancel gives a partial result, an error, and NotExamined[0].Cause == "coverage cut short".
    • ScanText on a size-capped input gives a non-empty NotExamined.
    • RedactFile on broken.pdf, on a CLOUD_RESOURCES-only over-cap log, and with a pre-cancelled ctx gives ErrIncomplete and no output file.
    • The over-cap log plus one e-mail gives ErrIncomplete and the partial copy.
    • Cut-short cases set res.Coverage.Incomplete and list the file in res.Coverage.NotExamined; a complete run leaves res.Coverage == nil; RedactFileResult stays Comparable().
  • cmd: matrix A; --stdin stdout equals the --file copy wherever a copy exists.
  • Track C:
    • A stub that returns N matches with DeadlineExceeded keeps all N through the bridge and RunValidators, with confidence and metadata equal to the nil-error case.
    • A partial sent 10 ms after Done is kept. A stub that never returns is abandoned within the bound.
    • Under DefaultValidatorRetryStrategy(), both the cut check and its sibling run once.
    • A kept partial's generateFindingHash equals its twin's.
  • Goldencorpus: byte-identical, plus an unparseable-PDF case in golden_file_redact_test.go. Time-cut recall depends on timing, so it is pinned by the stubs instead.

Docs & CHANGELOG

pkg/redact/doc.go, pkg/scan doc comments, README library section, README-Stdin.md, COVERAGE_DISCLOSURE.md, configuration.md:249, THREAT_MODEL.md:39, the Lambda README. CHANGELOG [Unreleased]: Security (Track A), Bug Fixes (B, C).

Doc changes
  • pkg/redact/doc.go and the engine.go comments: document ErrIncomplete, and tell callers to check err before using Redacted.
  • scan.go: document the ctx semantics, and fix the Result.Incomplete comment at :240.
  • redactfile.go:50-55: "safe to share" holds only when err == nil; document Coverage.
  • README.md (:205-260): stop discarding errors.
  • docs/user-guides/README-Stdin.md: cover exit 3 for the gateway. The statement at :257 ("same exit-code semantics as file mode") then becomes true.
  • docs/COVERAGE_DISCLOSURE.md, "The library and the web UI": add pkg/redact, RedactFile and the gateway, and correct the pkg/scan NotExamined row.
  • docs/configuration.md:249 and the --validator-budget help: a budget is stopped once, its partial results are kept, and the scan is reported incomplete.
  • THREAT_MODEL.md:39: the exit-3 claim becomes true for stdin.
  • CHANGELOG:
    • The Security entry names every surface.
    • The Bug Fixes entry notes that budgeted runs gain findings and suppression entries, and that complete runs are unchanged.

Acceptance criteria

  • Engine.Redact (pre-cancelled ctx, past deadline, >5 MiB CLOUD_RESOURCES request): errors.Is(err, redact.ErrIncomplete), Result.Incomplete and AuditRecord().Incomplete true, no input bytes in reason or error; examples/lambda-redact returns an error and no redacted field.
  • RedactFile (unparseable .pdf; >5 MiB CLOUD_RESOURCES-only log): errors.Is(err, scan.ErrIncomplete), res.Coverage.Incomplete true, no file at the output path; complete run: res.Coverage == nil.
  • one.txt all=1ns (also --pre-commit-mode) and infra.log: --stdin and --file redaction both exit 0 without --fail-on-incomplete, 3 with it, disclose on stderr; stdin JSON has stats.files_not_examined == 1 (with {{U33}}).
  • ScanText/ScanFile: pre-cancelled ctx → nil, context.Canceled; stub mid-scan cancel → Incomplete, NotExamined[0].Cause == "coverage cut short", error wrapping ctx.Err(); size-capped ScanText → non-empty NotExamined, reason prefix once.
  • scan.RedactFileContext exists, RedactFile keeps its signature, RedactFileResult stays comparable ({{U38}} snapshot), /scan passes r.Context().
  • records.txt, --checks SSN,EMAIL --validator-budget SSN=<d> that fires: non-zero SSN findings, each identical (line, text, type, confidence) in the unbudgeted run; --debug shows 1 email_validator validate_content record (today 3); exit 3 under --fail-on-incomplete.
  • A stub returning N matches with DeadlineExceeded yields N findings, confidence and metadata equal to the nil-error case.
  • The AST guard passes, and fails when errors.Is(err, execguard.ErrMatchBudgetExceeded) is added outside execguard.
  • Goldencorpus byte-identical, make pr-checklist green, docs and CHANGELOG entries present.

Related

Audit item U07 · tier P0 · effort L · rank 6/51. Measurements pinned to 3df48acd (v2.5.3); a {{U##}} token is a sibling item not yet filed.

Activity

  1. sectoramen
    added
    bugSomething isn't working
    P0Audit tier P0: fail-open or cleartext leak; fix first
    on Oct 14, 2026
  2. sectoramen
    added this to the v2.6.0 milestone on Oct 24, 2026
  3. sectoramen
    added 2 commits that reference this issue on Oct 26, 2026

    fix(redaction): fail a cut-short redaction on every sink, keep partials

    1a01027

    chore(changelog): place the #770 bullet under Bug Fixes after the rebase

    d5a395c
  4. rustic
    closed this as completedin #813on Oct 27, 2026
  5. rustic
    added a commit that references this issue on Oct 29, 2026

    chore(changelog): collect release notes as one fragment per PR under …

    4063be9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Audit tier P0: fail-open or cleartext leak; fix firstbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions