redaction: overlapping findings leave part of a value in cleartext; identical spans render by sort order, not a rule #773

mkdocx.py OUT PARA... writes a minimal WordprocessingML package, one <w:p> per argument. mkdoc OUT TEXT is a short Go program calling internal/olefixture.LegacyDoc(TEXT+"\r", nil), built in a module whose path is under github.com/awslabs/ferret-scan/v2/, with a replace to the checkout.

F=${F:-ferret-scan}
show() { case $1 in
  *.txt)  cat "$1" ;;
  *.docx) unzip -p "$1" word/document.xml | perl -pe 's#</w:p>#\n#g; s/<[^>]*>//g' ;;
  *.doc)  LC_ALL=C strings "$1" | grep -a -E 'Residence|Located|Passport' ;; esac | grep -v '^$' | tail -n1; }
i=0
for line in 'Residence: 77 August Lane, Boise ID 83702' 'Located at 100 Martin Luther King Blvd'; do
  i=$((i+1)); printf '%s\n' "$line" > x$i.txt; python3 -I mkdocx.py x$i.docx "$line"; ./mkdoc x$i.doc "$line"
done
set --; : > x3.txt     # the identical span after 11 unrelated email lines (13 findings)
for n in $(seq 1 11); do echo "Contact $n: user$n@example.com" >> x3.txt; set -- "$@" "Contact $n: user$n@example.com"; done
echo 'Passport number: 512010001' >> x3.txt; set -- "$@" 'Passport number: 512010001'
python3 -I mkdocx.py x3.docx "$@"; ./mkdoc x3.doc "$(printf '%s\r' "$@")"
for c in x1 x2 x3; do for ext in txt docx doc; do o=""
  for s in simple format_preserving; do rm -rf out
    $F --file $c.$ext --enable-redaction --redaction-strategy $s --redaction-output-dir out --quiet --format json >/dev/null 2>&1
    o="$o rc=$? $(show "$(find out -type f -name $c.$ext)") |"; done
  printf '%-7s%s\n' $c.$ext "$o"; done; done

x1.txt  rc=0 Residence: 77 August [PERSON-NAME-REDACTED] ID 83702 | rc=0 Residence: 77 August *********** ID 83702 |
x1.docx rc=0 Residence: [US_STREET_ADDRESS-REDACTED], Boise ID 83702 | rc=0 Residence: **************, Boise ID 83702 |
x1.doc  rc=0 Residence: 77 ***********, Boise ID 83702 | rc=0 Residence: 77 ***********, Boise ID 83702 |
x2.txt  rc=0 Located at [US_STREET_ADDRESS-REDACTED] | rc=0 Located at *************************** |
x2.docx rc=0 Located at [US_STREET_ADDRESS-REDACTED] | rc=0 Located at *************************** |
x2.doc  rc=0 Located at 100 ****************** Blvd | rc=0 Located at 100 ****************** Blvd |
x3.txt  rc=0 Passport number: [SSN-REDACTED] | rc=0 Passport number: *****0001 |
x3.docx rc=0 Passport number: [PASSPORT-REDACTED] | rc=0 Passport number: ********* |
x3.doc  rc=0 Passport number: ********* | rc=0 Passport number: ********* |

files_processed: 1 on every run. Under synthetic the Residence line keeps 77 August in .txt and , Boise in .docx; .doc writes the same 77 ***********, Boise as under the other two strategies ({{U45}} owns that remap). Findings (--format sarif --show-match, 1-based byte columns, end exclusive): x1 = US_STREET_ADDRESS 95 77 August Lane [12,26), PERSON_NAME 92 August Lane [15,26) (contained), PERSON_NAME 67 Lane, Boise [22,33). x2 = US_STREET_ADDRESS 65 over the whole address [12,39), PERSON_NAME 93 Martin Luther King [16,34) inside it. The accented-names fixture line Renée Müller, José García, François Lefèvre (internal/preprocessors/plaintext_sniff_test.go; PERSON_NAME 100 Renée Müller, PERSON_NAME 100 Müller, José García) leaves Renée in .txt and , José García in .docx, under both strategies. With --redaction-audit-log, x1.txt records total_redactions: 1, successful_redactions: 1, failed_redactions: 0 for 3 findings, and the names line the same for 2. .xls/.ppt go through the same legacyole loop as .doc (measured on .doc).

Roster: 20 lines Passport number: 5120100NN (NN = 01..20), 40 findings, through --stdin. Three repeated runs hash identically and --file gives the same 12, so the split is fixed per file, not random per run.

txt  format_preserving: last 4 kept 12, full mask 8
txt  simple:            [SSN-REDACTED] 12, [PASSPORT-REDACTED] 8
docx format_preserving: last 4 kept 0 of 20

Other identical pairs (synthetic: NPI 1000000004 with a computed check digit, 2125550100 in 555-01xx), each line alone, then after 11 unrelated email lines (default strategy; under simple the label flips the same way, e.g. [NPI-REDACTED] → [PHONE-REDACTED]):

Passport number: 512010001             [PASSPORT 100 + SSN 50]          *********   ->  *****0001
driver license 512010001               [DRIVERS_LICENSE 85 + SSN 50]    *********   ->  *****0001
Provider NPI: 1000000004 at the clinic [NPI 100 + PHONE 10]             **********  ->  ******0004
MRN: 2125550100                        [MRN 80 + PHONE 5]               **********  ->  ******0100
Account number: 2125550100             [US_BANK_ACCOUNT 85 + PHONE 15]  **********  ->  ******0100

SECRETS/EMAIL. A value from python3 -I -c 'import secrets,string; a=string.ascii_letters+string.digits; print("".join(secrets.choice(a) for _ in range(24)))', written as password = "<value>@example.org", gives API_KEY_OR_SECRET 75 and BUSINESS 40 on the same columns [13,49). Output shapes (X = the value's first character, verbatim), identical with 0, 5, 11, 12, 20, 40 or 60 unrelated lines before and after:

simple            password = "[EMAIL-REDACTED]"
format_preserving password = "X***********************@example.org"

secret = "4532-0000-0000-0009" (VISA 85 + API_KEY_OR_SECRET 75; computed Luhn digit) writes ****-****-****-0009. Scanning each of these two lines alone with pkg/scan.ScanText and passing its findings to pkg/scan.RedactText in reverse order writes a full mask for both lines, so the CLI result is input order, not a rule.

Threshold, through plaintext.RedactString (format_preserving; k unrelated matches, then the identical pair; P = *********, S = *****0001):

[..,PASSPORT,SSN]  n=2..62: PPPPPPPPPPPSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
[..,SSN,PASSPORT]  n=2..62: SSSSSSSSSSSPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP
200 shuffled input orders (PASSPORT rendering / SSN rendering): n=2 99/101, n=12 100/100, n=13 102/98, n=50 101/99
ResolveOverlaps kept (identical pair): 2 of 2

• plaintext (--stdin, text --file, pkg/scan, pkg/redact, svg, the rtf inner pass): the sort key is (line desc, strings.LastIndex(FullLine, Text) desc). Identical spans tie. For a partial overlap the later span is written first, and the earlier span's text can no longer be found, so its head survives.
• Office: replacements are keyed by value, first write wins, and applied longest-first by a left-to-right strings.Replacer, so the earlier-starting value is written and the later value's tail survives. The comment at :1062 says this ordering "keeps partially-overlapping values leak-safe".
• legacy OLE: matches are overwritten in input order. The comment at :160-161 defers to "the caller's overlap resolution", but the only non-test callers of ResolveOverlaps are plaintext/redactor.go:228 and office/redactor.go:926.
• tagmeta (tagmeta/overwrite.go:105-237, Plan + mergeOccurrences) is already correct: it locates every occurrence in the original bytes, merges overlapping spans and masks any span built from more than one match. Its comment (:94-102) describes this exact failure.
• internal/parallel/validator_runner.go:243-251 and PR fix(redaction): remove a reported value everywhere, and verify the written output #574 ("Also left: … crossing overlaps … a real second defect") record the partial-overlap leak. No issue was filed.
• redactverify.SweepRemaining (internal/redactverify/redactverify.go:212), Office recordCoverageForRemainingParts (office/partcoverage.go:118) and the written-output check (docs/user-guides/README-Redaction.md:242-246) all look for whole values.
• replacement.FormatPreserving (internal/redactors/replacement/replacement.go:138-154) keeps the last 4 digits for SSN, PHONE and card types, the first character and domain for EMAIL/GMAIL/BUSINESS, and two octets for IP, and masks everything else. So the arbitrary winner decides which bytes survive.

git ls-files | grep -v -x 'config.yaml' | grep -v -E '\.(png|gz|pyc)$' | sed "s#^#$PWD/#" > /tmp/files.txt   # 1,501
cd "$(mktemp -d)"   # absolute paths from an empty dir: no project config is auto-loaded
ferret-scan --format sarif --confidence all --limit 0 --quiet --output repo.sarif $(cat /tmp/files.txt)

Results are grouped by (uri, startLine, line text) and each pair is classified by SARIF byte columns: identical, strictly contained, or partial. Touching spans count as disjoint, and 13 findings have no columns. Unaffected paths: tagmeta formats (audio, video), images (metadata stripped), PDF (refused). The same classifier over the 52 golden *.sarif.json files finds 1 identical and 2 contained pairs, all in clip.wav (tagmeta), and no partial pair; file_docx_numeric_character_reference pairs a metadata finding (AUTHOR_INFO) with a body SSN that both report "line 1" on different FullLines, so it is not a group.

• Identical: PASSPORT/SSN 52, API_KEY_OR_SECRET/OTP_SECRET 13, NPI/PHONE 10, API_KEY_OR_SECRET/VISA 8, PHONE/US_BANK_ACCOUNT 6, ABA_ROUTING/SSN 3, DRIVERS_LICENSE/SSN 3, PHONE/SSN 3, DRIVERS_LICENSE/PHONE 2, and 6 singletons. In 85, exactly one type keeps the last 4. --file oracle: each of the 51 files redacted alone; the reported columns of the written line hold the last-4 rendering or a full mask (no pair was left in cleartext; every file produced output).
• Partial: PERSON_NAMEPERSON_NAME 17, US_STREET_ADDRESSPERSON_NAME 2, AZURE_RESOURCE_IDGCP_RESOURCE_NAME 1, GCP_RESOURCE_NAMEGCP_RESOURCE_NAME 1, BUSINESS~INTELLECTUAL_PROPERTY 1. Oracle: each line through --stdin under simple and format_preserving; it leaks if the earlier span's head is immediately followed by the later span's replacement.
• Contained: IBAN>PHONE 29, OTPAUTH_URI>API_KEY_OR_SECRET 25, OTPAUTH_URI>BUSINESS 16, API_KEY_OR_SECRET>AWS_SECRET_ACCESS_KEY 15, IP_ADDRESS>DATE_OF_BIRTH 14, and 19 smaller rows. In 21 the survivor's format_preserving rendering keeps bytes (IP_ADDRESS>DATE_OF_BIRTH 14, BUSINESS>DATE_OF_BIRTH 3, EMAIL>EMAIL 2, MASTERCARD>PHONE 2). In 17 of those (the two DATE_OF_BIRTH rows) it keeps a byte inside the contained span that the contained type would mask. None of the 21 has a SECRETS-check match inside.

"Copied" means the renderer writes the original byte back at the same offset by construction: last 4 digits, an email's first character and domain, two IP octets, a synthetic card's issuer prefix. replacement should return these ranges with the string rather than leave them to be inferred. The representative rule picks labels only; the Phase 3 same-span arbitration policy (docs/proposals/ARCHITECTURE_AND_PLAN.md:428) may later change which label wins, never which bytes are copied. SECRETS membership comes from Match.Check ({{U35}}). At runtime {{U35}} counts an unattributed match under unattributed and {{U31}} redacts it with simple plus a disclosed warning; inside a group it is treated as SECRETS (strictest).

Requirements from other drafts, asserted here with synthetic match pairs:

• {{U31}} Resolution 8: members of one group resolving to different strategies take the most protective; two adjacent, non-overlapping values of different checks keep their own strategies (the group must not over-merge).
• {{U18}} Resolution 6: a national-ID match on the same bytes as SSN or PHONE is fully masked under format_preserving, and the SSN/PHONE finding is still reported.
• secrets: a password or token in URI user info (scheme://user:pass@host) is never detected, so it survives redaction #766 Resolution 6 (and its REDIS_URL acceptance criterion): a SECRETS match nested inside an EMAIL match leaves no secret byte under any strategy.

Expected under the rule:

• Passport line: ********* at any file size in every format; [PASSPORT-REDACTED] under simple in .txt/.docx.
• Residence line: Residence: ********************* ID 83702 / Residence: [US_STREET_ADDRESS-REDACTED] ID 83702.
• Password line: a full mask / [API_KEY_OR_SECRET-REDACTED].
• Card-in-secret line: ****-****-****-****.
• MLK .doc: a full mask.

Re-measured with the same command over the same 1,501 files and the same SARIF classifier. PR #761's head (a0ba05b) gives 10,138 findings and 146 contained, because the API_KEY_OR_SECRET>API_KEY_OR_SECRET row drops from 8 to 7; the identical and partial pairs are unchanged. PR #762's head (64e6589) gives 10,134 findings with every pair unchanged; the four golden *.sarif.json files it adds hold no overlapping pair.