secrets: a password or token in URI user info (scheme://user:pass@host) is never detected, so it survives redaction #766

Summary

SECRETS never reports a credential in URI user info (RFC 3986 §3.2.1), scheme://user:PASSWORD@host or https://TOKEN@host, the usual shape of DATABASE_URL, broker, proxy and registry URLs and JDBC DSNs. Through the CLI, 0 of 185 cases (25 schemes × 7 placements, plus lengths) report anything from any validator, every file processed, while DB_PASSWORD=<pw> reports the same password at 100. Since only findings reach the redactor, a DATABASE_URL-only .env gets no redacted copy, a YAML file with an extra api_token: gets a "redacted" copy still holding the password, the --stdin --enable-redaction gateway passes it through and --pre-commit-mode exits 0. The cause is structural: every capture path needs a provider prefix, a stem before =/: or a quoted base64/hex run, and the M24 cap then holds an uncorroborated candidate at 55, under the threshold of 60. A throwaway prototype of the Resolution reports 185/185 at ≥ 85, changes 0 of main's findings (11,515 SECRETS on 202 public files) and adds 10 MEDIUM in 164,054 public text files, all published examples. --help SECRETS and docs/checks.md:160 promise password detection.

Reproduction

Measured on main @ 3df48ac (v2.5.3); line references are at d5c4da1 (main, 2026-10-14). main has since gained dependency patch bumps and #751, #752 and #759 (changelog CI, perfguard, build tooling), none of which changes detection or redaction.

Isolated config, no suppressions, consonant-and-digit values, .invalid hosts.

pw=$(LC_ALL=C tr -dc 'BCDFGHJKLMNPQRSTVWXZbcdfghjkmnpqrstvwxz23456789' </dev/urandom | head -c 16)
u="DATABASE_URL=postgresql://app_user:$pw@db1.orders.invalid:5432/appdb"
for l in "$u" "DB_PASSWORD=$pw"; do printf '%s\n' "$l" | ferret-scan --stdin --format json --confidence all --quiet | jq -c '[.results[] | "\(.type) \(.confidence)"]'; done
printf '%s\n' "$u" | ferret-scan --stdin --enable-redaction --quiet 2>/dev/null | grep -c -F "$pw"

Prints [] (URI), ["API_KEY_OR_SECRET 100"] (keyword), then 1 (password in gateway output).

#!/bin/sh
set -eu
FERRET=${FERRET:-ferret-scan}
W=$(mktemp -d "${TMPDIR:-/tmp}/uri-userinfo.XXXXXX"); D="$W/in"; mkdir "$D" "$W/red"
rnd() { LC_ALL=C tr -dc 'BCDFGHJKLMNPQRSTVWXZbcdfghjkmnpqrstvwxz23456789' </dev/urandom | head -c "$1"; }
PW=$(rnd 16); TOK=$(rnd 40)
URI="postgresql://app_user:$PW@db1.orders.invalid:5432/appdb"
# A. URI user info in keyword-free placements
printf 'DATABASE_URL=%s\n' "$URI" > "$D/a01.env"
printf 'REDIS_URL=redis://default:%s@cache.orders.invalid:6379/0\n' "$PW" > "$D/a02.env"
printf 'spring:\n  datasource:\n    url: jdbc:mysql://app_user:%s@db1.orders.invalid:3306/shop\n' "$PW" > "$D/a03.yaml"
printf '{"mongo": {"uri": "mongodb+srv://app_user:%s@cluster0.orders.invalid/appdb?retryWrites=true"}}\n' "$PW" > "$D/a04.json"
printf 'engine = create_engine("postgresql+psycopg2://app_user:%s@db1.orders.invalid/appdb")\n' "$PW" > "$D/a05.py"
printf "broker_url = 'amqp://app_user:%s@mq.orders.invalid:5672//'\n" "$PW" > "$D/a06.py"
printf 'services:\n  api:\n    environment:\n      - DATABASE_URL=mysql://app_user:%s@db:3306/shop\n' "$PW" > "$D/a07-compose.yml"
printf 'Connect with `psql postgresql://app_user:%s@db1.orders.invalid/appdb` and run the migration.\n' "$PW" > "$D/a08.md"
printf '2026-10-01T12:00:00Z ERROR could not connect to postgres://app_user:%s@db1.orders.invalid:5432/appdb: timeout\n' "$PW" > "$D/a09.log"
printf 'CONNECTION_STRING=sqlserver://app_user:%s@db1.orders.invalid:1433?database=shop\n' "$PW" > "$D/a10.env"
printf 'HTTPS_PROXY=http://proxy_user:%s@proxy.orders.invalid:3128\n' "$PW" > "$D/a11.env"
printf 'git clone https://%s@git.orders.invalid/orders/app.git\n' "$TOK" > "$D/a12.sh"
printf -- '--extra-index-url https://__token__:%s@pypi.orders.invalid/simple/\n' "$TOK" > "$D/a13-requirements.txt"
printf '      "resolved": "https://deploy:%s@registry.orders.invalid/left-pad/-/left-pad-1.3.0.tgz",\n' "$TOK" > "$D/a14-package-lock.json"
# B. controls: the same password through a keyword
printf 'DB_PASSWORD=%s\n' "$PW" > "$D/b01.env"
printf 'DATABASE_URL=postgres://db1.orders.invalid:5432/appdb?user=app_user&password=%s\n' "$PW" > "$D/b02.env"
printf 'DB_PASSWORD=%s\n' "$URI" > "$D/b03.env"
"$FERRET" --file "$D" --recursive --format json --confidence all --show-match --limit 0 --quiet > "$W/scan.json"
jq -r '.stats | "files_processed=\(.files_processed)/\(.total_files) skipped=\(.files_skipped) findings=\(.total_findings)"' "$W/scan.json"
for f in $(ls "$D"); do
  printf '%-22s %s\n' "$f" "$(jq -r --arg f "$f" --arg pw "$PW" --arg tok "$TOK" '[.results[] | select(.filename == $f)
    | "\(.type) \(.confidence) \(.metadata.detection_method // "-") span=\(if .text == $pw or .text == $tok then "value" elif (.text | contains($pw)) or (.text | contains($tok)) then "wider(\(.text | length))" else "other" end)"]
    | if length == 0 then "NONE" else join("; ") end' "$W/scan.json")"
done
# C. the sink: redaction, the stdin gateway, pre-commit
printf 'DATABASE_URL=%s\n' "$URI" > "$W/red/r01.env"
printf 'database:\n  url: %s\napi_token: "%s"\n' "$URI" "$(rnd 32)" > "$W/red/r02.yaml"
printf 'DB_PASSWORD=%s\nDATABASE_URL=%s\n' "$PW" "$URI" > "$W/red/r03.env"
"$FERRET" --file "$W/red" --recursive --enable-redaction --redaction-output-dir "$W/out" --format json --limit 0 --quiet > "$W/red.json"
for f in r01.env r02.yaml r03.env; do
  p=$(find "$W/out" -name "$f" 2>/dev/null | head -1)
  n=$(jq --arg f "$f" '[.results[] | select(.filename == $f)] | length' "$W/red.json")
  if [ -n "$p" ]; then echo "$f findings=$n redacted_copy=written password_in_copy=$(grep -c -F "$PW" "$p" || true)"
  else echo "$f findings=$n redacted_copy=none"; fi
done
echo "stdin results=$(printf 'DATABASE_URL=%s\n' "$URI" | "$FERRET" --stdin --format json --quiet 2>/dev/null | jq '.results | length')"
echo "stdin gateway: password_in_stdout=$(printf 'DATABASE_URL=%s\n' "$URI" | "$FERRET" --stdin --enable-redaction --quiet 2>/dev/null | grep -c -F "$PW" || true)"
rc=0; "$FERRET" --file "$W/red/r01.env" --pre-commit-mode >/dev/null 2>&1 || rc=$?; echo "pre-commit r01.env exit=$rc"
rc=0; "$FERRET" --file "$D/b01.env" --pre-commit-mode >/dev/null 2>&1 || rc=$?; echo "pre-commit b01.env exit=$rc"

files_processed=17/17 skipped=0 findings=3
a01.env … a14-package-lock.json  NONE
b01.env                API_KEY_OR_SECRET 100 keyword_pattern span=value
b02.env                API_KEY_OR_SECRET 100 keyword_pattern span=value
b03.env                API_KEY_OR_SECRET 100 keyword_pattern span=wider(68)
r01.env findings=0 redacted_copy=none
r02.yaml findings=1 redacted_copy=written password_in_copy=1
r03.env findings=1 redacted_copy=written password_in_copy=0
stdin results=0
stdin gateway: password_in_stdout=1
pre-commit r01.env exit=0
pre-commit b01.env exit=1

Root cause

In internal/validators/secrets/validator.go:

1. No capture path anchors on URI syntax. Quoted (:499, :556-569) and unquoted (:1000, stems :961-971) patterns need a stem right before =/:; a URI password follows user:. The entropy path (:716-753, :487) needs quotes around a run excluding :, @, .. connectionstring (:413) only corroborates.
2. The M24 cap (:2406-2413) holds a candidate with no specific pattern or token-bounded keyword at 55, below the keyword path's > 60 (:653, :2032): 5 of 7 line shapes score 55, 85–100 uncapped.
3. No owner: EMAIL's hasURLStructureAfter (email/validator.go:972-1000) rightly treats user@host:port after :// as a URI; SECRETS never claimed it.

Width of the class

• Every scheme (0/185) and user-info shape (empty user, JDBC, +psycopg2, multi-host, percent-encoded, uppercase); tokens as password or user name; --file (13 types), --stdin, the gateway (library paths share ValidateContentCtx).
• Today's hits are accidents: a password query parameter, a key stem (whole URI), a stem-ending user name (overrun), provider prefixes, EMAIL's <pw>@host.

Resolution

1. Add findURIUserinfo(line), a separate candidate source in ValidateContentCtx's line loop (:648-654), the region fix(secrets): stop reporting code expressions as credentials in Python, JS/TS, Go and HCL files (#753) #762 rewrites (it adds gateable and gateDrops). The uri_userinfo candidate is non-gateable, like {{U20}}'s structural sources. It is prefiltered on :// and @ and runs outside containsSecretIndicators, changing none of its arms; not a keyword pattern, so {{U36}}'s gate guard does not apply.
2. Parse the RFC 3986 authority, not a scheme list or label: any scheme (JDBC too), last-@ split (as Go net/url and WHATWG), §3.2.1 characters only (templates fail by grammar), validated host (a path or query @ never matches); hand-written, as net/url.Parse fails JDBC and host lists and echoes the password (below).
3. Value-intrinsic dispositions, no line or host veto (author-controlled, TM-11): references and *** give no candidate; default vocabulary and implausible values are LOW (postgres:postgres can be real).
4. Score as structural: detection_method: uri_userinfo, threshold 50, a structural flag exempting it from the M24 cap; other adjustments unchanged.
5. Existing type, exact span, no new metadata key (validators: every finding's metadata map sits exactly at a Go bucket cliff — the next key added costs ~342 B per finding #621): getSecretType decides (ghs_… stays GITHUB_TOKEN); the span is the password as written, so the redacted URI still parses; no URI part in metadata, logs or --explain.
6. Additive overlap: an identical span keeps the first claim (:308); a span inside an emitted SECRETS finding is dropped after thresholds; EMAIL's <pw>@host stays (cross-validator deconfliction: 5.15% of findings are sub-spans of a wider, more-confident finding, and only redaction knows it #687); redacting SECRETS inside it is redaction: overlapping findings leave part of a value in cleartext; identical spans render by sort order, not a rule #773's rule; this adds the fixture (land after redaction: overlapping findings leave part of a value in cleartext; identical spans render by sort order, not a rule #773 or mark it pending).
7. Compatibility: no existing finding's Type, Text, line, context or confidence changes, so suppression hashes (internal/suppressions/suppression.go:301-348) still match; new findings change exit status (r01.env 0 → 1) and add redacted copies. b03's whole-URI span stays (PR fix(secrets): end unquoted value at a query-string parameter separator #760 leaves it); narrowing it would move its hash.
8. Linear: each authority ends at the next /, so O(line length); no length cap (a suppressor); the complexity guard pins dense and no-terminator lines.

Tests

• goldencorpus secrets_uri_userinfo and _negative, Tier 1 FileCases, redaction goldens, unit and self-policing tests (below); goldens unchanged.

Docs & CHANGELOG

• --help SECRETS (:2603): list URI user info; fix the :2627 run-on ("handling.words").
• Secrets README.md: "Credentials in URIs" (grammar, LOW band, recall cost) and a :81-86 bullet; docs/checks.md:156-162.
• CHANGELOG.md [Unreleased] 🔒 Security: the detection, its exit-status and redacted-copy effect; existing findings and hashes unchanged.

Acceptance criteria

• Full script: one SECRETS finding per a01–a14, span=value, ≥ 60; b01–b03 byte-identical to main.
• Same run: r01.env and r02.yaml copies written with password_in_copy=0, stdin results=1, password_in_stdout=0, pre-commit r01.env exit=1.
• Matrix A 185/185 (text = password, ≥ 60); matrix B forms give the exact span, unchanged rows equal main's with no nested second finding; 24 structural decoys silent, 6 default-credential decoys ≤ LOW.
• Additivity: goldens unchanged; on a public corpus named in the PR every main finding keeps Type, Text, LineNumber, confidence.
• a14 reports whatever containsSecretIndicators returns for it; no arm of that function changes.
• No new type; metadata-key-census.txt unchanged; detection_method is uri_userinfo; the complexity guard has both targets and passes.
• REDIS_URL=redis://default:<pw>@cache.orders-svc.io is a TestGoldenRedact case; with redaction: overlapping findings leave part of a value in cleartext; identical spans render by sort order, not a rule #773 in, no password byte survives any strategy (main: X***************).
• The Docs & CHANGELOG items above are present.

Related

• secrets: a "url", "checksum" or version/boolean member anywhere on a line skips token and keyword detection on that line #765 (line skip, :856-863), redaction: overlapping findings leave part of a value in cleartext; identical spans render by sort order, not a rule #773 (ResolveOverlaps; first-character leak when SECRETS nests in EMAIL), secrets: Stripe, GitLab and Slack tokens are matched at a fixed length, so redaction leaves each tail in the clear #769/{{U19}}/{{U36}} (edit containsSecretIndicators first; {{U36}} owns per-line cost), {{U20}} (auth headers, .netrc, curl -u).
• secrets: an unquoted value runs past '&' in a query string, so two parameters are reported (and redacted) as one #758 / PR fix(secrets): end unquoted value at a query-string parameter separator #760: a query-string value ends at &key=; measured at its head, ?user=u&password=<pw>&sslmode=require changes from <pw>&sslmode=require to <pw>. Matrix B (?user=u&password=<pw> at end of line; ;password= JDBC) and b02 are unchanged.
• secrets: the #749 list-tail splitter is quadratic on separated assignments and reports every element of a minified line #754 / PR fix(secrets): make the list-tail and collection scans linear, and stop reporting every element of a minified line (#754) #761: on …;Password=<pw>;Encrypt=True;TrustServerCertificate=False main also reports Encrypt=True and TrustServerCertificate=False at 100; PR fix(secrets): make the list-tail and collection scans linear, and stop reporting every element of a minified line (#754) #761's head (a0ba05b) reports only <pw> (100).
• cross-validator deconfliction: 5.15% of findings are sub-spans of a wider, more-confident finding, and only redaction knows it #687 (EMAIL's <pw>@host), validators: every finding's metadata map sits exactly at a Go bucket cliff — the next key added costs ~342 B per finding #621 (metadata cliff); closed: Generic secret assignments (password=, api_key=, token=) are not detected at all, despite the code claiming to catch them #360, secrets: unquoted-assignment detection is single-value, single-line and keyword-adjacent — quoted secrets in JSON arrays, nested objects and YAML sequences are missed entirely #746.

_{Audit item U42 · tier P0 · effort M · rank 2/51. Measurements pinned to 3df48acd (v2.5.3); a {{U##}} token is a sibling item not yet filed.}