Credential leakage in agent output: systemic failure to redact secrets in chat and reasoning blocks

[frogwraps]

Summary

Agents running Hermes consistently leak credential values in chat output and reasoning/thinking blocks. Despite having credential-safety instructions, the model writes literal passwords, API keys, and tokens into responses -- especially when explaining what it "fixed."

This is a systemic, recurring failure -- not a one-time mistake.

Root Cause

1. Credentials stored in agent memory get quoted verbatim when referenced in explanations
2. "Meta-discussion" is the most dangerous trigger -- an agent that just fixed a credential exposure will write the value again to "show what it fixed"
3. Reasoning/thinking blocks leak credentials in interfaces where they are visible (Discord, Telegram, CLI)
4. Post-hoc instructions are not sufficient -- the model violates them in the same turn when explaining past actions

Impact

• Severity: Critical. Actual passwords, API keys, and auth tokens written into chat transcripts.
• Requires credential rotation each time it occurs.
• Erodes user trust permanently.

Proposed Solutions

Short-term (output pipeline)

• Pre-delivery regex scrubber: scan model output for patterns matching known credential formats and auto-redact before delivery
• Think-block redaction: apply same filtering to reasoning blocks, not just final output

Medium-term (framework)

• Taint tracking: mark strings originating from credential sources (.env, credential pool) and refuse to emit tainted values in user-facing output

Long-term (architecture)

• Abstract credentials: agents should never hold credential VALUES in context -- only opaque references resolved by the tool layer at execution time

Note

Users are trying to solve this with custom skills and system prompt hardening -- none of it works because the model violates its own instructions. The fix must be in the output pipeline, not the prompt.

[teknium1]

Short-term output-pipeline recommendation from this issue addressed in PR #21193 (commit fb1ce79, merged 2026-05-07).

Secret redaction is now ON by default. The pre-delivery regex scrubber you described already existed as redact_sensitive_text in agent/redact.py, wired into send_message_tool.py (the gateway's user-facing output path), _capture_log_snapshot, context_compressor, and copilot_acp_client — it was just gated off by default. Flipping the default closes the short-term half of this issue.

The medium/long-term recommendations (taint tracking, opaque credential references) remain open design questions — happy to keep them on the radar as separate issues if you want to scope them. The mechanical "scrub known credential patterns from chat output" ask is now the default behaviour.

[frogwraps]

This response needs a significant clarification because the fix described does not solve the problem reported in this issue. I'm speaking from direct, repeated experience across multiple real-world incidents. Let me be concrete.

What was actually deployed

The PR flips _REDACT_ENABLED from false to true on the existing redact_sensitive_text() regex scrubber. That function applies ~10 regex patterns — known-prefix tokens (sk-, ghp_, etc.), KEY=VALUE env assignments, "key": "value" JSON fields, auth headers, JWTs, private key blocks, DB connection strings, URL query params, form bodies, Discord mentions, and phone numbers.

Why this does not solve the problem — with real examples

1. Natural language / meta-discussion leakage (the most dangerous trigger)

The model does not leak credentials only in structured token formats. It leaks them when explaining what it did.

Real incident: After being told to stop exposing credentials, the agent read a password from the env file, then tried to explain what it had done differently this time. The explanation included the password verbatim in natural English prose — "I used the password XXXX to log in" — because to the model, it was describing its successful action. No regex pattern matches a password expressed as a quoted word in a sentence. The _PREFIX_RE won't fire because the password has no standard token prefix. The _ENV_ASSIGN_RE won't fire because there's no = sign. The _JSON_FIELD_RE won't fire because it's not in a JSON structure. The value passes through completely unredacted.

This is not theoretical. It happened multiple times in a single session.

2. Reasoning/thinking blocks are not scrubbed

The regex pipeline runs in send_message_tool.py, _capture_log_snapshot, context_compressor, and copilot_acp_client. It does not run on reasoning/thinking blocks — which are visible in the user-facing output on Discord, Telegram, and CLI when display.show_reasoning is enabled.

Real incident: The agent's internal reasoning explicitly quoted credential values while weighing whether to use them. Because reasoning blocks are captured and emitted separately from the final response content, the model's own deliberation was sent to the user interface with the credential in plain text. The regex scrubber never touched it.

3. Tool results containing credentials go back into model context

When the agent reads a config file or env file via a tool (terminal, file read), the tool result — including any credential values — enters the model's context window. The model can then paraphrase, explain, or accidentally re-emit those values in its next response, in any format.

Real incident: cat ~/.hermes/.env returned the file contents in a tool result. The model then summarized what it found, re-stating a credential in a different syntactical structure. The extracted credential did not match any regex pattern because it was no longer in KEY=VALUE format — the model had rephrased it.

Another incident: Reading a JSON config file with service credentials, the model extracted the password value from the JSON structure and then referred to it in natural language. The regex might have caught the original "password": "value" in the tool result, but the model's subsequent natural-language reference to that value was not in any pattern the scrubber recognizes.

4. Non-standard credential formats

Pattern-based redaction only catches what the patterns know about.

Real example: Custom tokens, service-specific API keys, and internally-generated passwords that don't start with any of the 15-20 known prefixes in _PREFIX_RE pass through untouched. A token like XXXX-api-key-YYYY or a password like MyP@ssw0rd! has no structural signature the regex can identify. The regex approach is inherently incomplete — it can only match what the maintainers have thought of.

5. The core architectural problem remains unaddressed

This issue's body outlined three layers of fix:

Layer	What	Status
Short-term	Regex output scrubber	✅ Flipped default ON
Medium-term	Taint tracking — mark strings from .env/credential pool and refuse to emit tainted values	❌ "Remains open design question"
Long-term	Opaque credential references — agents never hold credential VALUES, only references resolved at execution time	❌ "Remains open design question"

The short-term fix alone is insufficient because the model cannot be prompted or regex'd out of quoting credentials it holds in context. The risk surface is fundamentally the same: every credential the agent can see, the agent can accidentally write out. The only structural fix is taint tracking or opaque references — remove the credential VALUE from the LLM's context entirely.

What this means operationally

Every time a credential is leaked this way, it must be rotated. For a production deployment with 24+ distinct credentials (API keys for GHL, Hostinger, Outscraper, DataForSEO, Brave Search, n8n, Titan Email, GitHub, Cloudflare, Obsidian, multiple SMTP/IMAP accounts, VPS root passwords, etc.), a single incident means rotating everything — because you don't know which ones were captured in transcript history. This is not theoretical; it's been our reality.

I am reopening this issue because the described fix does not meet the severity of the problem reported. The P0 label is appropriate — this is a critical security vulnerability — and the P1 downgrade on PR #21193 does not reflect the actual risk. The regex default flip is a helpful incremental improvement, but closing this issue as "completed" when none of the core problems (reasoning block leakage, meta-discussion leakage, natural language leakage, and especially the structural need for taint tracking / opaque references) have been addressed is premature.