LLM inspection for suspicious Write/Edit content

[manuelschipper]

Problem

Write/Edit content inspection is currently deterministic regex patterns only. While this catches common destructive patterns, secrets, and obfuscation, creative or domain-specific payloads can slip through. The deterministic layer is fast and catches the obvious cases, but has blind spots for:

• Obfuscated code that doesn't match known patterns
• Context-dependent payloads (e.g., modifying package.json scripts to run something malicious)
• Novel exfiltration techniques that avoid pattern signatures

Proposed solution

Add an optional LLM inspection layer for Write/Edit content:

1. When content inspection flags something as suspicious (partial match, heuristic trigger) but not definitively malicious, route to the LLM with the content + file path + recent conversation context
2. For high-risk file types (shell scripts, CI configs, package manifests, credential files), optionally always route through LLM regardless of deterministic results
3. LLM prompt context: "This content is about to be written to [path]. Given the recent conversation context, should this write be allowed?"

This complements the deterministic layer — the fast path handles 95% of cases, the LLM handles the ambiguous remainder.

Context

Raised in the Show HN discussion by several commenters:

• gruez: "given that you allow npm test, it's not too hard to bypass protections by first modifying package.json so npm test runs an evil command"
• injidup: "what about simply a base64 encoded string of text dropped into the code designed to be unpacked and evaluated later... Will any of these fast scanning heuristics work against such attacks?"
• ibrahim_h: "the scariest exfiltration pattern isn't a single bad command, it's a chain of totally normal ones. Agent reads .env, writes a script that includes those values, then runs it. Every step looks fine individually."

Committed to on HN: "LLM inspection for Write/Edit: for content that's suspicious but doesn't match any deterministic pattern, route it to the LLM for a second opinion"

[FeilixX]

In my logs (~25k Claude Code actions over 2 months), the "chain attack" pattern ibrahim_h described is real — but most instances are benign. Agent reads config, writes a new file referencing those values, runs tests. Structurally identical to a malicious chain (read .env → write exfil script → execute).

One signal I've noticed that might help distinguish: malicious-looking chains tend to cluster at session boundaries (early or late), while normal config-propagation chains happen mid-session during active development. Timing patterns might be more discriminating than content patterns alone, especially for the "every step looks fine individually" scenario.

Also worth noting: even with LLM inspection, some chains will slip through. Having a complete operation timeline (not the agent's own summary) makes post-incident forensics possible — you can trace the full read → write → execute sequence and see exactly when each step happened.

Curious if you've seen similar patterns in nah's decision logs.

[manuelschipper]

Implemented in v0.5.1 (just released).

What landed:

• Every Write/Edit call goes through deterministic content inspection first, then LLM inspection when the LLM layer is enabled
• LLM can only veto (make stricter), never loosen — if deterministic says allow, LLM can escalate to ask or block
• Write sends file path + content to the LLM; Edit sends file path + old/new diff
• LLM warnings surface as systemMessage in the hook output — visible before the user approves the write
• llm_max_decision cap respected (LLM block + cap=ask -> user prompted)
• Content capped at 8KB, fail-open on LLM errors

This addresses the HN feedback about creative payloads, context-dependent modifications (package.json scripts), and obfuscated content that pattern matching can't catch.

Upgrade: pip install --upgrade nah

[manuelschipper]

@FeilixX

Thanks for sharing. 25k actions is a good dataset. I had not thought about session-boundary clustering, nor done that kind of analysis yet.

Read write execute chains are normal config propagation, That's partly why we went with LLM-as-veto here rather chains structurally, the LLM gets the conversation transcript so it can tell the difference between "agent writing test config" and "agent writing an exfil script." Pattern matching alone can't make that call.

For the "every step looks fine individually" problem: I have taint tracking on the roadmap (correlating sensitive reads with subsequent writes/executes at the session level) Right now the defense is layered: deterministic patterns catch the obvious stuff, LLM catches the creative stuff, and the structured decision log gives you the forensic timeline after the fact.

If you'd be open to sharing anonymized decision logs at some point, I'd love to use them for benchmarking the taint tracker when it's ready. That kind of real-world data is hard to come by.

[FeilixX]

Happy to share. I can export anonymized sequences — event_type + tool_name + relative timestamps, paths scrubbed — in JSONL. Should give you real read→write→execute chains to test against.

Let me put together a sample set and drop it in a gist — I'll ping you when it's ready.

[FeilixX]

Hey Manuel,

Dataset is ready — 42K anonymized events from 273 sessions (18 days of real Claude Code usage, PunkGo Jack v0.5.4).

What's in:

• event_type, tool_name, hook_event, timestamp, session_id (mapped to s001-s273)
• exit_code, energy, operation sequence
• file paths hashed with extension preserved (e.g., 701c24f9.rs) — so you can distinguish .env writes from .rs writes for taint tracking

What's removed:

• All prompts, code content, command bodies, AI responses
• User identifiers, absolute paths, emails, search queries

~6MB gzip / 32MB JSONL. Drop me an email at feijiu@punkgo.ai and I'll send it over.

Looking forward to seeing how the taint tracker handles real-world operation chains — the session-boundary clustering pattern should show up clearly in this dataset.

[manuelschipper]

Shipped in v0.5.1 — LLM veto gate for Write/Edit.