Bug: model hallucination on exploration test #20 — bypasses tools entirely

[andreasronge]

Problem

Test #20 (decoy: certification reimbursement) triggers hallucination where the model fabricates a policy description and document ID without ever calling the search or fetch tools.

Observed behavior

Turn 1: No valid code produced (parse failure)
Turn 2: Model returns fabricated text:
  "The policy for professional certification reimbursement, as detailed 
   in document pol-883, states that employees are eligible for full 
   reimbursement of certification fees..."

The document ID "pol-883" does not exist. The model never called tool/search or tool/fetch. It hallucinated the entire answer.

Expected behavior

The model should search for certification/reimbursement documents, fetch candidates, compare content, and return "DOC-021" (Certification Reimbursement document).

Root cause analysis

The system prompt says "You have NO direct data access" (coordinator mode) or shows data inventory (direct mode), but the model sometimes ignores tools and generates plausible-sounding text answers. This is worse in auto-return mode because the fabricated text has no println → auto-returns as the answer.

In explicit multi-turn mode, the model would need to wrap the fabrication in (return ...), which provides a slight friction barrier. But the core issue is model behavior, not prompt mechanics.

Reproduction

cd demo && source .env
mix run -e 'PtcDemo.LispTestRunner.run_one(20, model: "openrouter:google/gemini-3.1-flash-lite-preview", prompt: :auto_return, verbose: true, debug: true)'

Occurs intermittently — approximately 1 in 3 runs.

Potential mitigations

1. Prompt addition: "You MUST use tools to access data. Never fabricate or guess answers."
2. Validation: Detect when no tool calls were made but a non-trivial answer was returned
3. Signature constraint: Force DOC-XXX format pattern in the expected output
4. Model capability: May only affect the lite model — test on stronger models (Experiment: cross-model validation of auto-return #834)

Affects

• Test test: add error tests for operations without piped input (#19) #20 in demo/lib/ptc_demo/test_runner/test_case.ex
• Both auto_return and multi_turn modes (untested on multi_turn for this specific test)

[andreasronge]

Investigation: REPL-style conversation framing

Reproduction

Reproduced the hallucination across 2 runs with gemini-3.1-flash-lite-preview:

Run 1 (2 turns): Model searched correctly, got DOC-020 in results, but fabricated ID policy_cert_reimbursement_123 for fetch (returned nil). Ignored nil, returned made-up title. Turn 2 returned the title instead of the document ID.

Run 2 (1 turn, full hallucination): Searched correctly, got DOC-020 in results, but fabricated ID policy_882 for fetch (returned nil). Then returned (return "5000") — a completely fabricated reimbursement amount.

Root cause

The model pre-writes the entire program speculatively — it hardcodes fetch IDs as literals before seeing search results, even though PTC-Lisp binds tool results immediately and they could be used in the same program via def. The auto-return prompt already says "inspect with println first, answer next turn" but the model ignores this instruction.

Proposed mitigation: REPL-style conversation framing

LLMs have massive training data on REPL sessions. In a REPL, the natural behavior is: type a short expression → see the result → type the next expression based on what you saw. No one hardcodes values in a REPL because you can just look at the output. This deeply internalized behavioral pattern may be more effective than explicit instructions the model ignores.

Level 1 — Prompt + feedback formatting (minimal code changes)

• New prompt addon (lisp-addon-repl.md) that frames the session as a Clojure REPL
• Tweak feedback formatting in TurnFeedback.format/3 to look like REPL output (e.g., => prefixes)
• Execution engine stays unchanged — just the conversation looks like a REPL session
• Add as a new :repl prompt option alongside :auto_return and :multi_turn

Level 2 — Per-expression result display (bigger change)

• Modify the interpreter to emit per-expression results (currently only prints and memory are captured)
• Feedback replays each expression with its result, like a real REPL transcript
• More convincing framing but requires changes to the Lisp execution layer

Level 1 is the practical starting point — it's enough to shift model behavior without touching the interpreter. Level 2 can follow if Level 1 shows promise.

[andreasronge]

REPL framing experiment — results

Setup

Built a proof-of-concept that executes LLM-generated code in a real long-lived clj REPL process (via Elixir Port/GenServer). Key components:

• demo/priv/clj_prelude.clj — Clojure prelude with tool/search, tool/fetch, return, fail as real functions. Documents use idiomatic keyword keys.
• demo/lib/ptc_demo/clj_repl.ex — GenServer wrapping clj -M via stdin/stdout with sentinel protocol for output delimiting.
• demo/lib/ptc_demo/clj_repl_experiment.ex — Drives a real LLM through the REPL with a simple system prompt.

The system prompt just says: "You are working in a Clojure REPL. Work interactively. Each turn, write ONE short expression. You'll see the REPL output, then decide your next step."

Conversation shape

The conversation naturally looks like a REPL session:

SYSTEM: You are working in a Clojure REPL...

USER: The REPL is ready. Here's your task: Find the policy document about...

ASSISTANT: (tool/search {:query "reimbursement certification"})

USER: user=> {:results [{:id "DOC-021", :title "Certification Reimbursement", ...}], :total 1}

ASSISTANT: (tool/fetch {:id "DOC-021"})

USER: user=> {:id "DOC-021", ..., :content "Reimbursement policy for professional certifications."}

ASSISTANT: (return "DOC-021")

USER: user=> ::RETURN:: "DOC-021"

Results (gemini-3.1-flash-lite-preview, 5 runs each)

Test	PTC-Lisp (auto_return)	REPL framing
#20 (hallucination — this issue)	~33% pass	100% pass (5/5)
#21 (security+compliance intersection)	—	100% pass (5/5)
#22 (sabbatical leave)	—	100% pass (5/5)
#23 (premature answer — #837)	~33% pass	60% pass (3/5)

Why it works

1. One expression per turn — the REPL framing naturally prevents speculative multi-step scripts. The model writes (tool/search ...), sees the result, then decides what to do next.
2. No fabricated IDs — in a REPL, you use values you've seen. The model never hardcoded IDs like policy_882 or policy_cert_reimbursement_123 (the original bug).
3. Deeply internalized behavior — LLMs have massive REPL training data. "Type expression, see output, type next" is a behavioral pattern the model follows without explicit instruction. This is more effective than prompt instructions like "verify before answering" which the model ignores.

Remaining #23 failures

The 2/5 failures on test #23 are a different class of bug: the model fetches both documents, sees the full data (including :topics ["ergonomics"]), but only checks the :content field for "ergonomics" (which isn't there — it's in :topics). This is a reasoning limitation, not the original premature-answer/hallucination problem.

Key takeaway

The REPL framing completely eliminated the hallucination problem from this issue. The model never fabricated data, never bypassed tools, and always used values from actual REPL output. This suggests the conversation structure (interactive REPL vs "write a program") matters more than prompt instructions for preventing this class of failure.

[andreasronge]

Update: var previews fix remaining #23 failures — 20/20 perfect score

Problem with do blocks

When the model batches tool calls in a do block, the REPL only shows the last def binding:

user=> (do (def doc1 (tool/fetch {:id "DOC-001"}))
           (def doc2 (tool/fetch {:id "DOC-002"})))
#'user/doc2     ;; doc1 and doc2 content is invisible!

This caused #23 failures: the model fetched both docs but couldn't see their content, then wasted turns debugging or guessed wrong.

Solution: automatic var previews

Added a __preview-vars function to the Clojure prelude that runs after each eval and prints truncated previews of newly bound/changed vars:

user=> (do (def doc1 (tool/fetch {:id "DOC-001"}))
           (def doc2 (tool/fetch {:id "DOC-002"})))
#'user/doc2
;; doc1 = {:id "DOC-001", :title "Remote Work Guidelines", :topics ["remote work" "home office" ...}, 
;;  doc2 = {:id "DOC-002", :title "Home Office Setup Requirements", :topics [..."ergonomics"...], ...

Key properties:

• Only shows changed/new vars (unchanged vars from previous turns are suppressed)
• Truncated to 120 chars per var (avoids flooding)
• Excludes builtins (return, fail, etc.)
• Consistent with PTC-Lisp's "Stored" hint but with actual value previews
• No prompt instructions needed — the model just sees the data

Also fixed: keyword keys in prelude

Switched document maps from string keys ("content") to keyword keys (:content). This is idiomatic Clojure and eliminates NullPointerExceptions from (:content doc) on string-keyed maps that were causing #23 to waste turns debugging.

Results: 5 runs × 4 tests = 20/20 pass

--- Run 1 --- 4/4 passed
--- Run 2 --- 4/4 passed
--- Run 3 --- 4/4 passed
--- Run 4 --- 4/4 passed
--- Run 5 --- 4/4 passed

Test	PTC-Lisp (auto_return)	REPL (no preview)	REPL (with preview)
#20 (hallucination)	~33% pass	100%	100%
#21 (intersection)	—	0%	100%
#22 (sabbatical)	—	100%	100%
#23 (premature answer)	~33% pass	60%	100%

Summary of all changes that contributed

1. REPL framing — conversation structured as interactive REPL session, not "write a program"
2. Keyword keys — idiomatic Clojure maps, eliminates string/keyword mismatch errors
3. Var previews — model always sees what it bound, even in do blocks
4. (return value) / (fail reason) — typed completion signaling as real Clojure functions
5. DOC-042 added — "Security Compliance Audit Process" in IT dept with both security+compliance topics (fixed test [Phase 2] Add comparison operations (neq, gt, gte, lt, lte) #21 data gap)

[andreasronge]

Update: PTC-Lisp REPL prompt + result previews + test data fix

Test data bug found

DOC-020 in the Elixir sample data IS "Certification Reimbursement" — the test expected DOC-021 which is "Code of Conduct" (wrong). The model was returning the correct answer all along with the REPL prompt. Fixed the test constraint to {:eq, "DOC-020"}.

Test #20 redesigned with proper decoy

The original test was too easy — any search immediately returned the correct doc as the only result. Redesigned DOC-018 ("Training and Certification Budget") as a decoy:

• Both DOC-018 and DOC-020 match searches for "reimbursement certification"
• DOC-018 content: "Covers tuition reimbursement for degree programs and training courses only. For professional certification exam fees, see the dedicated certification policy."
• DOC-020 content: "Reimbursement policy for professional certification exam fees..."
• Model must fetch and compare content to pick the right one

Result preview added to TurnFeedback

When there's no println output and the result isn't a def binding, PTC-Lisp now shows a truncated preview of the expression result:

=> %{"cursor" => nil, "has_more" => false, "results" => [%{"department" => "HR", "id" => "DOC-018", ...
... (truncated, use println to see full value)

This matches the clj REPL behavior where bare expressions show their result. The truncation hint tells the model to use println for full output.

Var preview truncation hints

When def binding previews are truncated, a hint is now appended:

;; results = %{"cursor" => nil, "has_more" => false, "results" => [%{"department" => "HR", ...
;; (truncated, use println to see full value)

Key finding: prompt weight matters

The clj REPL system prompt is ~750 chars (plain text, no XML, no code examples). The PTC-Lisp :repl prompt is ~1800 chars with XML tags, multiple BAD/GOOD code blocks, and <state> sections. Even without lisp-base, the PTC-Lisp prompt is twice as heavy.

With truncated feedback (250 chars, same as PTC-Lisp), the clj REPL still passes 3/3 on test #20 with the decoy. The model sees the truncation, uses println, then proceeds correctly. The difference is the simpler prompt — not the feedback mechanism.

clj REPL test with truncation + decoy

Turn 1: (tool/search ...) → truncated, sees DOC-018 but DOC-021 cut off
Turn 2: (println (tool/search ...)) → sees both DOC-018 and DOC-021 in full
Turn 3: (tool/fetch {:id "DOC-021"}) → verifies content
Turn 4: (return "DOC-021") → correct answer

3/3 PASS even with truncated feedback.

Next step

Strip the PTC-Lisp :repl prompt down to match the clj REPL's simplicity (~750 chars, plain text, no XML/code blocks) and re-test. The hypothesis is that prompt weight, not feedback format, is the remaining gap.

[andreasronge]

Remaining differences between clj REPL and PTC-Lisp feedback

Beyond the prompt weight difference, there are structural differences in what the model sees each turn:

1. Data format (EDN vs Elixir inspect)

• clj: {:id "DOC-021", :title "Certification Reimbursement"}
• PTC-Lisp: %{"id" => "DOC-021", "title" => "Certification Reimbursement"}

The Elixir %{... => ...} format is foreign to the model — it's neither Clojure nor JSON.

2. Context repeated every turn

• clj: feedback is just the REPL output (a few lines)
• PTC-Lisp: the full data inventory + tool schemas (~2000 chars) is repeated as the first user message on every turn, then feedback comes as a separate user message

The model has to dig through 2000 chars of repeated context to find the actual result each turn.

3. Turn metadata

• clj: nothing extra
• PTC-Lisp: Turn 2 of 6 (5 remaining) appended to feedback

4. Result visibility and ordering

• clj: bare expression result is always the first thing the model sees (the REPL output)
• PTC-Lisp: result preview or var preview comes after println output, formatted with ;; var = ... prefix

5. Conversation shape

clj REPL:
  user: "Here's your task..."
  assistant: (tool/search {:query "..."})
  user: {:results [{:id "DOC-018" ...} {:id "DOC-021" ...}] :total 2}
  assistant: (tool/fetch {:id "DOC-021"})
  user: {:id "DOC-021" :content "Reimbursement policy for..."}

PTC-Lisp:
  user: [2000 chars: tool schemas + data inventory + mission]
  assistant: ```clojure\n(def results (tool/search {:query "..."}))\n```
  user: [2000 chars: same context repeated]\n;; results = %{"cursor" => nil, ...}\nTurn 2 of 6 (5 remaining)

Next steps to close the gap

1. Strip prompt — match the clj REPL's ~750 char plain text prompt
2. Stop repeating context — data inventory only on first turn (or move to system prompt)
3. Format as EDN/Clojure — output values in Clojure format, not Elixir inspect
4. Simplify feedback — result first, no extra metadata unless needed

[andreasronge]

Addendum: feedback structure comparison

clj REPL feedback (what the LLM sees as the next user message)

The expression result and println output are interleaved naturally — the model sees exactly what a human would see in a REPL:

user=> {:results [{:id "DOC-018", :title "Training and Certification Budget", ...} {:id "DOC-021", :title "Certification Reimbursement", ...
... (truncated, use println to see full value)

PTC-Lisp feedback (what the LLM sees as the next user message)

The feedback is layered — println output comes first, then var previews (with different prefix), then turn metadata. The result format uses Elixir inspect syntax:

;; search-results = %{"cursor" => nil, "has_more" => false, "results" => [%{"department" => "HR", "id" => "DOC-018", ...
;; (truncated, use println to see full value)

Turn 2 of 6 (5 remaining)

The key difference: in the clj REPL the feedback looks like REPL output (the model recognizes this format from training data). In PTC-Lisp it looks like structured metadata with a custom format the model hasn't seen before.

[andreasronge]

Update: simplified REPL prompt — closing the gap

Prompt stripped to 650 chars

Removed all XML tags, code block examples, and <state> section from the REPL addon. Now plain text, ~650 chars (down from 1800):

You are in an interactive REPL. Tools and data are described below.
Call (return value) when you have the final answer.

Work interactively. Each turn, write ONE short expression.
You'll see the output, then decide your next step.

Rules:
- Do NOT guess or fabricate data. Only use values you've seen in output.
- If output is truncated, use println to see the full value.
- Explore incrementally: search, inspect results, fetch details, then return.
- Keep expressions short — this is a REPL, not a script.

Use def to store values across turns. Defined vars persist.

Results with harder decoy test (gemini-flash-lite, 5 runs)

Backend	#20 (decoy)	#23 (ergonomics)
clj REPL	100% (3/3)	100%
PTC-Lisp :repl (650 chars)	~40% (2-3/5)	~20%
PTC-Lisp :auto_return	~20% (1/5)	~20%

Remaining structural gap

The prompt is now comparable but PTC-Lisp still has structural differences:

1. 2000-char context repeated every turn — data inventory + tool schemas appear as the first user message in every turn's conversation history. The clj REPL has only the feedback per turn.
2. Elixir format in context — the data inventory uses Elixir inspect (%{"id" => "DOC-021"}) while tool results now use Clojure EDN. Mixed formats in the same conversation.
3. clj REPL feedback is just the value — user=> {:results [...]}. PTC-Lisp has var previews, truncation hints, and turn metadata as separate sections.

The clj REPL works because each turn's feedback is minimal and in a format the model deeply understands from training. Addressing item #1 (stop repeating context) would likely have the biggest impact.

[andreasronge]

Current status: model finds answer but doesn't call (return ...)

What works

The PTC-Lisp :repl prompt with all improvements (EDN format, result previews, context in system prompt, simplified prompt) now gets the model to correctly explore and find the right document:

Turn 1: (tool/search {:query "reimbursement certification"})
        → sees DOC-018 and DOC-020 (truncated)
Turn 2: (tool/fetch {:id "DOC-018"})
        → reads "For professional certification exam fees, see the dedicated certification policy"
Turn 3: (tool/search {:query "certification policy"})
        → re-searches  
Turn 4: (tool/fetch {:id "DOC-020"})
        → reads "Reimbursement policy for professional certification exam fees..."
Turn 5: "Employees are eligible for full reimbursement..."  ← WRONG: bare text summary, not (return "DOC-020")

The model finds DOC-020 and reads its content, but instead of (return "DOC-020") it writes a bare text paraphrase as the final expression. Without completion_mode: :auto, the loop needs an explicit (return ...) to terminate correctly.

Adding completion_mode: :auto doesn't help because auto-return treats every bare expression (including exploration turns) as a final answer, causing premature returns.

Why the clj REPL works

In the clj REPL experiment, the model reliably calls (return "DOC-021"). The key difference: the clj REPL conversation has zero competing formats. Every message is either a Clojure expression or a Clojure value. The PTC-Lisp conversation still mixes:

• System prompt with Elixir-style data inventory (%{"id" => "DOC-021"})
• Feedback with Clojure-style results ({"id" "DOC-021"})
• Turn metadata (Turn 3 of 6 (4 remaining))

This format mixing may confuse the model about what kind of session it's in, making it less likely to use Clojure idioms like (return ...).

Next step

Make the PTC-Lisp REPL conversation look as close as possible to the clj REPL:

1. Format data inventory in Clojure style (currently Elixir inspect)
2. Remove or minimize turn metadata for REPL mode
3. Ensure the entire conversation uses one consistent format

[andreasronge]

Final results: PTC-Lisp REPL mode matches clj REPL on #20

Changes that closed the gap

Removed all non-REPL formatting from the system prompt:

• Removed <output_format> XML block — the REPL addon already says "write ONE short expression in a ```clojure block"
• Removed role prefix ("You are a data analyst...") — not needed for REPL framing
• Added output_format: "" override in demo agent for :repl mode

The system prompt is now 1197 chars total (REPL instructions + tools + data), with zero XML tags.

Final results (gemini-flash-lite, 5 runs, harder decoy test)

Test	Baseline (:auto_return)	PTC-Lisp :repl	clj REPL
#20 (decoy)	~20%	100% (5/5)	100%
#23 (ergonomics)	~20%	80% (4/5)	100%

Key insight

Every piece of non-REPL formatting degraded performance:

• <output_format>...</output_format> XML tags
• <role>...</role> XML tags
• Role prefix ("You are a data analyst")
• Turn metadata ("Turn 3 of 6 (4 remaining)")
• BAD/GOOD code block examples
• <multi_turn_rules> XML wrappers
• <state> XML sections

The model performs best when the entire conversation looks like a clean REPL session — plain text instructions, Clojure EDN output, no XML, no competing formats. The conversation structure matters more than the instruction content.

All changes made (summary)

1. Clojure EDN format in feedback (Format.to_clojure instead of Elixir inspect)
2. user=> result prefix for bare expressions
3. Var previews with truncated values and truncation hints
4. Simplified REPL prompt (650 chars plain text)
5. Context moved to system prompt (context_in_system option)
6. Minimal turn info (only warns when ≤2 turns remain)
7. Stronger return instruction with example
8. Removed output_format and role prefix for REPL mode
9. Test data: DOC-018 decoy, DOC-020 constraint fix, DOC-042 for intersection

[andreasronge]

Closing — hallucination bug (test #20) is solved (100% pass rate with REPL framing). Test #23 remaining failures are a model reasoning limitation shared by both clj REPL (90%) and PTC-Lisp REPL (80%) backends at similar rates (not statistically significant at n=10).