feat: Add interactive demo CLI for PTC with ReqLLM integration

[andreasronge]

Summary

• Add interactive CLI demo showcasing PtcRunner's key value proposition
• Demonstrates how LLMs can query large datasets (2500 records) efficiently by keeping data in BEAM memory
• Only small results pass through LLM context, reducing token costs by 10-100x vs traditional function calling

Features

• Interactive chat interface - Ask natural language questions about sample data
• Two generation modes:
  • structured (default) - Uses generate_object! for guaranteed valid JSON
  • text - Uses generate_text! with retry logic for debugging
• Sample datasets: 500 products, 1000 orders, 200 employees, 800 expenses
• Automated test runner - 11 test cases with constraint-based assertions
• Schema metadata - Simulates MCP tool schema discovery

Usage

cd demo
export OPENROUTER_API_KEY=sk-or-v1-...
mix deps.get
mix run -e "PtcDemo.CLI.main([])"

Test plan

• Manual testing with interactive CLI
• All 11 automated tests pass in structured mode
• Text mode works with retry logic
• Code formatted and compiles without warnings

🤖 Generated with Claude Code

[github-actions]

PR Review: feat: Add interactive demo CLI for PTC with ReqLLM integration

Summary

This PR adds a standalone demo application (demo/) that showcases PtcRunner's value proposition - enabling LLMs to query large datasets efficiently by keeping data in BEAM memory while only passing small programs and results through the LLM context. The implementation is solid and well-documented.

What's Good

• Excellent value demonstration: The README clearly explains the problem (large datasets consuming LLM context) and PtcRunner's solution (programs stay small, data stays local)
• Clean architecture: Proper separation between Agent (LLM interaction), CLI (user interface), SampleData (data generation), and TestRunner (automated testing)
• Two generation modes: Structured mode for reliability, text mode for debugging - good approach for a demo
• Comprehensive documentation: README covers installation, usage, example queries, commands, and architecture
• Thoughtful test design: Using constraint-based assertions (:between, :gt, etc.) handles random data elegantly
• Good use of existing library patterns: Uses PtcRunner.Schema.to_llm_schema() for structured generation

Issues (Must Fix)

1. Potential nil crash in structured mode - demo/lib/ptc_demo/agent.ex:217-218

   • Problem: When extracting user question, List.last() returns nil if the list is empty, then Map.get(nil, :content, "") will raise an error
   • Impact: Will crash if generate_program/3 is called with an empty context (unlikely in normal flow, but defensive coding is better for a demo)
   • Fix: Add nil check:

     user_question =
       context.messages
       |> Enum.filter(fn msg -> msg.role == :user end)
       |> List.last()
       |> case do
         nil -> ""
         msg -> msg |> Map.get(:content, "") |> extract_text_content()
       end

   • Complexity: Mechanical

2. Test runner regenerates data on each validation - demo/lib/ptc_demo/test_runner.ex:226-230

   • Problem: get_last_result/0 calls SampleData.products(), SampleData.orders(), etc. which generate fresh random data. But the Agent already has datasets loaded in state. This means test validation might use different data than what the program was executed against.
   • Impact: Tests could have false positives/negatives since the data at validation time differs from execution time
   • Fix: Store the result value in Agent state alongside last_program, or have get_last_result return the actual result from the Agent rather than re-running
   • Complexity: Moderate - requires API change to Agent

Suggestions (Optional)

1. Nice-to-have: Add --verbose flag to CLI - Currently the mode is determined by --text, but verbose output could help with debugging without switching modes

2. Nice-to-have: Consider seeding random data - For more reproducible demos, SampleData could accept an optional seed parameter for :rand.seed/2. Not critical since constraint-based tests handle randomness.

3. Nice-to-have: Error handling for LLM response - demo/lib/ptc_demo/agent.ex:133-136 uses pattern matching {:ok, response} = ReqLLM.generate_text(...) which will crash on LLM errors. A case with error handling would be more robust.

Security

No concerns. The demo:

• Runs LLM-generated programs through PtcRunner's existing sandbox
• Uses environment variables for API keys (not hardcoded)
• Sample data is procedurally generated, no real data exposure

Verdict

Request Changes - The data regeneration issue in the test runner (Issue #2) could lead to confusing test results. Issue #1 is lower risk but worth fixing for robustness. Once these are addressed, this is ready to merge.

The demo effectively showcases PtcRunner's core value proposition and will be useful for users evaluating the library.

[andreasronge]

@claude please fix these issues from PR review:

Issue 1: Add nil check for empty message list in structured mode

Location: demo/lib/ptc_demo/agent.ex:214-219
Problem: List.last() returns nil if the filtered messages list is empty, then Map.get(nil, :content, "") will raise an error. While unlikely in normal flow (user question is always added before generate_program), defensive coding is important for a demo.
Fix: Add a nil check using case statement:

user_question =
  context.messages
  |> Enum.filter(fn msg -> msg.role == :user end)
  |> List.last()
  |> case do
    nil -> ""
    msg -> msg |> Map.get(:content, "") |> extract_text_content()
  end

Please fix this issue in a single commit with minimal changes.

[andreasronge]

Auto-Triage Summary

Decisions Made

#	Issue	Decision	Action
1	Nil check for empty message list in structured mode	FIX_NOW	Included in fix request above
2	Test runner regenerates random data causing validation inconsistency	DEFER_ISSUE	Created issue #76
3	Add --verbose flag to CLI	IGNORE	Optional nice-to-have, not critical for MVP demo
4	Consider seeding random data	IGNORE	Optional nice-to-have, constraint-based tests already handle randomness
5	Error handling for LLM response	IGNORE	Optional nice-to-have, current pattern matching is acceptable for demo

Rationale

Issue 1 (FIX_NOW): Although unlikely in practice (user question is always added to context before generate_program), defensive nil checking is important for a demo showcasing the library. The fix is mechanical (simple case statement).

Issue 2 (DEFER_ISSUE): This is a real architectural issue - test validation uses freshly generated random data instead of the data the program was executed against. However, fixing it requires moderate changes (adding last_result to state, updating API). Given that constraint-based assertions provide some resilience, this is better addressed in a follow-up issue.

Issues 3-5 (IGNORE): These are optional enhancements marked as "nice-to-have" in the review. They don't affect correctness and can be considered for future iterations if needed.

Status

• FIX_NOW items: 1 (batched in single fix comment above)
• Issues created: 1 ([From PR #75] Test runner regenerates random data causing validation inconsistency #76)
• Items ignored: 3 (all optional enhancements)

[andreasronge]

Fix Summary

Issue Fixed: Added nil check for empty message list in structured mode

Location: demo/lib/ptc_demo/agent.ex:214-221

Changes:

• Replaced direct piping of List.last() result to Map.get() with a case statement
• This defensively handles the scenario where the filtered messages list is empty, returning an empty string instead of causing an error
• Minimal change with one commit: f6a201b (fix: Add nil check for empty message list in structured mode)

Verification:

• All precommit checks passed (formatting, compilation, credo linting, all 318 tests)
• Changes committed and pushed to feat/demo-chat-app branch

[github-actions]

PR Review: feat: Add interactive demo CLI for PTC with ReqLLM integration

Summary

This PR adds a well-structured interactive demo application demonstrating PtcRunner's value proposition with LLM integration. The demo shows how PTC keeps large datasets in BEAM memory while only passing compact programs to LLMs. The implementation is solid overall, with good separation of concerns.

What's Good

• Excellent demonstration of PTC value: The demo clearly shows the context efficiency advantage (2500+ records stay in memory, only ~200 byte programs pass to LLM)
• Two generation modes: Structured mode (reliable JSON schema) and text mode (with retry) provide good flexibility for different use cases
• Well-documented README: Clear explanation of the problem/solution, quick start, and architecture diagram
• Property-based test constraints: Tests use constraints like {:between, 1, 499} instead of exact values since data is random - this is a smart approach
• Clean module separation: Agent, CLI, SampleData, and TestRunner are well-separated

Issues (Must Fix)

1. Missing explicit Dotenvy dependency - demo/mix.exs:25-29
   • Problem: PtcDemo.CLI uses Dotenvy.source!/1 at line 13-14, but dotenvy is not listed as an explicit dependency in mix.exs - it only works because it's a transitive dependency of req_llm
   • Impact: If req_llm changes its dependencies, or if the code is used in isolation, it will fail at runtime with UndefinedFunctionError
   • Fix: Add {:dotenvy, "~> 1.1"} to deps in demo/mix.exs
   • Complexity: Mechanical (one-line addition)

Suggestions (Optional)

1. Test runner data regeneration issue - Already tracked in Issue [From PR #75] Test runner regenerates random data causing validation inconsistency #76
   • The test runner regenerates random data in get_last_result/0 (lines 226-231) which differs from the Agent's data
   • This is already captured as a GitHub issue, so no action needed in this PR

Security

No concerns. The demo:

• Uses environment variables for API keys (not hardcoded)
• .gitignore correctly excludes .env files
• PtcRunner's sandbox provides timeout/memory limits for program execution

Verdict

Request Changes - The missing explicit Dotenvy dependency should be added before merge. This is a one-line fix.

[andreasronge]

@claude please fix this issue from PR review:

Issue 1: Add explicit Dotenvy dependency to demo/mix.exs

Location: demo/mix.exs:25-29
Problem: The demo CLI uses Dotenvy.source!/1 at demo/lib/ptc_demo/cli.ex:13-14, but dotenvy is not listed as an explicit dependency in mix.exs. It currently works because it's a transitive dependency of req_llm, but Elixir best practice is to explicitly declare dependencies that you directly reference in your code. If req_llm changes its dependencies, this will break.
Fix: Add {:dotenvy, "~> 1.1"} to the deps list in demo/mix.exs:

defp deps do
  [
    {:ptc_runner, path: ".."},
    {:req_llm, "~> 1.0.0-rc"},
    {:dotenvy, "~> 1.1"}
  ]
end

Please fix this issue in a single commit with minimal changes.

[andreasronge]

Auto-Triage Summary (Cycle 2)

Decisions Made

#	Issue	Decision	Action
1	Missing explicit Dotenvy dependency in demo/mix.exs	FIX_NOW	Included in fix request above

Rationale

Issue 1 (FIX_NOW): The demo CLI directly calls Dotenvy.source\!/1 but doesn't explicitly list dotenvy as a dependency in mix.exs. While it works now as a transitive dependency of req_llm, Elixir best practice is to explicitly declare all dependencies you directly reference. This is:

• Mechanical: One-line addition to deps list
• In-scope: The PR adds the demo CLI that uses Dotenvy
• Best practice: Follows Elixir convention of explicit dependency declaration
• Future-proof: Prevents breakage if req_llm changes its dependencies

Status

• FIX_NOW items: 1 (batched in single fix comment above)
• Issues created: 0
• Items ignored: 0

Notes

All other items from the review were already addressed in the first triage cycle or marked as optional suggestions.

[andreasronge]

Summary

Issue Fixed: Added explicit Dotenvy dependency to demo/mix.exs

Changes Made:

• Added {:dotenvy, "~> 1.1"} to the deps list in demo/mix.exs:29

Verification:

• Confirmed that Dotenvy.source! is directly used in demo/lib/ptc_demo/cli.ex:13-14
• Ran mix precommit - all checks passed (formatting, compilation, credo, and 318 tests)
• Commit: a3b22d8

Rationale: While dotenvy was available as a transitive dependency of req_llm, Elixir best practice requires explicitly declaring all dependencies that are directly referenced in code. This ensures the application won't break if req_llm changes its dependencies in the future.

[github-actions]

PR Review: feat: Add interactive demo CLI for PTC with ReqLLM integration

Summary

This PR adds a standalone demo application under demo/ that showcases PtcRunner's programmatic tool calling capabilities with an interactive chat interface. The demo effectively illustrates the key value proposition: keeping large datasets in BEAM memory while only passing small results through LLM context.

What's Good

• Clear value demonstration: The README and architecture diagrams effectively communicate the PTC advantage over traditional function calling
• Dual generation modes: Supporting both structured (generate_object!) and text (with retry) modes shows flexibility and aids debugging
• Clean code structure: Well-organized GenServer-based agent with proper separation of concerns
• Comprehensive schema metadata: The SampleData.schemas/0 function simulates MCP tool discovery nicely
• Compiles cleanly with --warnings-as-errors and passes mix format --check-formatted

Issues (Must Fix)

None identified. The PR is well-implemented for its purpose as a demo application.

Suggestions (Optional)

1. Test runner data consistency (already tracked in GitHub issue [From PR #75] Test runner regenerates random data causing validation inconsistency #76)

   • Issue: get_last_result/0 in test_runner.ex:220-237 regenerates fresh random data via SampleData.products() etc., while the Agent uses data generated at init time
   • Impact: Test validation may use different data than the program executed against
   • Recommendation: Already has a GitHub issue tracking this - no action needed in this PR

2. Potential error case in Phase 3 - Nice-to-have

   • Location: agent.ex:133-134
   • Observation: ReqLLM.generate_text/3 could fail but uses pattern match {:ok, response} = which would crash
   • Suggestion: Could wrap in case for graceful degradation, but for a demo this is acceptable
   • Complexity: Low, but not critical for demo purposes

3. Consider :rand.seed/1 for reproducible demo data - Out of scope

   • For presentations or recordings, seeding random data would give consistent outputs
   • Recommendation: Create GitHub issue for future consideration if needed

Security

No concerns. The demo uses environment variables for API keys (standard practice), doesn't expose sensitive data, and runs PTC programs in PtcRunner's existing sandbox.

Verdict

Approve - This is a well-crafted demo that clearly illustrates PtcRunner's value proposition. The code is clean, compiles without warnings, and the one known issue (test data consistency) is already tracked in GitHub issue #76.

[andreasronge]

Auto-Triage Summary (Cycle 3 - Final)

Review Verdict

✅ APPROVED - All must-fix issues have been resolved across 3 review cycles.

Complete Triage History

Cycle 1

#	Issue	Decision	Action	Status
1	Nil check for empty message list	FIX_NOW	Fixed in f6a201b	✅ Complete
2	Test runner data regeneration	DEFER_ISSUE	Created #76	✅ Tracked
3-5	Various nice-to-haves	IGNORE	Optional enhancements	✅ N/A

Cycle 2

#	Issue	Decision	Action	Status
1	Missing Dotenvy dependency	FIX_NOW	Fixed in a3b22d8	✅ Complete

Cycle 3

#	Issue	Decision	Action	Status
-	No must-fix issues	-	Approved	✅ Complete

Final Status

• Total FIX_NOW items: 2 (both resolved)
• Issues created: 1 ([From PR #75] Test runner regenerates random data causing validation inconsistency #76 - test data consistency)
• Review verdict: APPROVE
• All checks: Passing (formatting, compilation, 318 tests)

Ready for Merge

PR is now approved with all critical issues resolved. Auto-merge will handle merging once all CI checks complete.