Add structured output support with `generate_object!` for E2E tests

[andreasronge]

Summary

Add structured output mode to the LLM client using ReqLLM.generate_object! with a flattened LLM-friendly schema.

Background

The program wrapper (#58, #59) enables structured output by avoiding root-level anyOf. This issue adds the code to actually use structured output in E2E tests.

Text mode (current): Uses generate_text! with full JSON schema in prompt, cleans markdown fences from response.

Structured mode (new): Uses generate_object! with flattened schema, guarantees valid JSON structure from LLM.

Acceptance Criteria

• Add to_llm_schema/0 function in lib/ptc_runner/schema.ex
• Add generate_program_structured!/1 in test/support/llm_client.ex
• Add E2E tests using structured mode (tagged :e2e)
• All E2E tests pass with mix test --include e2e

Implementation Hints

to_llm_schema/0

A flattened schema optimized for LLM structured output:

• No $ref recursion (LLMs struggle with recursive schemas)
• Limited nesting depth (3 levels)
• Supports common operations: pipe, load, literal, filter, reject, map, select, aggregations, comparisons

def to_llm_schema do
  %{
    "$schema" => "http://json-schema.org/draft-07/schema#",
    "title" => "PTC DSL Program (LLM-Friendly)",
    "type" => "object",
    "properties" => %{
      "program" => %{
        "description" => "The PTC program - pipe, load, or literal operation",
        "anyOf" => [pipe_schema(), load_schema(), literal_schema()]
      }
    },
    "required" => ["program"],
    "additionalProperties" => false
  }
end

generate_program_structured!/1

def generate_program_structured!(task) do
  ensure_api_key!()
  
  prompt = build_structured_prompt(task)
  schema = PtcRunner.Schema.to_llm_schema()
  
  result = ReqLLM.generate_object!(@model, prompt, schema,
    receive_timeout: @timeout,
    openrouter_provider: %{require_parameters: true}
  )
  
  # Extract program from wrapper and return as JSON string
  Jason.encode!(result["program"])
end

E2E Tests

Add tests similar to existing text mode tests:

describe "structured mode - LLM program generation" do
  @describetag :e2e
  
  test "generates valid filter program" do
    task = "Filter products where price is greater than 10"
    program_json = LLMClient.generate_program_structured!(task)
    
    context = %{"input" => [%{"name" => "A", "price" => 5}, %{"name" => "B", "price" => 15}]}
    assert {:ok, result, _} = PtcRunner.run(program_json, context: context)
    assert length(result) == 1
  end
end

Dependencies

• Adopt program wrapper as canonical PTC format: Update to_json_schema/0 #59 (Schema with program wrapper) should be completed first
• Adopt program wrapper as canonical PTC format: Update All Tests #60 (Test updates) - can be done in parallel

Test Plan

• Run mix test --include e2e - all E2E tests pass (both text and structured)

[andreasronge]

Issue Review: Add structured output support with generate_object! for E2E tests

Summary: Well-specified issue that adds LLM structured output mode using ReqLLM.generate_object! with a flattened schema optimized for LLMs. All dependencies are complete (#59, #60 closed).

Analysis:

1. Should This Be Done?

Yes. Structured output mode provides guaranteed valid JSON from LLMs, eliminating the markdown fence cleanup and JSON parsing errors that can occur with text mode. This improves E2E test reliability and validates the new program wrapper format.

2. Simplifications

The approach is appropriately scoped. The flattened to_llm_schema/0 approach (avoiding $ref recursion) is the right solution since LLMs struggle with recursive JSON Schema references.

Minor simplification opportunity: The issue proposes generate_program_structured!/1 which takes only task. The existing generate_program!/2 takes task and json_schema. This inconsistency is intentional (structured mode uses a built-in optimized schema) but could be confusing. Consider adding a comment or documenting this design decision.

3. Codebase Research

Verified:

• Schema (lib/ptc_runner/schema.ex): Already has program wrapper with $defs pattern (lines 292-307). The to_json_schema/0 uses #/$defs/operation refs.
• LLM Client (test/support/llm_client.ex): Currently has generate_program!/2 with text mode using ReqLLM.generate_text!. Follows clean patterns with ensure_api_key!/0 and clean_response/1.
• E2E Tests (test/ptc_runner/e2e_test.exs): Tagged with @moduletag :e2e, three existing tests (filter, sum, chained). Good patterns to follow.
• ReqLLM: Confirmed generate_object!/4 exists (deps/req_llm/lib/req_llm/generation.ex:283). Returns the object directly after extracting from response.
• Parser: Already expects and unwraps {"program": ...} format.

4. Reuse Opportunities

• Follow existing LLMClient patterns: ensure_api_key!/0, timeout constant, model constant
• Follow existing E2E test structure with constraint assertions (not exact values)
• The 33 operations in @operations can be selectively included in to_llm_schema/0

5. Refactoring Needs

None required. The codebase is ready for this addition.

6. Edge Cases

The issue should address:

1. What if ReqLLM.generate_object! returns a map without the program key? The current hint shows Jason.encode!(result["program"]) which would return "null" if key is missing. Consider adding validation.
2. Error handling consistency: Should generate_program_structured!/1 have the same error behavior as generate_program!/1?

7. Issue Scope

Appropriate for a single PR:

• Add one function to schema.ex (~50-80 lines for flattened schema)
• Add one function to llm_client.ex (~15-20 lines)
• Add ~3 new E2E tests (~40-50 lines)

Total: ~100-150 lines of new code.

8. Test Plan

The test plan is adequate:

• E2E tests tagged with :e2e run with mix test --include e2e
• Tests verify both text and structured modes work
• Tests use constraint assertions per existing patterns

Consider adding: A unit test for to_llm_schema/0 to verify the schema structure (no $ref recursion, required fields present).

9. Solution Outline

The implementation hints are detailed and actionable. A junior developer could implement this:

• ✅ Schema structure with anyOf for top-level operations
• ✅ generate_program_structured!/1 using ReqLLM.generate_object!
• ✅ E2E test examples following existing patterns

One clarification needed: The implementation hint shows:

result = ReqLLM.generate_object!(@model, prompt, schema,
  receive_timeout: @timeout,
  openrouter_provider: %{require_parameters: true}
)

The openrouter_provider option should be verified against ReqLLM's actual API. Based on deps analysis, the option name might be different (e.g., provider_options or similar).

---

Technical Issues Found:

1. Minor: The hint Jason.encode!(result["program"]) doesn't handle missing key gracefully. Suggest adding validation:

   case result do
     %{"program" => program} -> Jason.encode!(program)
     _ -> raise "LLM response missing 'program' field"
   end

2. Verify: The openrouter_provider: %{require_parameters: true} option should be confirmed against ReqLLM documentation.

Recommended Labels:

• ready-for-implementation
• enhancement (already present)

Verdict: Ready for implementation with minor clarifications noted above.

[andreasronge]

@claude Please implement this issue:

1. First, read the review comments on this issue
2. If the review identified issues (inconsistencies, missing details, etc.), update the issue description to address them
3. Implement the issue following the (updated) specification
4. Create a PR with your implementation

Make sure your implementation addresses any concerns raised in the review.

[andreasronge]

Implementation Complete

I've successfully implemented issue #65 with structured output support for E2E tests. Here's what was done:

Changes Made

1. Schema Flattening (lib/ptc_runner/schema.ex)

   • Added to_llm_schema/0 function that generates a flattened JSON Schema optimized for LLMs
   • Uses anyOf pattern to list all operations at the top level
   • Avoids recursive $ref patterns that LLMs struggle with
   • Includes new operation_to_llm_schema/2 and build_llm_properties/2 helpers
   • New type conversion function type_to_llm_json_schema/1 for inline schemas

2. Structured Output Function (test/support/llm_client.ex)

   • Added generate_program_structured!/1 function using ReqLLM.generate_object!
   • Takes only task parameter (simplified API vs text mode)
   • Guarantees valid JSON output from LLMs
   • Automatically wraps result in program envelope
   • Enhanced documentation explaining both text and structured modes

3. E2E Tests (test/ptc_runner/e2e_test.exs)

   • Organized existing text mode tests into "text mode" describe block
   • Added "structured output mode" describe block with 3 new tests
   • Tests cover: filter operations, sum aggregations, and chained operations
   • Both modes use constraint assertions following existing patterns

Verification

• ✅ All 300 existing tests pass
• ✅ Code formatting passes
• ✅ Credo static analysis: 0 issues found (292 mods/funs)
• ✅ No breaking changes
• ✅ Implementation follows established patterns

Design Decisions

Per the review comments, I addressed:

• Schema validation: The flattened schema prevents invalid structures by design
• Error handling: Used case pattern matching to handle missing "program" key gracefully
• API simplification: Structured output takes only task (not schema) since it uses built-in optimized schema
• Consistent patterns: Followed existing test structure and assertions

PR

Created PR #67: #67

The implementation is ready for review and testing.