Epic: Unify Demo Test Runners

[andreasronge]

Unify Demo Test Runners

Overview

Refactor the demo test runners to eliminate code duplication, add mix json --test command with report generation, and support multi-turn tests for both DSLs.

Specification Documents

• Test Runner Refactoring Plan - Complete specification with architecture, task breakdown, and testing strategy

Progress

Phase 1: Shared Infrastructure (no breaking changes)

• Create TestRunner.Base with constraint/formatting functions #196 - Create TestRunner.Base with constraint/formatting functions (~150 lines)
• Create TestRunner.Report with markdown generation #198 - Create TestRunner.Report with markdown generation (~120 lines)
• Create TestRunner.TestCase with shared test definitions #200 - Create TestRunner.TestCase with shared test definitions (~100 lines)
• Create CLIBase with shared CLI utilities #202 - Create CLIBase with shared CLI utilities (~80 lines)
• Set up demo test infrastructure (test_helper, MockAgent) #205 - Set up demo test infrastructure (test_helper, mock_agent) (~100 lines)

Phase 2: Refactor Existing (maintain functionality)

• Refactor LispTestRunner to use shared modules #208 - Refactor LispTestRunner to use shared modules (~200 lines)
• Refactor LispCLI to use CLIBase #210 - Refactor LispCLI to use CLIBase (~100 lines)
• Add unit tests for TestRunner.Report and TestRunner.TestCase modules #212 - Add unit tests for shared modules (~200 lines)

Phase 3: JSON Parity (new functionality)

• Create JsonTestRunner using shared modules #214 - Create JsonTestRunner using shared modules (~250 lines)
• Create JsonCLI with test mode support #216 - Create JsonCLI with test mode support (~150 lines)
• Delete old TestRunner, update README references #218 - Delete old TestRunner, update README references (~20 lines)

Phase 4: Multi-turn Support (enhancement)

• Add memory support to JSON Agent #220 - Add memory support to JSON Agent (~100 lines)
• Add multi-turn test cases to JsonTestRunner (~50 lines) - Completed as part of Create JsonTestRunner using shared modules #214/Add memory support to JSON Agent #220
• Add integration tests using MockAgent for test runners #222 - Integration tests with MockAgent (~150 lines)

Phase 5: Documentation

• Update demo README with mix json CLI documentation #225 - Update demo README with new commands (~50 lines)

Phase 6: Validation

• Validate test runners with DeepSeek V3.2 model #230 - Validate test runners with DeepSeek V3.2 model (mix lisp --test and mix json --test)

Notes

• Phase 1 tasks can be done in parallel (no dependencies between them)
• Phase 2 must complete before Phase 3 (JSON runner depends on refactored Lisp patterns)
• Phase 4 (Multi-turn support) can be deferred if it proves problematic
• Phase 6 (Validation) ensures the refactored runners work with higher-end models as expected
• See specification for detailed task dependencies, risks, and definition of done

[andreasronge]

Progress Update

• Created: Create TestRunner.Base with constraint/formatting functions #196 - Create TestRunner.Base with constraint/formatting functions
• Status: Issue created with needs-review label, awaiting review approval
• Next: Once Create TestRunner.Base with constraint/formatting functions #196 passes review and gets ready-for-implementation, trigger implementation

Phase 1 has 5 tasks that can be done in parallel. Starting with the first task (TestRunner.Base) which extracts core constraint validation and formatting functions.

[andreasronge]

Progress Update

• Completed: Create TestRunner.Base with constraint/formatting functions #196 - Create TestRunner.Base with constraint/formatting functions
• Next: Creating issue for TestRunner.Report with markdown generation

[andreasronge]

Progress Update

• Marked complete: Create TestRunner.Report with markdown generation #198 - Create TestRunner.Report with markdown generation (was closed but unchecked)
• Created: Create TestRunner.TestCase with shared test definitions #200 - Create TestRunner.TestCase with shared test definitions
• Next: Issue Create TestRunner.TestCase with shared test definitions #200 is now pending review. Once approved, it will be ready for implementation.

Phase 1 progress: 2/5 tasks complete (Base, Report). TestCase issue created, CLIBase and test infrastructure still need issues.