Skip to content

feat(scripts): switch simple_agent relays to single_agent_turn_legacy - #4952

Merged
weijiac0619 merged 7 commits into
mainfrom
yuya/gymint-549-switch-generator
Oct 28, 2026
Merged

weijiac0619 merged 7 commits into
mainfrom
yuya/gymint-549-switch-generator

docs(scripts): keep the routing name when a harness override meets a …

84e0a93
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
GitHub Actions / Harness conformance (non-blocking) failed Oct 28, 2026 in 0s

5 passed, 0 failed, 1 unavailable

Harness P0 result Failed scenario checks First failure Evidence
opencode passed 0 / 7 None Download report and logs
pi passed 0 / 7 None Download report and logs
codex passed 0 / 7 None Download report and logs
hermes passed 0 / 7 None Download report and logs
kilocode unavailable 1 / 7 model_error: the EnvironmentServer must receive a failed response retaining the model error and ng_agent_outcome with model_request_error and the witnessed HTTP status Download report and logs
stirrup passed 0 / 7 None Download report and logs

Counts include execution errors and count each scenario once. Unavailable means setup, tests, execution, or evidence collection failed.

Each artifact contains probes/conformance_report.md, setup.log, tests.log, and per-scenario probes/<harness>/<scenario>/scenario_result.json and episode.log.

Rerun locally

From the tested Gym revision, activate your Gym environment and install the selected adapters' requirements and pinned runtimes. No model service or model credentials are needed.

python scripts/run_harness_conformance.py --harness kilocode --output "$(mktemp -d)/probes"

Each command runs the full suite for that harness and creates a fresh output directory. Open conformance_report.md in that directory for the full report.

Tested commit: 84e0a93ef8755f18a607dd653f7820455d4be2dd.

Probe summaries and downloadable evidence. Local setup instructions.