Repository navigation
docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute deliverables dir - #4655
Closed
qiaochuz-nv wants to merge 1 commit into
Closed
qiaochuz-nv wants to merge 1 commit into
qiaochuz-nv wants to merge 1 commit into
GitHub Actions / Harness conformance P0
failed
Oct 15, 2026 in 0s
0 passed, 1 failed, 3 unavailable
| Harness | P0 result | Failed scenario checks | First failure | Evidence |
|---|---|---|---|---|
| opencode | unavailable | 7 / 7 | tool_success: required evidence does not match its schema; execution also failed in model_error: required input is unavailable; expected a nonempty collection of saved model_calls | Download report and logs |
| pi | unavailable | 3 / 7 | retry_429: producer explicitly reports unavailable evidence; execution also failed in model_error: required input is unavailable; expected a nonempty collection of saved model_calls | Download report and logs |
| codex | unavailable | 7 / 7 | tool_success: expected a nonempty collection of saved turns; execution also failed in retry_429: required input is unavailable; expected a nonempty collection of saved model_calls | Download report and logs |
| hermes | failed | 3 / 7 | retry_429: each saved call must have exactly one invocation owner | Download report and logs |
Counts include execution errors and count each scenario once. Unavailable means setup, tests, execution, or evidence collection failed.
Each artifact contains probes/conformance_report.md, setup.log, tests.log, and per-scenario probes/<harness>/<scenario>/scenario_result.json and episode.log.
Rerun locally
From the tested Gym revision, activate your Gym environment and install the selected adapters' requirements and pinned runtimes. No model service or model credentials are needed.
python scripts/run_harness_conformance.py --harness opencode --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness pi --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness codex --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness hermes --output "$(mktemp -d)/probes"Each command runs the full suite for that harness and creates a fresh output directory. Open conformance_report.md in that directory for the full report.
Tested commit: 76937de3e8f544d80cab6a6dca647da79fdd71b6.
Probe summaries and downloadable evidence. Local setup instructions.
Loading