Skip to content

docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute deliverables dir - #4655

Closed
qiaochuz-nv wants to merge 1 commit into
mainfrom
qiaochuz/nvb-6649458-gdpval-no-metrics-exit0
Closed

qiaochuz-nv wants to merge 1 commit into
mainfrom
qiaochuz/nvb-6649458-gdpval-no-metrics-exit0

docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute…

76937de
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
GitHub Actions / Harness conformance P0 failed Oct 15, 2026 in 0s

0 passed, 1 failed, 3 unavailable

Harness P0 result Failed scenario checks First failure Evidence
opencode unavailable 7 / 7 tool_success: required evidence does not match its schema; execution also failed in model_error: required input is unavailable; expected a nonempty collection of saved model_calls Download report and logs
pi unavailable 3 / 7 retry_429: producer explicitly reports unavailable evidence; execution also failed in model_error: required input is unavailable; expected a nonempty collection of saved model_calls Download report and logs
codex unavailable 7 / 7 tool_success: expected a nonempty collection of saved turns; execution also failed in retry_429: required input is unavailable; expected a nonempty collection of saved model_calls Download report and logs
hermes failed 3 / 7 retry_429: each saved call must have exactly one invocation owner Download report and logs

Counts include execution errors and count each scenario once. Unavailable means setup, tests, execution, or evidence collection failed.

Each artifact contains probes/conformance_report.md, setup.log, tests.log, and per-scenario probes/<harness>/<scenario>/scenario_result.json and episode.log.

Rerun locally

From the tested Gym revision, activate your Gym environment and install the selected adapters' requirements and pinned runtimes. No model service or model credentials are needed.

python scripts/run_harness_conformance.py --harness opencode --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness pi --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness codex --output "$(mktemp -d)/probes"
python scripts/run_harness_conformance.py --harness hermes --output "$(mktemp -d)/probes"

Each command runs the full suite for that harness and creates a fresh output directory. Open conformance_report.md in that directory for the full report.

Tested commit: 76937de3e8f544d80cab6a6dca647da79fdd71b6.

Probe summaries and downloadable evidence. Local setup instructions.