Skip to content

test: mirror nemo_gym packages in the unit test tree - #5589

Merged
ananthsub merged 3 commits into
mainfrom
ananthsub/tests-mirror-packages
Nov 4, 2026
Merged

ananthsub merged 3 commits into
mainfrom
ananthsub/tests-mirror-packages

docs(agents): say where library code and tests go

05f0f17
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
GitHub Actions / Harness conformance (non-blocking) failed Nov 4, 2026 in 0s

7 passed, 0 failed, 1 unavailable

Harness P0 result Failed scenario checks First failure Evidence
simple passed 1 / 9 context_compaction: expected one completed automatic compaction, or none in the control Download report and logs
opencode passed 0 / 9 None Download report and logs
pi passed 1 / 9 context_compaction: expected one completed automatic compaction, or none in the control Download report and logs
codex passed 0 / 9 None Download report and logs
hermes passed 0 / 9 None Download report and logs
kilocode unavailable 1 / 9 model_error: the EnvironmentServer must receive a failed response retaining the model error and ng_agent_outcome with model_request_error and the witnessed HTTP status Download report and logs
stirrup passed 0 / 9 None Download report and logs
openclaw passed 1 / 9 context_compaction: expected one completed automatic compaction, or none in the control Download report and logs

Counts include execution errors and count each scenario once. Unavailable means setup, tests, execution, or evidence collection failed.

Each artifact contains probes/conformance_report.md, setup.log, tests.log, and per-scenario probes/<harness>/<scenario>/scenario_result.json and episode.log.

Rerun locally

From the tested Gym revision, activate your Gym environment and install the selected adapters' requirements and pinned runtimes. No model service or model credentials are needed.

python scripts/run_harness_conformance.py --harness kilocode --output "$(mktemp -d)/probes"

Each command runs the full suite for that harness and creates a fresh output directory. Open conformance_report.md in that directory for the full report.

Tested commit: 05f0f17f6f4131bdbc079cd5ec09e4c5398425bb.

Probe summaries and downloadable evidence. Local setup instructions.