Repository navigation
fix(terminal_bench_4): give a sidecar agent infrastructure failure a failure_kind - #6114
Conversation
…failure_kind A legacy-agent result whose agent or task environment failed on infrastructure (a Compose service that did not start, a log directory that could not be created) is routed to the failures sidecar with _ng_failure_class=infrastructure_error but failure_kind=None. Recovery stops a run on a failure with no kind instead of retrying it. Fall back to infrastructure_error for these rows, as #5768 did for masked results; typed kinds and terminal setup rows are unchanged. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE — one validation-evidence RISK; no code correctness regression found.
RISK — Add the required model-backed smoke evidence. The PR description reports unit tests and Ruff checks, but the trusted quality bar requires representative real model rollouts for behavior-changing environment work. The fallback change changes recovery behavior for legacy TB4 failures, so unit assertions alone don't satisfy that requirement. This is missing validation evidence, not a demonstrated runtime defect.
Could you run a representative model-backed legacy TB4 smoke with one controlled transient setup failure and add the command/configuration, artifact references, and agent/verifier observations to the PR description? Expected: the first attempt produces a sidecar row with _ng_failure_class and failure_kind both infrastructure_error, with mask_sample=false; recovery schedules another attempt within its budget; the retry runs the agent and records the official verifier grade. No code rewrite is requested.
Retain the neighboring contracts: typed kinds still take precedence, deterministic setup failures remain terminal, verifier-outcome zeros remain unmasked, and session-path infrastructure failures retain their masked fallback. The existing response tests and terminal setup regression should keep those assertions.
I inspected both changed files and traced setup, kind selection, sidecar persistence, and recovery consumers. I couldn't execute tests in this read-only review.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43428639259 |
Terminal-Bench 4 routes an agent or task-environment infrastructure failure on the legacy agent path to the failures sidecar with
_ng_failure_class=infrastructure_errorbutfailure_kind=None. Recovery resolves a kind fromfailure_kind, then the row's failure events, then the canonical failure record. A row carrying none of the three is classified unknown, and the run stops instead of retrying it.This falls back to
infrastructure_errorfor these sidecar rows, as #5768 did for masked results. Typed kinds and terminal setup rows keep their behaviour.Tests:
resources_servers/terminal_bench_4tests pass; the changed and new assertions fail on unchangedmain.ruff checkandruff format --checkpass on both files.Live check, mini-SWE profile against a real model with real sandboxes, one Compose task capped at 3 steps. Forcing one sidecar service's healthcheck to fail produced
RuntimeError: Compose service 'api' unhealthy, and the sidecar row carriedfailure_kind=infrastructure_error,_ng_failure_class=infrastructure_error,mask_sample=falseand no terminal flag.decide_recoveryon that row returnedresubmit(retryable: 1). Re-running the same task with the induced fault cleared completed the agent and recorded an official grade (evaluation_completed=true, reward 0.0 at the step cap) with trajectory and agent-log artifacts.Scope, measured on the seven failure rows of a real stuck run whose runtime wrote no canonical failure record: as produced,
decide_recoveryreturnsstopwith all seven unknown ("no failure kind"); with the kind set, five of the seven become retryable while two remain unowned, so the run-level action is stillstop. This change removes the missing-kind cause. It does not by itself make such a run resumable.