Skip to content

fix(terminal_bench_4): give a sidecar agent infrastructure failure a failure_kind - #6114

Merged
lafrancef merged 1 commit into
mainfrom
fix/terminal-bench-4-legacy-failure-kind
Nov 13, 2026
Merged

lafrancef merged 1 commit into
mainfrom
fix/terminal-bench-4-legacy-failure-kind

Conversation

@wprazuch

@wprazuch wprazuch commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Terminal-Bench 4 routes an agent or task-environment infrastructure failure on the legacy agent path to the failures sidecar with _ng_failure_class=infrastructure_error but failure_kind=None. Recovery resolves a kind from failure_kind, then the row's failure events, then the canonical failure record. A row carrying none of the three is classified unknown, and the run stops instead of retrying it.

This falls back to infrastructure_error for these sidecar rows, as #5768 did for masked results. Typed kinds and terminal setup rows keep their behaviour.

Tests:

  • resources_servers/terminal_bench_4 tests pass; the changed and new assertions fail on unchanged main.
  • ruff check and ruff format --check pass on both files.

Live check, mini-SWE profile against a real model with real sandboxes, one Compose task capped at 3 steps. Forcing one sidecar service's healthcheck to fail produced RuntimeError: Compose service 'api' unhealthy, and the sidecar row carried failure_kind=infrastructure_error, _ng_failure_class=infrastructure_error, mask_sample=false and no terminal flag. decide_recovery on that row returned resubmit (retryable: 1). Re-running the same task with the induced fault cleared completed the agent and recorded an official grade (evaluation_completed=true, reward 0.0 at the step cap) with trajectory and agent-log artifacts.

Scope, measured on the seven failure rows of a real stuck run whose runtime wrote no canonical failure record: as produced, decide_recovery returns stop with all seven unknown ("no failure kind"); with the kind set, five of the seven become retryable while two remain unowned, so the run-level action is still stop. This change removes the missing-kind cause. It does not by itself make such a run resumable.

…failure_kind

A legacy-agent result whose agent or task environment failed on infrastructure (a Compose service that did not start, a log directory that could not be created) is routed to the failures sidecar with _ng_failure_class=infrastructure_error but failure_kind=None. Recovery stops a run on a failure with no kind instead of retrying it. Fall back to infrastructure_error for these rows, as #5768 did for masked results; typed kinds and terminal setup rows are unchanged.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 13, 2026 10:10

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — one validation-evidence RISK; no code correctness regression found.

RISK — Add the required model-backed smoke evidence. The PR description reports unit tests and Ruff checks, but the trusted quality bar requires representative real model rollouts for behavior-changing environment work. The fallback change changes recovery behavior for legacy TB4 failures, so unit assertions alone don't satisfy that requirement. This is missing validation evidence, not a demonstrated runtime defect.

Could you run a representative model-backed legacy TB4 smoke with one controlled transient setup failure and add the command/configuration, artifact references, and agent/verifier observations to the PR description? Expected: the first attempt produces a sidecar row with _ng_failure_class and failure_kind both infrastructure_error, with mask_sample=false; recovery schedules another attempt within its budget; the retry runs the agent and records the official verifier grade. No code rewrite is requested.

Retain the neighboring contracts: typed kinds still take precedence, deterministic setup failures remain terminal, verifier-outcome zeros remain unmasked, and session-path infrastructure failures retain their masked fallback. The existing response tests and terminal setup regression should keep those assertions.

I inspected both changed files and traced setup, kind selection, sidecar persistence, and recovery consumers. I couldn't execute tests in this read-only review.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@yaoyu-33
yaoyu-33 requested a review from init-nikhil November 13, 2026 10:33
@lafrancef
lafrancef added this pull request to the merge queue Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43428639259

Merged via the queue into main with commit d56ea74 Nov 13, 2026
67 checks passed
@lafrancef
lafrancef deleted the fix/terminal-bench-4-legacy-failure-kind branch November 13, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants