Skip to content

fix(terminal_bench_4): label a masked agent infrastructure failure wi… - #5768

Merged
oyilmaz-nvidia merged 8 commits into
mainfrom
fix/terminal-bench-4-masked-failure-kind
Nov 10, 2026
Merged

oyilmaz-nvidia merged 8 commits into
mainfrom
fix/terminal-bench-4-masked-failure-kind

Conversation

@oyilmaz-nvidia

@oyilmaz-nvidia oyilmaz-nvidia commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Several agents and resources servers mask a result (mask_sample=True) without saying why, so the row is excluded from metrics but has no failure_kind for retry, reporting, or triage. This PR gives each of the masked paths below a kind. Reward and mask_sample do not change anywhere.

Refs #5693, part of #4882.

terminal_bench_4

A masked result whose termination was an infrastructure_error with no typed kind (only rejected sandbox credentials set one) and no harness-selected kind had failure_kind=None. It now falls back to infrastructure_error, the registered kind the legacy sidecar class already uses for the same condition. Priority is unchanged: typed termination kind, then harness-selected kind, then the fallback. Legacy results are unchanged. The missing-reward arm already resolved to verifier_error; a test now locks that in.

anyswe_agent

SWEBenchMetrics and the verify response carry failure_kind and failure_reason. A new _masked_failure_kind returns a kind exactly when _should_mask_sample masks:

Masked because failure_kind
Sandbox failure (agent command or eval) the failure event's kind, else provider_unavailable; reason from repr(exc); a returned eval timeout/sandbox result gets an evaluation-specific reason
Agent timeout agent_timeout
Eval timeout anyswe_agent:eval_timeout
Missing agent runtime in the task image anyswe_agent:runtime_missing
Resolved run ended on max_iteration agent_limit
Resolved run ended on context_window model_context_limit

anyterminal_agent

_run_agent and _run_eval now return the exec error_type (timeout or sandbox) instead of a bool, so sandbox errors and timeouts separate:

Masked because failure_kind
Sandbox error (agent or eval exec) provider_unavailable
Sandbox exception the exception's kind when it is a failure event, else provider_unavailable
Missing task tests directory or agent runtime in the task image anyterminal_agent:setup_failed
Agent timeout agent_timeout
Eval timeout anyterminal_agent:eval_timeout

A recorded exec error names the run before a later sandbox exception (for example a failing stop()), since the stages run in order.

The existing boolean metrics (agent_timed_out, container_timed_out, sandbox_failed) and the mask are unchanged.

Validation

Unit tests (new tests assert the kind, reason and mask of each case):

  • pytest tests/unit_tests/test_public_api_facades.py: 844 passed (failed before the import fix).
  • pytest responses_api_agents/anyswe_agent responses_api_agents/anyterminal_agent: 365 passed, 6 failed. The 6 are test_r2e_grading.py::test_eval_output_is_opt_in_without_changing_grading, which needs the swebench package missing from my environment; the files are untouched by this PR.
  • gym env test terminal_bench_4: 350 passed, 1 failed (test_shared_logs.py::test_efs_snapshot_matches_host_restore_and_survives_agent_deletion, a file-mode assertion on a file this PR does not touch), 1 skipped.
  • pre-commit run --files <changed files> and ruff check / ruff format: pass.

Rollout evidence (anyterminal_agent, Terminus-2, Qwen3-8B on local vLLM, Docker provider, Terminal Bench 2.1 fix-git):

Run Result
global_agent_timeout=30 mask_sample=true, failure_kind=agent_timeout, failure_reason="the agent command timed out", reward=0.0
Default timeout (300s per task) The model hit the agent timeout: same masked row as above. No unmasked row was produced because the 8B model did not finish.
Task whose tests/ directory does not exist mask_sample=true, failure_kind=anyterminal_agent:setup_failed, failure_reason names the missing path

N/A for anyswe_agent and terminal_bench_4: no rollout was run. The SWE-bench instance images do not fit on my machine (about 4 GB of disk free). Both changes only add failure_kind/failure_reason beside unchanged reward and mask_sample, and their paths are covered by the new tests (including test_returned_grading_failure_has_an_evaluation_reason and the terminal_bench_4 _verified_response tests).

…th a failure_kind

A masked result whose termination was an infrastructure_error with no typed kind
and no harness-selected kind had mask_sample=True and failure_kind=None. Fall back
to infrastructure_error, the kind the sidecar class already uses, for masked results.

Refs #5693

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…ure_kind

anyswe_agent: sandbox failures keep the event's kind (provider_unavailable otherwise,
reason from repr(exc)); agent timeout is agent_timeout; eval timeout is
anyswe_agent:eval_timeout; resolved runs ending on max_iteration/context_window are
agent_limit/model_context_limit.

anyterminal_agent: carry the exec error_type through so sandbox errors
(provider_unavailable) and timeouts (agent_timeout, anyterminal_agent:eval_timeout)
separate. The existing boolean metrics and the mask are unchanged.

Refs #5693

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/review model=claude

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/review

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the public-import test failure and the inaccurate failure classifications before merging. There is also a smaller correction to the evaluation-failure message. I independently reproduced the findings with focused tests; no live model rollouts were run. Please fill in the PR’s Validation section, including rollout evidence or a justified N/A.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE. The labeling logic matches the masks, but the new imports will turn the core unit suite red, and one new failure_reason value is wrong on the normal eval-timeout path.

Reward and mask_sample don't change anywhere in this PR:

  • terminal_bench_4: The infrastructure_error fallback is correct. In _verified_response, the _agent_failure_kind arm is reached only when termination.reason == "infrastructure_error" and ungraded_verifier is false. In that case masked=not legacy equals the mask_sample value the same response sets. The new tests would fail if the change were reverted.
  • anyswe_agent: _masked_failure_kind returns a kind exactly when _should_mask_sample masks the sample.
  • anyterminal_agent: mask_sample is true exactly when _masked_failure returns a kind.

Findings

# Grade Where Summary
1 RISK anyswe_agent/app.py#L45, anyterminal_agent/app.py#L39 The submodule imports fail test_code_outside_the_library_imports_exported_names_from_the_public_api.
2 RISK anyswe_agent/app.py#L949-L954 An eval timeout or eval sandbox error reported by exec gets failure_reason="the resolved run ended with …".
3 NOTE anyterminal_agent/app.py#L438 A later sandbox exception takes priority over an earlier agent timeout, which contradicts the docstring.
4 NOTE anyterminal_agent/app.py#L702 A typed NgFailureEvent kind is replaced by provider_unavailable; anyswe keeps it.
5 RISK PR body The Validation section is empty (details below).

5. RISK: no validation or rollout evidence

The PR body's ## Validation heading has nothing under it. AGENTS.md asks a PR body to list the exact validation performed and to include rollout evidence or an explicit N/A with a justification. It also asks for representative smoke rollouts for behavior-changing agent work. Reward and the mask don't change here, but three components now emit new failure_kind and failure_reason values, which the description says feed retry, reporting and triage.

Could you list the tests you ran? Please include tests/unit_tests/test_public_api_facades.py, which finding 1 expects to fail. Then add either a short smoke rollout per agent that shows the new fields on a masked row, or an N/A with the reason.

  • Acceptance: the Validation section names the commands run and their results, and it has rollout evidence or an N/A with a justification.

Coverage

I reviewed all six changed files and traced their direct paths:

  • the terminal_bench_4 _verified_response and _agent_failure_kind arms and their fixture;
  • the anyswe _run_agent_in_sandbox, the grading helpers and how run() parses the metrics;
  • the anyterminal process_single_datapoint, _masked_failure and its callers;
  • the nemo_gym.core public API and failure_kinds registration.

Two of the three delegated topic reviews finished, but their reports never reached me. I therefore reviewed the terminal_bench_4 and anyswe_agent topics myself. I couldn't run any tests, so every pass/fail claim above comes from reading the source.

Review generated by claude. This was a full review. Request a full review with /review reset=true.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior needs-review PR is ready for code review and waiting on a reviewer labels Nov 7, 2026
… labels

Import the failure kinds through nemo_gym.core, give returned grading
failures an evaluation-specific reason, let an earlier exec error name an
anyterminal run over a later sandbox exception, keep typed sandbox kinds, and
label missing runtime or task tests as setup failures instead of provider
failures. Reward and mask_sample are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews. All findings are addressed in e62cba0:

  • Imports go through nemo_gym.core. The public-API test passes.
  • anyswe: returned eval failures get an evaluation-specific failure_reason.
  • anyterminal: a recorded exec error wins over a later cleanup exception.
  • anyterminal: typed sandbox kinds are kept. Local setup errors get anyterminal_agent:setup_failed instead of a provider label. anyswe gets the matching anyswe_agent:runtime_missing.
  • The PR description now has the Validation section, including rollout evidence. Reward and mask_sample are unchanged.

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 4753cbd

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to merge 59f976f

@yuhezhang-ai

Copy link
Copy Markdown
Contributor

@oyilmaz-nvidia I’ve resolved the fixed review threads. One classification follow-up remains: failed runtime probes are labeled as missing runtimes, and invalid archives are labeled as provider outages. Could you take a look before merging?

…dbox failures

A runtime probe or extraction whose exec did not complete (timeout or
sandbox error_type) now keeps provider_unavailable instead of being labeled a
missing runtime. A completed extraction that reports a bad archive gets a
setup kind (anyterminal_agent:setup_failed, anyswe_agent:setup_failed) instead
of provider_unavailable, so it is not retried. Reward and mask_sample are
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

Pushed d678d6d for the remaining classification follow-up: failed runtime probes no longer become missing-runtime labels, and invalid archives are labeled as setup failures instead of provider outages. Details are in the reply on the typed-kinds thread.

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 10, 2026

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the review feedback. The remaining runtime setup cases are verified, and the focused regression tests pass. Approved; current-head CI should complete before merging.

@yuhezhang-ai

Copy link
Copy Markdown
Contributor

/ok to test a17cd57

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test df9e0e3

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42921573504

Merged via the queue into main with commit ada2efa Nov 10, 2026
61 of 63 checks passed
@oyilmaz-nvidia
oyilmaz-nvidia deleted the fix/terminal-bench-4-masked-failure-kind branch November 10, 2026 23:25
katerynaCh pushed a commit to katerynaCh/Gym that referenced this pull request Nov 13, 2026
…failure_kind (NVIDIA-NeMo#6114)

A legacy-agent result whose agent or task environment failed on infrastructure (a Compose service that did not start, a log directory that could not be created) is routed to the failures sidecar with _ng_failure_class=infrastructure_error but failure_kind=None. Recovery stops a run on a failure with no kind instead of retrying it. Fall back to infrastructure_error for these rows, as NVIDIA-NeMo#5768 did for masked results; typed kinds and terminal setup rows are unchanged.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants