Repository navigation
fix(terminal_bench_4): label a masked agent infrastructure failure wi… - #5768
Conversation
…th a failure_kind A masked result whose termination was an infrastructure_error with no typed kind and no harness-selected kind had mask_sample=True and failure_kind=None. Fall back to infrastructure_error, the kind the sidecar class already uses, for masked results. Refs #5693 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
…ure_kind anyswe_agent: sandbox failures keep the event's kind (provider_unavailable otherwise, reason from repr(exc)); agent timeout is agent_timeout; eval timeout is anyswe_agent:eval_timeout; resolved runs ending on max_iteration/context_window are agent_limit/model_context_limit. anyterminal_agent: carry the exec error_type through so sandbox errors (provider_unavailable) and timeouts (agent_timeout, anyterminal_agent:eval_timeout) separate. The existing boolean metrics and the mask are unchanged. Refs #5693 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
|
/review model=claude |
|
/review |
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Please address the public-import test failure and the inaccurate failure classifications before merging. There is also a smaller correction to the evaluation-failure message. I independently reproduced the findings with focused tests; no live model rollouts were run. Please fill in the PR’s Validation section, including rollout evidence or a justified N/A.
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE. The labeling logic matches the masks, but the new imports will turn the core unit suite red, and one new failure_reason value is wrong on the normal eval-timeout path.
Reward and mask_sample don't change anywhere in this PR:
- terminal_bench_4: The
infrastructure_errorfallback is correct. In_verified_response, the_agent_failure_kindarm is reached only whentermination.reason == "infrastructure_error"andungraded_verifieris false. In that casemasked=not legacyequals themask_samplevalue the same response sets. The new tests would fail if the change were reverted. - anyswe_agent:
_masked_failure_kindreturns a kind exactly when_should_mask_samplemasks the sample. - anyterminal_agent:
mask_sampleis true exactly when_masked_failurereturns a kind.
Findings
| # | Grade | Where | Summary |
|---|---|---|---|
| 1 | RISK | anyswe_agent/app.py#L45, anyterminal_agent/app.py#L39 | The submodule imports fail test_code_outside_the_library_imports_exported_names_from_the_public_api. |
| 2 | RISK | anyswe_agent/app.py#L949-L954 | An eval timeout or eval sandbox error reported by exec gets failure_reason="the resolved run ended with …". |
| 3 | NOTE | anyterminal_agent/app.py#L438 | A later sandbox exception takes priority over an earlier agent timeout, which contradicts the docstring. |
| 4 | NOTE | anyterminal_agent/app.py#L702 | A typed NgFailureEvent kind is replaced by provider_unavailable; anyswe keeps it. |
| 5 | RISK | PR body | The Validation section is empty (details below). |
5. RISK: no validation or rollout evidence
The PR body's ## Validation heading has nothing under it. AGENTS.md asks a PR body to list the exact validation performed and to include rollout evidence or an explicit N/A with a justification. It also asks for representative smoke rollouts for behavior-changing agent work. Reward and the mask don't change here, but three components now emit new failure_kind and failure_reason values, which the description says feed retry, reporting and triage.
Could you list the tests you ran? Please include tests/unit_tests/test_public_api_facades.py, which finding 1 expects to fail. Then add either a short smoke rollout per agent that shows the new fields on a masked row, or an N/A with the reason.
- Acceptance: the Validation section names the commands run and their results, and it has rollout evidence or an
N/Awith a justification.
Coverage
I reviewed all six changed files and traced their direct paths:
- the terminal_bench_4
_verified_responseand_agent_failure_kindarms and their fixture; - the anyswe
_run_agent_in_sandbox, the grading helpers and howrun()parses the metrics; - the anyterminal
process_single_datapoint,_masked_failureand its callers; - the
nemo_gym.corepublic API andfailure_kindsregistration.
Two of the three delegated topic reviews finished, but their reports never reached me. I therefore reviewed the terminal_bench_4 and anyswe_agent topics myself. I couldn't run any tests, so every pass/fail claim above comes from reading the source.
Review generated by claude. This was a full review. Request a full review with /review reset=true.
… labels Import the failure kinds through nemo_gym.core, give returned grading failures an evaluation-specific reason, let an earlier exec error name an anyterminal run over a later sandbox exception, keep typed sandbox kinds, and label missing runtime or task tests as setup failures instead of provider failures. Reward and mask_sample are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
|
Thanks for the reviews. All findings are addressed in e62cba0:
|
|
/ok to test 4753cbd |
|
/ok to merge 59f976f |
|
@oyilmaz-nvidia I’ve resolved the fixed review threads. One classification follow-up remains: failed runtime probes are labeled as missing runtimes, and invalid archives are labeled as provider outages. Could you take a look before merging? |
…dbox failures A runtime probe or extraction whose exec did not complete (timeout or sandbox error_type) now keeps provider_unavailable instead of being labeled a missing runtime. A completed extraction that reports a bad archive gets a setup kind (anyterminal_agent:setup_failed, anyswe_agent:setup_failed) instead of provider_unavailable, so it is not retried. Reward and mask_sample are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
|
Pushed d678d6d for the remaining classification follow-up: failed runtime probes no longer become missing-runtime labels, and invalid archives are labeled as setup failures instead of provider outages. Details are in the reply on the typed-kinds thread. |
|
/review |
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Thanks for addressing the review feedback. The remaining runtime setup cases are verified, and the focused regression tests pass. Approved; current-head CI should complete before merging.
|
/ok to test a17cd57 |
|
/ok to test df9e0e3 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42921573504 |
…failure_kind (NVIDIA-NeMo#6114) A legacy-agent result whose agent or task environment failed on infrastructure (a Compose service that did not start, a log directory that could not be created) is routed to the failures sidecar with _ng_failure_class=infrastructure_error but failure_kind=None. Recovery stops a run on a failure with no kind instead of retrying it. Fall back to infrastructure_error for these rows, as NVIDIA-NeMo#5768 did for masked results; typed kinds and terminal setup rows are unchanged. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Summary
Several agents and resources servers mask a result (
mask_sample=True) without saying why, so the row is excluded from metrics but has nofailure_kindfor retry, reporting, or triage. This PR gives each of the masked paths below a kind. Reward andmask_sampledo not change anywhere.Refs #5693, part of #4882.
terminal_bench_4
A masked result whose termination was an
infrastructure_errorwith no typed kind (only rejected sandbox credentials set one) and no harness-selected kind hadfailure_kind=None. It now falls back toinfrastructure_error, the registered kind the legacy sidecar class already uses for the same condition. Priority is unchanged: typed termination kind, then harness-selected kind, then the fallback. Legacy results are unchanged. The missing-reward arm already resolved toverifier_error; a test now locks that in.anyswe_agent
SWEBenchMetricsand the verify response carryfailure_kindandfailure_reason. A new_masked_failure_kindreturns a kind exactly when_should_mask_samplemasks:failure_kindprovider_unavailable; reason fromrepr(exc); a returned evaltimeout/sandboxresult gets an evaluation-specific reasonagent_timeoutanyswe_agent:eval_timeoutanyswe_agent:runtime_missingmax_iterationagent_limitcontext_windowmodel_context_limitanyterminal_agent
_run_agentand_run_evalnow return the execerror_type(timeoutorsandbox) instead of a bool, so sandbox errors and timeouts separate:failure_kindexec)provider_unavailableprovider_unavailableanyterminal_agent:setup_failedagent_timeoutanyterminal_agent:eval_timeoutA recorded
execerror names the run before a later sandbox exception (for example a failingstop()), since the stages run in order.The existing boolean metrics (
agent_timed_out,container_timed_out,sandbox_failed) and the mask are unchanged.Validation
Unit tests (new tests assert the kind, reason and mask of each case):
pytest tests/unit_tests/test_public_api_facades.py: 844 passed (failed before the import fix).pytest responses_api_agents/anyswe_agent responses_api_agents/anyterminal_agent: 365 passed, 6 failed. The 6 aretest_r2e_grading.py::test_eval_output_is_opt_in_without_changing_grading, which needs theswebenchpackage missing from my environment; the files are untouched by this PR.gym env test terminal_bench_4: 350 passed, 1 failed (test_shared_logs.py::test_efs_snapshot_matches_host_restore_and_survives_agent_deletion, a file-mode assertion on a file this PR does not touch), 1 skipped.pre-commit run --files <changed files>andruff check/ruff format: pass.Rollout evidence (anyterminal_agent, Terminus-2, Qwen3-8B on local vLLM, Docker provider, Terminal Bench 2.1
fix-git):global_agent_timeout=30mask_sample=true,failure_kind=agent_timeout,failure_reason="the agent command timed out",reward=0.0tests/directory does not existmask_sample=true,failure_kind=anyterminal_agent:setup_failed,failure_reasonnames the missing pathN/A for anyswe_agent and terminal_bench_4: no rollout was run. The SWE-bench instance images do not fit on my machine (about 4 GB of disk free). Both changes only add
failure_kind/failure_reasonbeside unchanged reward andmask_sample, and their paths are covered by the new tests (includingtest_returned_grading_failure_has_an_evaluation_reasonand the terminal_bench_4_verified_responsetests).