Skip to content

fix(rna_design,deepswe_external1): name the accurate failure kind of … - #5710

Merged
oyilmaz-nvidia merged 4 commits into
mainfrom
fix/masked-failure-kinds-rna-design-deepswe-external1
Nov 6, 2026
Merged

oyilmaz-nvidia merged 4 commits into
mainfrom
fix/masked-failure-kinds-rna-design-deepswe-external1

Conversation

@oyilmaz-nvidia

@oyilmaz-nvidia oyilmaz-nvidia commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Two resources servers already mask samples (mask_sample=True) with a failure_kind that names the wrong cause. This PR corrects the kind and leaves the mask, reward and retry behavior unchanged.

rna_design

  • A failed scoring call is now verifier_error (was agent_run_error).
  • A missing campaign for the session is now session_lost (was agent_run_error).
  • The other agent_run_error sites (no model call, API error, environment faults) are unchanged.

deepswe_external1

  • A missing or mismatched seeded session is now session_lost. A new SessionMismatchError replaces the three bare RuntimeErrors that signaled it.
  • A failed patch collection in agent mode is now environment_server_failed.
  • Everything else stays verifier_error. Golden-mode collection stays verifier_error, because that is the verifier's own work.
  • deepswe_external1:invalid_submission is unchanged.

Why

Part of #4882: masked results should carry an accurate, stable failure_kind and a bounded failure_reason. These sites already masked but named the wrong cause. This is one of the "free to fix" items from the audit in #5693.

Closes nothing; refs #4882 and #5693.

Compatibility

  • Rewards and masks are unchanged.
  • Retries are unchanged. agent_run_error, verifier_error, session_lost and environment_server_failed are all registered and evaluation-side, so rollout_recovery.masked_infrastructure_kind treats them the same.
  • Only the failure_kind string on masked rows changes.

Validation

  • pytest for resources_servers/rna_design and resources_servers/deepswe_external1: 85 passed, 5 skipped.
  • New tests: rna_design lost session (session_lost) and scoring timeout (verifier_error); deepswe_external1 session mismatch (session_lost) and agent-mode collection failure (environment_server_failed).
  • ruff check, ruff format and pre-commit run --files pass on the touched files. The guardwords hook was skipped locally because the private catalog is unavailable.

Rollout evidence

N/A. This is a diagnostic-only label change with no change to scoring, masking or retries. It is covered by unit tests at the verify-response boundary.

Not included

  • terminal_bench_4: a fallback kind would change retry behavior, and I couldn't confirm the no-kind path is reachable. It needs a decision.
  • anyswe_agent, anyterminal_agent, osworld_agent: blocked by open PRs touching the same files.

Refs #4882, #5693

…masked samples

rna_design reported a failed scoring call and a missing campaign as
agent_run_error; they are now verifier_error and session_lost.
deepswe_external1 reported every verifier failure as verifier_error; a
missing or mismatched seeded session is now session_lost and a failed
submission collection in agent mode is environment_server_failed.

All four kinds are evaluation-side, so recovery and retry behavior,
rewards and masks are unchanged.

Refs #4882

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

RISK — Record validation for the changed verifier responses

The supplied PR description leaves Validation and Rollouts empty. These edits change observable verifier responses, but no checks or rollout evidence against this head are reported. This is a validation gap, not a demonstrated scoring regression. The protected quality bar requires representative real model smoke rollouts for behavior-changing environment work. The review rubric also requires a before/after reward profile for verifier changes to fixed evaluation benchmarks, which RNA Design supports. The committed DeepSWE smoke record documents earlier revisions, not this change.

Could you add the exact component-test commands and results, plus representative model rollouts for both pairings with inspected agent/verifier behavior? For RNA's fixed-suite path, provide a bounded before/after comparison using the same tasks, model settings, and budgets. Report measured and masked counts and the reward distribution, and explain any difference. A full DeepSWE accuracy run isn't required for its generic task-row environment.

Acceptance checks should cover both the relabeling and the behavior it must preserve:

  • Retain the DeepSWE collection assertions: damaged Git/HEAD submissions remain measured zeros with deepswe_external1:invalid_submission; inconclusive agent collection remains a masked zero and now reports environment_server_failed. Missing or mismatched seeded sessions must report session_lost. Golden/null failures must remain verifier_error, and successful verification must remain unmasked without a failure kind.
  • Retain the RNA failure assertions: scoring timeouts return a masked zero with verifier_error and close the worker; missing campaigns return a masked zero with session_lost. Preserve the ordinary answered-but-unsolved case as an unmasked zero, and solved campaigns as reward 1.

I inspected all four changed files and their directly affected contracts and consumers. I found no source-evidenced correctness regression: the labels remain evaluation-side under the defaults, and reward/mask decisions are unchanged. I couldn't execute tests or rollouts in this read-only review.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@yuhezhang-ai

Copy link
Copy Markdown
Contributor

Nonblocking review notes on 48eddc18: no blocking correctness defect found. I have two small follow-ups and one optional coverage suggestion:

  1. Clarify the recovery impact (DeepSWE classification). Could we narrow the compatibility claim to unchanged rewards, masks and default in-run retries, and document the saved-answer recovery change? A DeepSWE failure retaining the model response previously selected regrading through verifier_error; the new labels select the masked-without-artifact path. That looks appropriate because this server cannot replay those patches, but recovery guidance changes. A focused saved-response test would make this explicit. Custom policies keyed on the old failure labels also need review.

  2. Finish worker cleanup in the new RNA test (line 456). Please finish cleanup before the TestClient exits. Once this worker is popped from server.workers, shutdown cannot close it, and proc.kill() does not wait for process/pipe cleanup. The test passed but logged an asyncio warning that the worker's event loop was already closed. Calling and awaiting worker.close() on the client's event loop would reuse the existing cleanup path. This is a test-cleanup issue, not a demonstrated production lifecycle defect.

  3. Optional: protect the agent-mode verifier-failure distinction (existing fallback test). Could we add an agent-mode verifier-failure case asserting verifier_error? This parametrization uses the default golden mode. In a scratch mutation check, removing the stage guard from the new agent-mode branch still left all 52 app tests passing, even though later verifier failures then received environment_server_failed. A small case after successful patch collection would protect the distinction this change intends to preserve. The submitted implementation behaved correctly in focused checks.

The affected test suites and lint/format checks passed during review. No live model/sandbox-provider smoke rollout was run.

oyilmaz-nvidia and others added 2 commits November 6, 2026 12:27
…kinds

Close the popped RNA worker on the client's event loop, and pin the DeepSWE
agent-mode stage split and its effect on saved-answer recovery.

Refs #4882

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @yuhezhang-ai. Addressed in 5af4a71:

  1. Recovery impact. Agreed, and I confirmed it in reverification_cause: a verifier_error row with a saved, intact model patch selects REVERIFY_VERIFIER, while session_lost and environment_server_failed do not. So the compatibility claim should read: rewards, masks and default in-run retries are unchanged, and saved-answer recovery changes for DeepSWE collection failures and lost sessions (those rows now take the masked-without-artifact path). That is intended, since there is no collected patch to replay. I'll narrow the PR description to say this, and note that custom policies keyed on the old labels need review. A new test pins both sides: test_agent_mode_names_the_stage_that_failed_and_keeps_recovery_aligned asserts that a collection failure gets environment_server_failed and no re-verification cause, and that a verifier failure with a saved patch keeps verifier_error and REVERIFY_VERIFIER.
  2. RNA test cleanup. The test now closes the popped worker with client.portal.call(worker.close), on the client's event loop, so the existing cleanup path runs before the TestClient exits.
  3. Agent-mode verifier failure. Added as part of the same test: after a successful patch collection, a failing verifier stays verifier_error in agent mode. The stage guard on the new branch is now covered by a test, but I have not repeated your mutation check.

Both servers' tests pass on the rebased branch: 87 passed, 5 skipped. I haven't run live model or sandbox rollouts yet, so I'll add them, or an explicit justification, to the PR description.

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test c6b866f

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the follow-ups. Reviewed c6b866f: the RNA worker cleanup and DeepSWE stage/recovery tests address the code and coverage notes, and CI is green on this head. No blocking correctness issues found.

One nonblocking follow-up remains: please update the compatibility description to distinguish unchanged rewards, masks and default in-run retries from the intended saved-answer recovery change, and note that custom policies keyed on the old labels need review.

@oyilmaz-nvidia
oyilmaz-nvidia added this pull request to the merge queue Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42275922922

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42276277438

Merged via the queue into main with commit 7e96a5b Nov 6, 2026
68 checks passed
@oyilmaz-nvidia
oyilmaz-nvidia deleted the fix/masked-failure-kinds-rna-design-deepswe-external1 branch November 6, 2026 19:18

This branch was previously deployed

1 inactive deployment
public — c6b866fa Deployed Nov 6, 2026 by copy-pr-bot[bot] via release / finalize / notify #10702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants