Repository navigation
fix(rna_design,deepswe_external1): name the accurate failure kind of … - #5710
Conversation
…masked samples rna_design reported a failed scoring call and a missing campaign as agent_run_error; they are now verifier_error and session_lost. deepswe_external1 reported every verifier failure as verifier_error; a missing or mismatched seeded session is now session_lost and a failed submission collection in agent mode is environment_server_failed. All four kinds are evaluation-side, so recovery and retry behavior, rewards and masks are unchanged. Refs #4882 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
RISK — Record validation for the changed verifier responses
The supplied PR description leaves Validation and Rollouts empty. These edits change observable verifier responses, but no checks or rollout evidence against this head are reported. This is a validation gap, not a demonstrated scoring regression. The protected quality bar requires representative real model smoke rollouts for behavior-changing environment work. The review rubric also requires a before/after reward profile for verifier changes to fixed evaluation benchmarks, which RNA Design supports. The committed DeepSWE smoke record documents earlier revisions, not this change.
Could you add the exact component-test commands and results, plus representative model rollouts for both pairings with inspected agent/verifier behavior? For RNA's fixed-suite path, provide a bounded before/after comparison using the same tasks, model settings, and budgets. Report measured and masked counts and the reward distribution, and explain any difference. A full DeepSWE accuracy run isn't required for its generic task-row environment.
Acceptance checks should cover both the relabeling and the behavior it must preserve:
- Retain the DeepSWE collection assertions: damaged Git/HEAD submissions remain measured zeros with
deepswe_external1:invalid_submission; inconclusive agent collection remains a masked zero and now reportsenvironment_server_failed. Missing or mismatched seeded sessions must reportsession_lost. Golden/null failures must remainverifier_error, and successful verification must remain unmasked without a failure kind. - Retain the RNA failure assertions: scoring timeouts return a masked zero with
verifier_errorand close the worker; missing campaigns return a masked zero withsession_lost. Preserve the ordinary answered-but-unsolved case as an unmasked zero, and solved campaigns as reward 1.
I inspected all four changed files and their directly affected contracts and consumers. I found no source-evidenced correctness regression: the labels remain evaluation-side under the defaults, and reward/mask decisions are unchanged. I couldn't execute tests or rollouts in this read-only review.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
Nonblocking review notes on
The affected test suites and lint/format checks passed during review. No live model/sandbox-provider smoke rollout was run. |
…kinds Close the popped RNA worker on the client's event loop, and pin the DeepSWE agent-mode stage split and its effect on saved-answer recovery. Refs #4882 Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
|
Thanks for the review, @yuhezhang-ai. Addressed in 5af4a71:
Both servers' tests pass on the rebased branch: 87 passed, 5 skipped. I haven't run live model or sandbox rollouts yet, so I'll add them, or an explicit justification, to the PR description. |
|
/ok to test c6b866f |
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Thanks for addressing the follow-ups. Reviewed c6b866f: the RNA worker cleanup and DeepSWE stage/recovery tests address the code and coverage notes, and CI is green on this head. No blocking correctness issues found.
One nonblocking follow-up remains: please update the compatibility description to distinguish unchanged rewards, masks and default in-run retries from the intended saved-answer recovery change, and note that custom policies keyed on the old labels need review.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42275922922 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42276277438 |
What changed
Two resources servers already mask samples (
mask_sample=True) with afailure_kindthat names the wrong cause. This PR corrects the kind and leaves the mask, reward and retry behavior unchanged.rna_design
verifier_error(wasagent_run_error).session_lost(wasagent_run_error).agent_run_errorsites (no model call, API error, environment faults) are unchanged.deepswe_external1
session_lost. A newSessionMismatchErrorreplaces the three bareRuntimeErrors that signaled it.environment_server_failed.verifier_error. Golden-mode collection staysverifier_error, because that is the verifier's own work.deepswe_external1:invalid_submissionis unchanged.Why
Part of #4882: masked results should carry an accurate, stable
failure_kindand a boundedfailure_reason. These sites already masked but named the wrong cause. This is one of the "free to fix" items from the audit in #5693.Closes nothing; refs #4882 and #5693.
Compatibility
agent_run_error,verifier_error,session_lostandenvironment_server_failedare all registered and evaluation-side, sorollout_recovery.masked_infrastructure_kindtreats them the same.failure_kindstring on masked rows changes.Validation
pytestforresources_servers/rna_designandresources_servers/deepswe_external1: 85 passed, 5 skipped.session_lost) and scoring timeout (verifier_error); deepswe_external1 session mismatch (session_lost) and agent-mode collection failure (environment_server_failed).ruff check,ruff formatandpre-commit run --filespass on the touched files. The guardwords hook was skipped locally because the private catalog is unavailable.Rollout evidence
N/A. This is a diagnostic-only label change with no change to scoring, masking or retries. It is covered by unit tests at the verify-response boundary.
Not included
Refs #4882, #5693