Repository navigation
fix(resources): name the failure kind on masked swe_atlas_qa, nemo_user_sim and gdpval results - #5315
Conversation
|
/ok to test b09ef38 |
|
/ok to test 6a23a2c |
|
🌿 Preview your docs: https://nvidia-preview-pr-5315-7a405c233446.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found.
Reviewed all ten changed files and affected response, collection/retry, and aggregation contracts. The diagnostic additions in SWE Atlas Q&A, NeMo User Sim, and GDPVal preserve existing reward and masking behavior. The updated tests assert concrete diagnostics and retain neighboring scored-path coverage; documentation matches the changes.
Static review only; tests and model rollouts were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…er_sim and gdpval results These servers mask some failed results but left failure_kind empty, so a consumer could not tell why a row was masked. NeMo RL plans to read the kind to decide whether a masked sample still counts in its advantage calculation. Name the kind on those paths and keep rewards, masks and evaluation_completed unchanged: - swe_atlas_qa: a verify with no seeded task session is session_lost, through lost_session_verification; an upstream evaluation that scores no rubric because every judge attempt failed is judge_failed (its reason now names RubricsUnscoredError); any other collection or grading failure is verifier_error. - nemo_user_sim: a failed simulation attributed outside the Assistant whose outcome names no failure_class is usersim:simulation_failed, and its reason falls back to the outcome's failure_reason. - gdpval: judge_deliverables=false masks rows as gdpval:judging_disabled, with a reason, so deliberately unjudged rows are not read as failures. Part of #4882. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
6a23a2c to
b104044
Compare
|
/ok to test b104044 |
The Assistant path and the masked-failure path both read failure_detail, then failure_reason, from the simulation outcome. Read it through one helper so the two stay in step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
|
/ok to test 73ee7fa |
|
/ok to test 4e0f380 |
|
/ok to test 4e0f380 |
@oyilmaz-nvidia, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 7a405c2 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42078814239 |
What does this PR do?
Second slice of #4882 (parent #3180), after #5223. Three more resources servers mask some failed results but leave
failure_kindempty. NeMo RL plans to readfailure_kindto decide whether a masked sample still counts in its advantage calculation, so an empty kind leaves it unable to decide. This PR names the kind on those paths. Rewards, masks andevaluation_completedstay as they are.failure_kindaddedsession_lost, through the sharedlost_session_verificationhelper; the reason is unchangedjudge_failed; the reason now namesRubricsUnscoredErrorinstead ofRuntimeErrorverifier_errorfailure_classusersim:simulation_failed; the reason falls back to the outcome'sfailure_reason, as the Assistant-attributed path already doesjudge_deliverables: false, which saves deliverables without judging themgdpval:judging_disabled, plus a reason, so deliberately unjudged rows are not mistaken for failuresThe two server-prefixed kinds use the
<server>:<kind>form for conditions the shared vocabulary should not grow a name for;usersim:matches the prefix nemo_user_sim already uses. Rollout collection readsfailure_kindonly on failures-sidecar rows, so on these completed, masked results it is a diagnostic field: routing, scoring, retries and health counts do not change.Validation
Against the final diff, Python 3.13.14:
failure_class, and checks the kind and reason. The case without a class fails against the unpatched server.Rollouts
N/A. Only diagnostic fields on already-masked paths change; rewards, masks and the scored paths are untouched, and the tests above call
verifydirectly on each changed path.Compatibility and benchmark impact
None for scores. Masked rows from these servers now carry a
failure_kind, so tools that group failures by kind will start counting them. swe_atlas_qa's reason for unscored rubrics now namesRubricsUnscoredError.Checklist