Skip to content

fix(resources): name the failure kind on masked swe_atlas_qa, nemo_user_sim and gdpval results - #5315

Merged
oyilmaz-nvidia merged 6 commits into
mainfrom
yuhez/4882-masked-failure-metadata-2
Nov 5, 2026
Merged

oyilmaz-nvidia merged 6 commits into
mainfrom
yuhez/4882-masked-failure-metadata-2

Conversation

@yuhezhang-ai

@yuhezhang-ai yuhezhang-ai commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Second slice of #4882 (parent #3180), after #5223. Three more resources servers mask some failed results but leave failure_kind empty. NeMo RL plans to read failure_kind to decide whether a masked sample still counts in its advantage calculation, so an empty kind leaves it unable to decide. This PR names the kind on those paths. Rewards, masks and evaluation_completed stay as they are.

Server Masked path failure_kind added
swe_atlas_qa verify with no seeded task session (never seeded, already verified or closed, server restart) session_lost, through the shared lost_session_verification helper; the reason is unchanged
swe_atlas_qa the upstream evaluator scores no rubric because every judge attempt failed (masked since #5105) judge_failed; the reason now names RubricsUnscoredError instead of RuntimeError
swe_atlas_qa any other failure while collecting or grading the answer verifier_error
nemo_user_sim a simulation failure attributed outside the Assistant whose outcome names no failure_class usersim:simulation_failed; the reason falls back to the outcome's failure_reason, as the Assistant-attributed path already does
gdpval judge_deliverables: false, which saves deliverables without judging them gdpval:judging_disabled, plus a reason, so deliberately unjudged rows are not mistaken for failures

The two server-prefixed kinds use the <server>:<kind> form for conditions the shared vocabulary should not grow a name for; usersim: matches the prefix nemo_user_sim already uses. Rollout collection reads failure_kind only on failures-sidecar rows, so on these completed, masked results it is a diagnostic field: routing, scoring, retries and health counts do not change.

Validation

Against the final diff, Python 3.13.14:

pytest resources_servers/swe_atlas_qa/tests
gym env test --resources-server nemo_user_sim
gym env test --resources-server gdpval
pre-commit run --files <changed files>
  • swe_atlas_qa: 68 passed. The failure tests now check the kind on each path: a missing session (including a duplicate verify), zero scored rubrics, a failed answer probe and a failed grader. A new test verifies without a seed. Scored results and missing or empty answers keep no kind. The 9 updated failure-path tests fail against the unpatched server.
  • nemo_user_sim, in its own server environment: 41 passed. A new test runs the real evaluator on a failed simulation, with and without a failure_class, and checks the kind and reason. The case without a class fails against the unpatched server.
  • gdpval, in its own server environment: 899 passed, 2 skipped. The unjudged-deliverables test checks the kind and reason, and fails against the unpatched server; an existing test already checks that a judged row has no kind.
  • Pre-commit passed on the changed files.

Rollouts

N/A. Only diagnostic fields on already-masked paths change; rewards, masks and the scored paths are untouched, and the tests above call verify directly on each changed path.

Compatibility and benchmark impact

None for scores. Masked rows from these servers now carry a failure_kind, so tools that group failures by kind will start counting them. swe_atlas_qa's reason for unscored rubrics now names RubricsUnscoredError.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated edits are excluded.
  • Tests added or updated and pass locally.
  • Documentation updated: swe_atlas_qa and nemo_user_sim READMEs, gdpval's config comment and benchmark README, and the NeMo User Sim rollout troubleshooting page.
  • Pre-commit checks pass locally on the changed files.
  • New source files include the required Apache-2.0 SPDX header — N/A, no new source files.
  • All commits have DCO sign-off.

@yuhezhang-ai yuhezhang-ai added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Oct 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test b09ef38

@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test 6a23a2c

@yuhezhang-ai
yuhezhang-ai marked this pull request as ready for review October 29, 2026 16:53
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 29, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 29, 2026 17:03
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

Reviewed all ten changed files and affected response, collection/retry, and aggregation contracts. The diagnostic additions in SWE Atlas Q&A, NeMo User Sim, and GDPVal preserve existing reward and masking behavior. The updated tests assert concrete diagnostics and retain neighboring scored-path coverage; documentation matches the changes.

Static review only; tests and model rollouts were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

yuhezhang-ai and others added 2 commits November 2, 2026 06:37
…er_sim and gdpval results

These servers mask some failed results but left failure_kind empty, so a consumer could not tell why a
row was masked. NeMo RL plans to read the kind to decide whether a masked sample still counts in its
advantage calculation. Name the kind on those paths and keep rewards, masks and evaluation_completed
unchanged:

- swe_atlas_qa: a verify with no seeded task session is session_lost, through lost_session_verification;
  an upstream evaluation that scores no rubric because every judge attempt failed is judge_failed (its
  reason now names RubricsUnscoredError); any other collection or grading failure is verifier_error.
- nemo_user_sim: a failed simulation attributed outside the Assistant whose outcome names no
  failure_class is usersim:simulation_failed, and its reason falls back to the outcome's failure_reason.
- gdpval: judge_deliverables=false masks rows as gdpval:judging_disabled, with a reason, so deliberately
  unjudged rows are not read as failures.

Part of #4882.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai
yuhezhang-ai force-pushed the yuhez/4882-masked-failure-metadata-2 branch from 6a23a2c to b104044 Compare November 2, 2026 14:55
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test b104044

Comment thread resources_servers/nemo_user_sim/app.py Resolved
oyilmaz-nvidia and others added 2 commits November 5, 2026 11:10
The Assistant path and the masked-failure path both read failure_detail, then failure_reason, from the
simulation outcome. Read it through one helper so the two stay in step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test 73ee7fa

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 4e0f380

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 4e0f380

@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

/ok to test 4e0f380

@oyilmaz-nvidia, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 7a405c2

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42078814239

Merged via the queue into main with commit 390c954 Nov 5, 2026
61 checks passed
@oyilmaz-nvidia
oyilmaz-nvidia deleted the yuhez/4882-masked-failure-metadata-2 branch November 5, 2026 18:07

This branch was previously deployed

1 inactive deployment
public — 7a405c23 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10394
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants