Skip to content

fix(resources): name the failure kind on masked osworld and chemistry results - #5223

Merged
oyilmaz-nvidia merged 4 commits into
mainfrom
yuhez/4882-masked-failure-metadata
Nov 5, 2026
Merged

oyilmaz-nvidia merged 4 commits into
mainfrom
yuhez/4882-masked-failure-metadata

Conversation

@yuhezhang-ai

@yuhezhang-ai yuhezhang-ai commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

First slice of #4882 (parent #3180). Three resources servers already mask their failed results, but leave failure_kind empty, so a consumer cannot tell why a row was masked without reading server-specific fields. This PR names the kind on those paths. Rewards, masks, statuses and the existing server-specific fields stay as they are.

Server Masked path failure_kind added
osworld evaluation raised (previously masked with no reason) verifier_error, plus a bounded failure_reason with the exception type and message
osworld session no longer held at verify (SessionNotFoundError) session_lost, through the shared lost_session_verification helper
chemcotbench scorer timeout, worker failure, invalid scorer result verifier_error (scoring_error and failure_reason kept)
chemcotbench enable_molopt: false with an optimization task chemcotbench:scoring_disabled, so disabled scoring is not reported as a failed scorer
chemreason_bench bad_task_type, bad_ground_truth, no_lm_protocol chemreason_bench:invalid_task, plus a failure_reason; status still carries the detail

The two server-prefixed kinds use the <server>:<kind> form nemo_gym.failure_kinds provides for conditions the shared vocabulary should not grow a name for. Rollout collection reads failure_kind only on failures-sidecar rows, so on these completed, masked results it is a diagnostic field: routing, scoring, retries and health counts do not change.

Left for later #4882 slices:

  • conversational_tool_use_simulation: its mask covers several judge verdicts (user, tool or verification failure) and message-generation errors, so choosing kinds needs its own decision.
  • swe_agents, anyswe_agent, anyterminal_agent and osworld_agent: open PRs are changing their failure paths.

Validation

Against the final diff, Python 3.13.14:

pytest resources_servers/osworld/tests
(cd resources_servers/chemreason_bench && pytest tests)
gym env test --resources-server chemcotbench
pre-commit run --files <changed files>
  • osworld: 18 passed. New tests call verify for a scored task (no kind), an evaluation that raises (verifier_error and its reason) and a missing session (session_lost), and check the session is closed in each case. OSWorld's verify request accepts extra fields, so both failure paths are also tested with a request that already carries failure_kind and failure_reason; the failure fields replace them.
  • chemreason_bench: 232 passed. The masked-row test now checks the kind and reason on all three invalid-task paths; a scored row and a wrong answer keep no kind.
  • chemcotbench, in its own server environment: 80 passed, including the upstream verifier fixture. Failure tests check verifier_error on the timeout, worker and invalid-result paths and chemcotbench:scoring_disabled on disabled optimization; empty answers and scored results keep no kind.
  • Pre-commit passed on the changed files.

Rollouts

N/A. Only diagnostic fields on already-masked failure paths change; rewards, masks and the scored path are untouched, and the tests above call verify directly on each changed path.

Compatibility and benchmark impact

None for scores. Masked rows from these servers now carry a failure_kind, so tools that group failures by kind will start counting them.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated edits are excluded.
  • Tests added or updated and pass locally.
  • Documentation updated (chemcotbench README).
  • Pre-commit checks pass locally on the changed files.
  • New source files include the required Apache-2.0 SPDX header — N/A, no new source files.
  • All commits have DCO sign-off.

… results

osworld, chemcotbench and chemreason_bench already mask their failed results but left failure_kind empty,
so a consumer could not tell why a row was masked without reading server-specific fields. Name the kind on
those paths and keep rewards, masks, statuses and the existing fields unchanged:

- osworld: an evaluation that raises is a verifier_error with a bounded reason; a session the server no
  longer holds at verify is session_lost, through the shared lost_session_verification helper.
- chemcotbench: scorer timeouts, worker failures and invalid scorer results are verifier_error; disabled
  optimization scoring is chemcotbench:scoring_disabled, so it is not reported as a failed scorer.
- chemreason_bench: rows it cannot score (bad task type, bad ground truth, no lm protocol) are
  chemreason_bench:invalid_task, with a reason.

Part of #4882.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working labels Oct 28, 2026
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test 636c1cd

…rries

OSWorld's verify request accepts extra fields, so a request that already carried failure_kind or
failure_reason passed it twice and raised TypeError instead of returning the masked result. Merge the
failure fields over the request instead, and test both failure paths with such a request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test 421e476

@yuhezhang-ai
yuhezhang-ai marked this pull request as ready for review October 29, 2026 14:02
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 29, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

Reviewed all seven changed files and traced the affected request models, scoring/session failure paths, shared failure-kind contract, collection, and aggregation. The added diagnostics preserve existing rewards, masks, and statuses. OSWorld’s failure-field merging also handles request extras without duplicate keyword arguments. The changed tests cover the named failure paths and distinguish them from valid scored or wrong answers.

Static inspection only; tests and rollouts were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test aed4270

@oyilmaz-nvidia oyilmaz-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with a comment.

**lost_session_verification(f"OSWorld session {session_id} was not found at verify"),
}
)
except Exception as error:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the bare except Exception used to swallow the error silently, and now it only puts a truncated string in failure_reason. A LOG.exception(...) would keep the traceback in the server logs. The repo guidance says broad catches should log, translate or preserve the error, and the traceback is lost here. If we don't need the traceback, it's Ok then.

@oyilmaz-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 069aa65

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42063119056

Merged via the queue into main with commit c68feee Nov 5, 2026
61 checks passed
@oyilmaz-nvidia
oyilmaz-nvidia deleted the yuhez/4882-masked-failure-metadata branch November 5, 2026 16:34

This branch was previously deployed

1 inactive deployment
public — 069aa65c Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10340
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants