Repository navigation
fix(resources): name the failure kind on masked osworld and chemistry results - #5223
Conversation
… results osworld, chemcotbench and chemreason_bench already mask their failed results but left failure_kind empty, so a consumer could not tell why a row was masked without reading server-specific fields. Name the kind on those paths and keep rewards, masks, statuses and the existing fields unchanged: - osworld: an evaluation that raises is a verifier_error with a bounded reason; a session the server no longer holds at verify is session_lost, through the shared lost_session_verification helper. - chemcotbench: scorer timeouts, worker failures and invalid scorer results are verifier_error; disabled optimization scoring is chemcotbench:scoring_disabled, so it is not reported as a failed scorer. - chemreason_bench: rows it cannot score (bad task type, bad ground truth, no lm protocol) are chemreason_bench:invalid_task, with a reason. Part of #4882. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
|
/ok to test 636c1cd |
…rries OSWorld's verify request accepts extra fields, so a request that already carried failure_kind or failure_reason passed it twice and raised TypeError instead of returning the masked result. Merge the failure fields over the request instead, and test both failure paths with such a request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
|
/ok to test 421e476 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found.
Reviewed all seven changed files and traced the affected request models, scoring/session failure paths, shared failure-kind contract, collection, and aggregation. The added diagnostics preserve existing rewards, masks, and statuses. OSWorld’s failure-field merging also handles request extras without duplicate keyword arguments. The changed tests cover the named failure paths and distinguish them from valid scored or wrong answers.
Static inspection only; tests and rollouts were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
|
/ok to test aed4270 |
oyilmaz-nvidia
left a comment
There was a problem hiding this comment.
Approving with a comment.
| **lost_session_verification(f"OSWorld session {session_id} was not found at verify"), | ||
| } | ||
| ) | ||
| except Exception as error: |
There was a problem hiding this comment.
Nit: the bare except Exception used to swallow the error silently, and now it only puts a truncated string in failure_reason. A LOG.exception(...) would keep the traceback in the server logs. The repo guidance says broad catches should log, translate or preserve the error, and the traceback is lost here. If we don't need the traceback, it's Ok then.
|
/ok to test 069aa65 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42063119056 |
What does this PR do?
First slice of #4882 (parent #3180). Three resources servers already mask their failed results, but leave
failure_kindempty, so a consumer cannot tell why a row was masked without reading server-specific fields. This PR names the kind on those paths. Rewards, masks, statuses and the existing server-specific fields stay as they are.failure_kindaddedverifier_error, plus a boundedfailure_reasonwith the exception type and messageSessionNotFoundError)session_lost, through the sharedlost_session_verificationhelperverifier_error(scoring_errorandfailure_reasonkept)enable_molopt: falsewith an optimization taskchemcotbench:scoring_disabled, so disabled scoring is not reported as a failed scorerbad_task_type,bad_ground_truth,no_lm_protocolchemreason_bench:invalid_task, plus afailure_reason;statusstill carries the detailThe two server-prefixed kinds use the
<server>:<kind>formnemo_gym.failure_kindsprovides for conditions the shared vocabulary should not grow a name for. Rollout collection readsfailure_kindonly on failures-sidecar rows, so on these completed, masked results it is a diagnostic field: routing, scoring, retries and health counts do not change.Left for later #4882 slices:
Validation
Against the final diff, Python 3.13.14:
verifyfor a scored task (no kind), an evaluation that raises (verifier_errorand its reason) and a missing session (session_lost), and check the session is closed in each case. OSWorld's verify request accepts extra fields, so both failure paths are also tested with a request that already carriesfailure_kindandfailure_reason; the failure fields replace them.verifier_erroron the timeout, worker and invalid-result paths andchemcotbench:scoring_disabledon disabled optimization; empty answers and scored results keep no kind.Rollouts
N/A. Only diagnostic fields on already-masked failure paths change; rewards, masks and the scored path are untouched, and the tests above call
verifydirectly on each changed path.Compatibility and benchmark impact
None for scores. Masked rows from these servers now carry a
failure_kind, so tools that group failures by kind will start counting them.Checklist