Repository navigation
fix(gdpval): count a model that delivers nothing as a lost battle, not a masked row - #6077
Conversation
…t a masked row When the scored model ran out of wall time (or stopped on its own context or turn limit) with no deliverable, GDPVal masked the row as agent_timeout: it left the denominator and the pairwise Elo fit, so a slow or failing model scored higher the more often it failed. In comparison mode such a row is now a success row that holds a battle the model lost against every active reference, with its failure_kind and model attribution; the fit keeps it. In rubric mode and in aa_briefcase_lite it is an unmasked zero with the same kind. An agent timeout carries no outcome action, so recovery still reruns one whose window held a server-side fault. Judge failures, transport and missing-reference faults are unchanged. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-6077-83226be0cc64.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — the new timeout rows can bias multistage scoring, and AA’s reported rates still exclude them.
I reviewed all five changed files and traced their affected verification, aggregation, persistence, recovery, and cross-stage reuse paths. The inline findings describe the correctness issues. This was static inspection; I didn’t execute tests or independently verify the reported test results.
RISK — Required benchmark validation is missing
The PR explicitly reports no live rollout or reward-profiling run. This changes scoring for the fixed GDPVal benchmark and its AA consumer. The pinned rubric requires a before/after reward comparison, and the repository quality bar requires representative real model rollouts for behavior-changing environment work. The synthetic win-rate fixture establishes arithmetic, not the end-to-end reward distribution.
Could you add representative rollout artifacts and before/after numbers using the same model, tasks, references, judge settings, and budgets? Include GDPVal win rate/Elo, scalar rewards, model-stop frequencies, and masked/scored counts; include AA’s reported rates for the shared timeout path. The expected change is confined to the intended model failures: ordinary delivered results and judge/reference faults must retain their handling. Acceptance is linked run evidence showing that separation and explaining the observed score shift. No production-code patch is needed for this evidence request.
NOTE — Update the documented stage-completion rule
The GDPVal benchmark guide still says an answered timeout holds no battle and makes a strict stage fail. That now contradicts the changed verifier and multistage test. Could you replace that paragraph with the clean-timeout counted-loss rule and note that results using the earlier masking policy aren’t comparable? Preserve the separate explanations for infrastructure failures, missing references, and partial-completion policies. Static acceptance: the guide should agree with the clean-timeout regression test and no longer recommend a partial-completion policy merely to accommodate such a row.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Inline comments (escaped v1):
-
resources_servers/gdpval/multistage_orchestrator.pyline 267: BLOCKER — Infrastructure timeouts now enter calibration as model losses.Removing
_ng_failure_classfrom GDPVal’s timeout response makes_is_success_rowaccept it even when the sharedis_infrastructure_agent_timeoutclassifier returns true. A concrete trigger is awall_timeresult whose last captured model call returned HTTP 503. The retry predicate then sees no failure class, so its imputed losses feed calibration instead of being retried. Resume selection also gates the persisted success permanently. Before this change the same timeout carried classagent_timeoutand remained retryable.Single-pass collection already handles this success shape in RolloutRecords._payload, but multistage bypasses that selector. Could you normalize infrastructure-attributed timeout results into excluded, retryable failures before routing, retry selection, and pooling? Apply equivalent classification when loading saved rows, reopening dependent stages through the existing restart machinery. The invariant is that an infrastructure-cut execution window contributes no model-loss evidence. This spans several paths, so a one-line predicate change isn’t sufficient.
Preserve clean timeouts as counted losses, the attempt cap and recorded reference assignment, and judge-only recovery without regenerating its saved answer.
Proposed acceptance: Extend the real-verifier multistage fixture with a final captured call carrying
response_metadata.status_code=503. With retries enabled, exclude that attempt’s losses and use the subsequent successful attempt. With retries disabled, persist and resume it; the affected stage must reopen. Retain the clean-timeout assertions: one attempt, counted losses, and the next stage planned. These checks weren’t executed. -
resources_servers/gdpval/multistage_orchestrator.pyline 1619: BLOCKER — A later stage regenerates a clean timeout instead of retaining its loss.The new imputed-loss success reaches this loop and is recorded as a produced deliverable, although a wall-time stop persists no files. If the task/repeat recurs with default
reuse_cached_deliverables=True, build_stage_rows marks it for reuse. The real cached resolver returns 404, and run_rollouts runs the policy again. A successful second generation can therefore replace the final model failure in the headline stage.The changed test misses this: its fake runner recognizes task index
0, while later stages offset that index, and it never exercises cached verification’s 404 fallback.Could you retain final no-deliverable model outcomes separately from reusable file artifacts, keyed by task and repeat, and re-impute their losses against later stages’ assigned references without generating again? Cover timeout/context/turn-limit outcomes when cross-stage reuse is enabled, including file-backed resume. Preserve ordinary cached-file judging, the fallback for a genuinely missing artifact with no saved final model outcome, and explicit
reuse_cached_deliverables=False. The cache representation needs a deliberate choice, so I haven’t proposed a small patch.Proposed acceptance: Exercise two overlapping stages through
run_e2e_multistageand the real cached resolver, with three ordinary results and one clean timeout in stage 0. Make the policy mock succeed if called again for that task. Assert one generation for that task/repeat and losses in both stages against their respective references, including after resume. Retain the existing missing-cache fallback for the distinct case without a recorded final model failure. Tests weren’t executed. -
resources_servers/aa_briefcase_lite/app.pyline 924: RISK — AA’s reported rates still exclude the newly scored timeout.Changing
_AGENT_TIMEOUTto an unmasked zero makes this early return a scored row, butAABriefcaseLiteVerifyResponsedefaultsrubric_totaland every pairwise count to zero. AA aggregation computes its rates from those counts, notreward. With the existing two-binary-check/two-reference fixture, one fully passing row plus onewall_timerow still reports both rates as1.0, rather than0.5. These are the rates users are told to report, so the scalar zero doesn’t fix the reported benchmark scores.Could the unmasked timeout branch produce applicable failed-check and lost-trial counts, consistently with AA’s existing no-submission behavior? Preserve the timeout kind, attribution, and zero reward without judging partial files. Keep binary-only runs free of pairwise rates, and retain skipped-reference/excluded-judge rules and masked evaluation faults. Calling
_grade(body)unchanged would grade partial files, so it isn’t a complete correction; the helper extraction needs author verification. The invariant is that each scored model zero contributes to every applicable reported rate’s denominator.Proposed acceptance: Aggregate a fully passing
allrow and the new timeout fixture; expect binary counts2/4, pairwise counts4 wins, 4 losses, both rates0.5, and no judge call for the timeout. Retain the existing no-submission case, binary-only absence of the pairwise rate, and masked missing-directory/reference faults. Static reasoning only; tests weren’t executed. -
resources_servers/gdpval/app.pyline 1369: RISK — Live context/turn-limit runs with no files miss the new loss branch.A seeded run that never calls
finishstill gets a persisted directory containingfinish_params.json = nulland reference inputs: collect_deliverables always writes that marker, as the session test confirms.task_attempted()therefore returns true even though no deliverable exists. Formax_turns,max_iterations, orcontext_limit,_model_stop_failureis never reached and the empty submission goes to the judge. If the judge is unavailable, this becomes a masked judge failure instead of the intended counted model loss. The new tests only use a nonexistent eval directory, bypassing normal session collection.Could you distinguish an attempted directory from one containing agent-produced files? A small replacement for the opening gate is:
model_stop = _model_stop_failure(body) if ( eval_task_dir is None or not task_attempted(str(eval_task_dir)) or ( model_stop is not None and not any(is_deliverable(path) for path in eval_task_dir.iterdir()) ) ):
Keep the existing branch body, but remove its later duplicate
model_stop = ...assignment. This uses the shared deliverable classifier and the collector’s flat output layout. Verify the suggestion before adopting it.Preserve judging when a stopped run has actual submitted files. Also preserve ordinary
null/{}finish-marker judging without a model-stop reason, explicit stage-one missing-eval imputation, missing-reference handling, and the lost-session guard.Proposed acceptance: Parameterize the three stop reasons through a live seeded comparison session with no
finishcall; make any judge call fail the test and expect typed, unmasked losses. Repeat with a submittedreport.mdand expect normal judging. Retaintest_existing_finish_marker_is_judged_with_loss_policyfor its no-stop-reason cases. Tests weren’t executed. -
resources_servers/gdpval/app.pyline 665: RISK — A timeout becomes scored even when judging is explicitly disabled.With
judge_deliverables=Falseandresponse.metadata={"stop_reason":"wall_time"}, this return bypasses the judging-disabled branch. Previously_AGENT_TIMEOUTsuppliedmask_sample=True; now GDPVal rubric and AA return an unmasked zero, while GDPVal comparison constructs counted losses. This breaks the masked, collection-only contract: a run with scoring disabled can acquire scored zero rows solely because it timed out.Could the timeout return preserve masking when judging is off? The smallest correction here is:
return body, {"reward": 0.0, "mask_sample": not server.config.judge_deliverables, **_AGENT_TIMEOUT}This retains the existing timeout behavior of not collecting or judging partial work, keeps judging-enabled timeouts unmasked, and prevents the comparison caller from manufacturing losses in collection-only mode. It doesn’t change the pre-existing timeout persistence policy. Verify the suggestion before adopting it.
Proposed acceptance: Parameterize GDPVal rubric/comparison and AA with judging enabled/disabled and
wall_time. Enabled must retain the new zero/loss behavior; disabled must be masked, call no judge, and contain no imputed comparison loss. Retain the ordinary judging-disabled test’s persisted-file andgdpval:judging_disabledassertions, plus the timeout session test’s close/cleanup assertions. Tests weren’t executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…ow writes GDPVal no longer writes the _ng_failure_class=agent_timeout failure: a model that ran out of wall time is a scored loss that keeps failure_kind=agent_timeout. The legacy timeout signal registry named the removed line, so its producer check failed. The entry now names the failure_kind form GDPVal writes, so health still counts those rows as timed out. test_run_timing covers the same two forms. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/ok to test 83226be |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43890914423 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43902912773 |
What does this PR do?
When the scored model delivers nothing (it runs out of wall time, or stops on its own context or turn limit), GDPVal masked the row: it left the denominator and the pairwise Elo fit, so a model scored higher the more often it failed. This is a GDPVal scoring change.
gdpval): anagent_timeoutrow (stop_reason=wall_time) is now a success row holding a battle the model lost against every active reference (num_comparison_trialslosses per reference repeat,judge_response.manual_imputation=model_failure_as_loss). It stays in the denominator and in the Elo fit, withfailure_kind=agent_timeoutandfailure_outcome_attribution=model. The same holds when the run stopped oncontext_limit,max_iterationsormax_turnsand there is no deliverable (model_context_limit/agent_limit,failure_outcome_action=count_as_zero). The existing multistage coverage gate no longer blocks a stage because of such a row.aa_briefcase_lite(which shareresolve_deliverables; it has no Elo fit): anagent_timeoutis an unmasked zero with the same kind and attribution.judge_failed, permanent judge errors, transport ineligibility), missing reference files, a lost session, and a missing deliverable with no model stop reason keep their masked/terminal handling and retry or regrade; none is ever scored as a loss.agent_timeoutcarries no outcome action, so recovery still reruns a timeout whose window held a recorded server-side fault and treats a clean one as the model's final result.Validation
pytest resources_servers/gdpval/tests resources_servers/aa_briefcase_lite/tests: 1076 passed, 1 skipped. Without the app change the new and updated tests fail (7 failures).agent_timeoutis a lost battle with no judge call; context and turn limits are counted losses; a missing deliverable without a model stop stays a terminal failure; a no-deliverable row lowers the win rate (0.75 to 0.5625 in the fixture) and the Elo versus the same run with the row masked; the multistage strict stage now completes with a timeout row and plans the next stage; theaa_briefcase_litetimeout is an unmasked zero.pre-commit run --from-ref origin/main --to-ref HEADpasses.Rollouts
N/A: unit tests over
verifyandaggregate_metrics; no live run.Compatibility and benchmark impact
Scores go down for models that time out or fail: each such task now counts as a loss (comparison mode) or a zero (rubric, AA-Briefcase-Lite) instead of being dropped, so slow or failing models score lower, and the drop grows with how often they fail. Earlier results that masked these rows are not comparable with new ones. The
comparison/imputed_loss_*metrics now include these rows.Checklist