Repository navigation
fix(critpt): allow terminal zero scores for comparator uncertainty - #5416
Conversation
Signed-off-by: Igor Gitman <igitman@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
One BLOCKER: the opt-in projection can turn reference-preflight uncertainty into a final scored zero, even though the candidate was never evaluated.
Completed static review of all three changed files and the affected grading, serialization, aggregation, persistence, and resume paths. Tests were not executed; the reported model evaluation was not independently reproduced.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
The revised implementation has now also passed a real end-to-end re-verification using saved model answers. Policy responses were preserved exactly, eligible candidate-comparison uncertainty retained its explicit provenance and was scored as zero, and the run completed successfully. This validates the updated implementation, in addition to the unit/regression coverage described above. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Merged upstream main 1,828 CritPt tests pass, with 1 skip. Verified that the provenance guard still excludes reference/preflight uncertainty, runtime errors and invalid expectations while allowing the explicitly supported completed candidate uncertainty. The previously reported live re-verification validates the revised provenance handling; the old “in progress” wording has been removed.
|
|
/ok to test 4105fd5 |
|
/review |
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks. The opt-in is narrow where it should be:
- The execution layer decides eligibility: every uncertain verdict must be comparator-side and listed in
CANDIDATE_UNCERTAINTY_REASONS. _projectthen re-checks full case coverage, a completed response and the category.- By default these rows stay unscored.
This path returns a GradeResult directly instead of raising Unscorable. I checked that grade() still fills in cleanup and sandboxes for it, so both paths carry the same observation and cleanup fields.
One readability nit inline. CI is green apart from the Autobot check.
|
🤖 On it — handling this as babysit-ready-to-merge. |
|
I'll watch this ready-to-merge PR and report substantive progress. Running on |
|
🤖 On it — handling this as babysit-ready-to-merge. |
|
💬 I couldn’t inspect the failing checks: the GitHub broker returned HTTP 503 (“native Git authority is unavailable”) on both the initial read and its bounded retry. The CI failure at |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Addressed in |
|
/ok to test 18bf3e3 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42089323245 |
|
💬 The failing KiloCode probe turns a controlled model HTTP 400 into an unhandled HTTP 500, losing the expected failure metadata. Current main passes that scenario and has newer error-handling code. I’m verifying the existing fix before deciding whether to incorporate it; I haven’t requested a CI rerun. |
|
💬 The main CI gate, unit-test aggregate, and lint passed at |
|
The PR is no longer an open ready-to-merge PR targeting main. |
Merge main at 330c461 into the agentic CritPt branch. Where main has since added its own handling for something this branch also changed, the merge keeps main's version: - responses_api_models/vllm_model is identical to main. The exact completion fitting is dropped; main's refit_output_budget_to_prompt (NVIDIA-NeMo#5864) covers what the agentic recipe needs. - Stirrup compaction failure uses main's handling (NVIDIA-NeMo#5822) and its max_turns stop reason (NVIDIA-NeMo#5937). In finish_tool mode the grader scores an agent_invalid_output response zero as candidate_no_answer. - Context-overflow recovery becomes the opt-in Stirrup option context_overflow_recovery (default false). The runner reports the model server's own rejection. - The grader's execution sessions use ResourcesSessionLocks and ResourcesSessionFences (NVIDIA-NeMo#5083); agentic_execution refuses several workers (NVIDIA-NeMo#4790). - A semantic validator failure or timeout stays unscorable with score_comparator_uncertain_as_incorrect (NVIDIA-NeMo#5416). - nemo_gym/ is unchanged. Two fixes found while validating the merge: - Since NVIDIA-NeMo#5128 the single_agent_turn environment servers and the agents that verify through verify_agent_response send failure_report with every /verify request. The grader read it as an unknown task field and answered task_invalid for every such rollout; it is now a framework field. - tests/test_runner_remote.py built ExecutionPolicy with four arguments the policy does not have and expected a 2 GiB address-space limit it never set, so nine of its tests failed; both are corrected. Also included: fixes from a review of the merged diff (session glossary wording, main's lost-session body on /code_exec, tighter execution limits, a stopped session releasing its capacity slot, route tests, and a local end-to-end grade of a semantic-validator task). Signed-off-by: Erik Arakelyan <erikarakelyan1997@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completed candidate comparisons can be inconclusive even after successful reference validation. Add
score_comparator_uncertain_as_incorrect(default false) so evaluation protocols can count these answers as zero while preserving the uncertainty category and avoiding regeneration on resume.Eligibility is established by the execution layer: successful reference preflight, completed candidate execution/comparison, and explicit symbolic/unordered undecidability or per-case proof-budget reasons.
candidate_uncertainty_reasonscarries that provenance into the response. Preflight uncertainty, invalid expectations, comparator/runtime errors, unsupported observations, and unknown reasons remain unscored. Existing decided-failure precedence and whole candidate-comparison timeouts are unchanged. This is a protocol option; record it when comparing scores.Validation:
/verifyintegration regressions use the real Grader and scripted provider for both option settings, covering reference and candidate failure paths plus accepted and rejected uncertainty reasons.pre-commit run --all-filespassed; commits are DCO signed.No separate issue is necessary for this focused opt-in behavior. No benchmark datasets or rollout artifacts are included.