Skip to content

fix(critpt): allow terminal zero scores for comparator uncertainty - #5416

Merged
adil-a merged 4 commits into
mainfrom
codex/critpt-uncertainty-public
Nov 5, 2026
Merged

adil-a merged 4 commits into
mainfrom
codex/critpt-uncertainty-public

Conversation

@Kipok

@Kipok Kipok commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Completed candidate comparisons can be inconclusive even after successful reference validation. Add score_comparator_uncertain_as_incorrect (default false) so evaluation protocols can count these answers as zero while preserving the uncertainty category and avoiding regeneration on resume.

Eligibility is established by the execution layer: successful reference preflight, completed candidate execution/comparison, and explicit symbolic/unordered undecidability or per-case proof-budget reasons. candidate_uncertainty_reasons carries that provenance into the response. Preflight uncertainty, invalid expectations, comparator/runtime errors, unsupported observations, and unknown reasons remain unscored. Existing decided-failure precedence and whole candidate-comparison timeouts are unchanged. This is a protocol option; record it when comparing scores.

Validation:

  • Full grader suite on the revised implementation: 1,828 passed, 1 skipped.
  • /verify integration regressions use the real Grader and scripted provider for both option settings, covering reference and candidate failure paths plus accepted and rejected uncertainty reasons.
  • The original PR head fails the zero-attempt provenance regression.
  • pre-commit run --all-files passed; commits are DCO signed.
  • The revised provenance handling passed real end-to-end re-verification using saved model answers: policy responses were preserved exactly, eligible candidate uncertainty retained its provenance and was scored as zero, and the run completed successfully (see the validation comment).

No separate issue is necessary for this focused opt-in behavior. No benchmark datasets or rollout artifacts are included.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Nov 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

One BLOCKER: the opt-in projection can turn reference-preflight uncertainty into a final scored zero, even though the candidate was never evaluated.

Completed static review of all three changed files and the affected grading, serialization, aggregation, persistence, and resume paths. Tests were not executed; the reported model evaluation was not independently reproduced.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

The revised implementation has now also passed a real end-to-end re-verification using saved model answers. Policy responses were preserved exactly, eligible candidate-comparison uncertainty retained its explicit provenance and was scored as zero, and the run completed successfully. This validates the updated implementation, in addition to the unit/regression coverage described above.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Merged upstream main 483674910 in 4105fd5b0af0de5490e75860ed395f2da8d05b72 and reconciled the package-layout changes.

1,828 CritPt tests pass, with 1 skip. Verified that the provenance guard still excludes reference/preflight uncertainty, runtime errors and invalid expectations while allowing the explicitly supported completed candidate uncertainty. The previously reported live re-verification validates the revised provenance handling; the old “in progress” wording has been removed.

pre-commit run --all-files, diff whitespace checks and 724 public-API/import-boundary tests pass. This update merges main and adapts imports; no new model rollout or full local coverage run is claimed. Trusted CI needs vetter authorization for the new head.

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/ok to test 4105fd5

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/review

yaoyu-33
yaoyu-33 previously approved these changes Nov 4, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The opt-in is narrow where it should be:

  • The execution layer decides eligibility: every uncertain verdict must be comparator-side and listed in CANDIDATE_UNCERTAINTY_REASONS.
  • _project then re-checks full case coverage, a completed response and the category.
  • By default these rows stay unscored.

This path returns a GradeResult directly instead of raising Unscorable. I checked that grade() still fills in cleanup and sandboxes for it, so both paths carry the same observation and cleanup fields.

One readability nit inline. CI is green apart from the Autobot check.

@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and waiting for final CI or merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

Copy link
Copy Markdown
Contributor

I'll watch this ready-to-merge PR and report substantive progress. Running on nim-gpt-6-1-sol.

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

Copy link
Copy Markdown
Contributor

💬 I couldn’t inspect the failing checks: the GitHub broker returned HTTP 503 (“native Git authority is unavailable”) on both the initial read and its bounded retry. The CI failure at 4105fd5b remains unclassified, so I haven’t requested a rerun or changed code. The PR watch remains attached; diagnosis needs the broker restored.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 18bf3e367: the candidate-uncertainty GradeResult now uses explicit keyword arguments for every field. No scoring change. Full grader suite: 1,828 passed, 1 skipped; pre-commit and whitespace checks pass. New head needs trusted-CI authorization.

@adil-a

adil-a commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/ok to test 18bf3e3

@adil-a
adil-a enabled auto-merge November 5, 2026 18:49
@adil-a
adil-a added this pull request to the merge queue Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42089323245

Copy link
Copy Markdown
Contributor

💬 The failing KiloCode probe turns a controlled model HTTP 400 into an unhandled HTTP 500, losing the expected failure metadata. Current main passes that scenario and has newer error-handling code. I’m verifying the existing fix before deciding whether to incorporate it; I haven’t requested a CI rerun.

Copy link
Copy Markdown
Contributor

💬 The main CI gate, unit-test aggregate, and lint passed at 18bf3e36. Only the optional KiloCode conformance checks remain red. Main already contains the corresponding error-handling fix from #5377, and its probe passes. I’m leaving the branch unchanged and watching for merge.

Merged via the queue into main with commit 4f2e967 Nov 5, 2026
62 of 65 checks passed
@adil-a
adil-a deleted the codex/critpt-uncertainty-public branch November 5, 2026 19:15

Copy link
Copy Markdown
Contributor

The PR is no longer an open ready-to-merge PR targeting main.

osoblanco added a commit to osoblanco/Gym that referenced this pull request Nov 15, 2026
Merge main at 330c461 into the agentic CritPt branch. Where main has
since added its own handling for something this branch also changed,
the merge keeps main's version:

- responses_api_models/vllm_model is identical to main. The exact
  completion fitting is dropped; main's refit_output_budget_to_prompt
  (NVIDIA-NeMo#5864) covers what the agentic recipe needs.
- Stirrup compaction failure uses main's handling (NVIDIA-NeMo#5822) and its
  max_turns stop reason (NVIDIA-NeMo#5937). In finish_tool mode the grader scores
  an agent_invalid_output response zero as candidate_no_answer.
- Context-overflow recovery becomes the opt-in Stirrup option
  context_overflow_recovery (default false). The runner reports the
  model server's own rejection.
- The grader's execution sessions use ResourcesSessionLocks and
  ResourcesSessionFences (NVIDIA-NeMo#5083); agentic_execution refuses several
  workers (NVIDIA-NeMo#4790).
- A semantic validator failure or timeout stays unscorable with
  score_comparator_uncertain_as_incorrect (NVIDIA-NeMo#5416).
- nemo_gym/ is unchanged.

Two fixes found while validating the merge:

- Since NVIDIA-NeMo#5128 the single_agent_turn environment servers and the agents
  that verify through verify_agent_response send failure_report with
  every /verify request. The grader read it as an unknown task field
  and answered task_invalid for every such rollout; it is now a
  framework field.
- tests/test_runner_remote.py built ExecutionPolicy with four arguments
  the policy does not have and expected a 2 GiB address-space limit it
  never set, so nine of its tests failed; both are corrected.

Also included: fixes from a review of the merged diff (session glossary
wording, main's lost-session body on /code_exec, tighter execution
limits, a stopped session releasing its capacity slot, route tests, and
a local end-to-end grade of a semantic-validator task).

Signed-off-by: Erik Arakelyan <erikarakelyan1997@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — 18bf3e36 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10418
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working ready-to-merge PR is approved, current, and waiting for final CI or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants