Skip to content

feat(genrm_compare): mask only the two rollouts of a failed judge comparison - #5330

Open
yfw wants to merge 8 commits into
mainfrom
yifuw/genrm-drop-failed-comparisons
Open

yfw wants to merge 8 commits into
mainfrom
yifuw/genrm-drop-failed-comparisons

Conversation

@yfw

@yfw yfw commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Since #3351, one failed judge comparison fails the whole cohort. A failed comparison is one with no completed answer, or a judge HTTP request that fails (an HTTP error, a connection error or a reply that is not JSON, after any retries). All of the cohort's rollouts are then masked, even when every other comparison succeeded. With 16 rollouts and a long judge answer, one comparison that exhausts max_output_tokens on both attempts discards 15 good comparisons.

This PR masks only the two rollouts of a failed comparison, and scores every other rollout from all of its comparisons. There is no new configuration key.

  • Masking: both rollouts of a comparison that raises a JudgeError are masked through judge_failsafe (reward 0.0, mask_sample, failure_kind: judge_failed). Under circular, one failed comparison (5, 6) masks rollouts 5 and 6, not all 16. Under all_pairs, or with several judge passes, a rollout with any failed comparison is masked.
  • Scoring: every other rollout is scored from all of its comparisons, including those against a masked rollout, which still measure it. So every scored rollout's reward averages the same number of comparisons. The group's length and style adjustments see only the scored rollouts.
  • Failing the group: only when every rollout is masked, for example when the judge is down, does the whole cohort fail as a judge failure, as today. Its reason says that every rollout was masked. The server now waits for every comparison to finish or fail, within cohort_evaluation_timeout_s.
  • Unchanged:
    • An error outside the judge request still fails a /verify group with HTTP 503 at once, and cancels the comparisons still running.
    • A completed but malformed answer keeps the default-score fallback.
    • cohort_evaluation_timeout_s still bounds the group. A judge_request_timeout_s expiry fails a comparison only if it comes before that deadline; both default to 1,800 seconds, so by default the group deadline fires first and the group fails.
    • /compare has no per-response mask, so its first failed comparison still returns 503 at once and cancels the comparisons still running.
  • Other paths: the fixed_baseline path follows the same rule: a rollout with a failed judge pass against the baseline is masked.
  • Checkpoints: a member masked alone is saved and restored as masked (masked_alone in its saved state); its siblings replay their rewards.
  • Failure reports: a masked member's failure has its failed comparison's Gym failure, such as the judge's HTTP error, as its cause, also after a checkpoint restore. The scored members' reports list that failure without selecting it. When every rollout is masked, the group's failure has the first failed comparison as its cause.
  • Late requests: a regenerated answer for a masked member gets its recorded judge failure, as a failed group's members do. So a --resume that reruns a masked row records it as masked again, instead of getting HTTP 409 and stopping the collection.
  • Metrics and logs: new genrm_failed_comparison_rate_per_group and genrm_masked_rollout_rate_per_group metrics. Each failed comparison is logged when it fails, and each masked rollout is logged with its reason.
  • Why both rollouts, not a score from the comparison that is left: in a group-relative trainer, that would mix rewards averaged over two comparisons with rewards from one. Under circular, the comparison that is left also shows one rollout only as response_1 and the other only as response_2, so a judge that favors one position moves their scores in opposite directions.

Consumer requirement, and merge order. A masked rollout carries a placeholder reward of 0.0. A trainer must keep it out of the group's baseline and normalization, or it shifts its siblings' advantages. In NeMo RL that is grpo.masked_reward_policy: exclude_infra (NVIDIA-NeMo/RL#4968; judge_failed is not a policy failure kind) or exclude. Under NeMo RL's default, include, the placeholder counts. NeMo RL also needs env.should_mask_flagged_samples on, its default; with it off, NeMo RL trains the masked rows on their 0.0 rewards. So this PR should merge after NVIDIA-NeMo/RL#4968, and GenRM recipes should set exclude_infra when they move to a Gym pin that has it.

Validation

Against the final diff (merged with main at 59b71b404), Python 3.13.14:

gym env test +entrypoint=resources_servers/genrm_compare +should_validate_data=true   # 288 passed
pre-commit run --all-files                                                          # passed

New tests in tests/test_failed_comparisons.py, tests/test_checkpoint.py and tests/test_http_lifecycle.py:

  • Masking, 16-rollout circular group:
    • One failed comparison: its two rollouts are masked, and the other 14 are scored with their own scores.
    • The neighbors of a masked rollout are still scored from both of their comparisons; the judge stub gives each position a different score, so a lost comparison would change their rewards.
    • Two non-adjacent failures: their four rollouts are masked.
  • Group of 4:
    • Two adjacent failures mask three rollouts. An exact retry replays the same outcomes without judging again.
    • The masked rollouts do not change the length adjustments of the scored rollouts.
    • all_pairs: one failed pair masks its two rollouts, and the other two are scored from all three comparisons; when every rollout is masked, the group fails.
    • Every comparison fails: the group fails.
  • Errors: a non-judge error gives 503 for every member and cancels the comparisons still running.
  • Logging: a failed comparison is logged when it fails, before a later non-judge error or the group deadline ends the group.
  • Late requests: a regenerated answer for a masked member gets its judge failure; a scored member still refuses a different answer with 409.
  • /compare: with one failing and one blocked judge call, the request returns 503 at once, and the blocked call is cancelled.
  • Fixed baseline: a failed judge pass masks its rollout, also through /verify.
  • Checkpoint: members masked alone and their scored siblings are exported and restored without judging again. A masked member's report names the judge failure as its cause before and after the restore.
  • main's failure-report test through collection: one failed comparison now gives two masked rows and two scored rows, and each masked row's report has the judge failure in its cause chain.
  • HTTP through /run and the failsafe, 16 rollouts: one failed comparison, and two adjacent failures, mask exactly their rollouts with judge_failed; the other rollouts return their rewards.
  • Regression check: with main's server code, 15 of these 20 test cases fail. The other 5 cover behaviour that does not change, including the /compare case.

Rollouts

This PR's arm ran at 6aeff06e4. The next commit, 9ee859305, changes only the reply to a regenerated answer for a masked member, the reason text when every rollout is masked, docs and tests; scoring and masking are the same. The later merges of main and the /compare fix do not change how /verify scores or masks rollouts. The main arm ran at 5911d628c (the base of this PR) in an earlier run, with the same judge, policy, data and settings.

Setup

  • Judge: nvidia/Qwen3-Nemotron-235B-A22B-GenRM, served by vLLM 0.27.1 with tensor parallel size 4 on one GB200 node.
  • Policy: Qwen/Qwen3.5-4B with thinking off, served by vLLM on one GPU of a second node.
  • Data: the 4 tasks in resources_servers/genrm_compare/data/example.jsonl. Each run holds one task with a new _ng_group_id, so it makes one 16-rollout circular group.
  • Settings: the shipped genrm_compare settings, with two changes that cause real judge failures:
    • genrm_responses_create_params.max_output_tokens: 1341. We first ran 16 judge comparisons with a 16,384-token budget. The answers were 531 to 2,981 tokens long, and 1,341 is their 55th percentile. So some answers run out of budget before the verdict.
    • genrm_parse_retries: 1, so each comparison gets 2 attempts.

Commands. Each arm starts its servers once, and they stay up for all 4 tasks.

gym env start "+config_paths=[genrm_5330_run.yaml]" \
    +policy_base_url=http://<policy-host>:8001/v1 +policy_api_key=dummy +policy_model_name=policy \
    +genrm_base_url=http://<judge-host>:8000/v1 +genrm_model_name=genrm +genrm_max_output_tokens=1341 \
    ++genrm_compare_resources_server.resources_servers.genrm_compare.genrm_parse_retries=1

# Once per task, with a new _ng_group_id in task<k>.jsonl:
gym eval run --no-serve --agent genrm_simple_agent \
    --input task<k>.jsonl --output task<k>-results.jsonl \
    --num-repeats 16 --concurrency 16 +route_failures_to_sidecar=true
genrm_5330_run.yaml

genrm_compare_resources_server and genrm_simple_agent are the same as in configs/genrm_compare.yaml, with max_output_tokens: ${genrm_max_output_tokens}. The two model servers point at the external vLLM servers:

policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py
      base_url: ${policy_base_url}
      api_key: ${policy_api_key}
      model: ${policy_model_name}
      return_token_id_information: false
      uses_reasoning_parser: false
      chat_template_kwargs:
        enable_thinking: false

genrm_model:
  responses_api_models:
    genrm_model:
      entrypoint: app.py
      base_url: ${genrm_base_url}
      api_key: dummy_key
      model: ${genrm_model_name}
      return_token_id_information: false
      uses_reasoning_parser: false
      supports_principle_role: true
      hf_home: null
      vllm_serve_env_vars: {}
      vllm_serve_kwargs:
        tensor_parallel_size: 4
      debug: false

Results (one 16-rollout group per task and arm)

Task This PR (6aeff06e4) main (5911d628c)
1 4 of 16 comparisons failed: (8, 9), (9, 10), (10, 11) and (14, 15). Their rollouts, 8 to 11, 14 and 15, were masked. The other 10 were scored (3.0 to 4.5). Rates: 0.25 and 0.375. The first failed comparison failed the group. All 16 rollouts were masked.
2 No comparison failed. All 16 were scored (3.0 to 5.25). Rates: 0.0 and 0.0. No comparison failed. All 16 were scored (2.0 to 5.0).
3 4 of 16 comparisons failed: (0, 1), (10, 11), (12, 13) and (14, 15). Their rollouts, 0, 1 and 10 to 15, were masked. The other 8 were scored (3.5 to 5.25). Rates: 0.25 and 0.5. The first failed comparison failed the group. All 16 were masked.
4 1 of 16 comparisons failed: (15, 0). Rollouts 15 and 0 were masked. The other 14 were scored (2.0 to 5.0). Rates: 0.0625 and 0.125. The first failed comparison failed the group. All 16 were masked.

"Rates" are genrm_failed_comparison_rate_per_group and genrm_masked_rollout_rate_per_group, as reported on each scored row.

This PR scored 48 of the 64 rollouts, and main scored 16. On this PR, the masked rollouts are exactly the rollouts of the failed comparisons, and no group failed as a whole. On both arms, every masked row has mask_sample: true, failure_kind: judge_failed and reward 0.0, and lands in the collector's _failures.jsonl sidecar.

The two arms ran separately, and each sampled its own policy and judge answers. Tasks 1, 3 and 4 had failed comparisons on both arms, but not the same ones. On main, a group fails at its first failed comparison, so its log shows one failed comparison per masked group.

Server log lines

This PR, task 4 (one failed comparison, two rollouts masked):

GenRM output budget exhausted for pair (15, 0) (attempt 2/2, max_output_tokens=1341): response is incomplete (reason=max_output_tokens) and contains no verdict
GenRM dropped comparison (15, 0) (judge 0) of 16 after a judge failure: Judge returned no completed answer after 2 attempts (2 of them exhausted max_output_tokens=1341 without emitting a verdict; raise the budget or constrain the judge's reasoning)
GenRM masked one rollout of key='group_id::996c6e1f-f9ce-4ceb-86d6-d05f765bcdb0::group_attempt::0': GenRM rollout_index=15 lost a comparison to a judge failure (1 of 16 comparisons failed): Judge returned no completed answer after 2 attempts (...)
GenRM masked one rollout of key='group_id::996c6e1f-f9ce-4ceb-86d6-d05f765bcdb0::group_attempt::0': GenRM rollout_index=0 lost a comparison to a judge failure (1 of 16 comparisons failed): Judge returned no completed answer after 2 attempts (...)

main, task 1 (one failed comparison, group failed):

GenRM output budget exhausted for pair (7, 8) (attempt 2/2, max_output_tokens=1341): response is incomplete (reason=max_output_tokens) and contains no verdict
GenRM cohort disposition=failed key='group_id::d1c0ec60-46a2-4bd0-ba08-4757723b0524::group_attempt::0' attempt=0 members=16 kind=judge reason="Judge returned no completed answer after 2 attempts (2 of them exhausted max_output_tokens=1341 without emitting a verdict; raise the budget or constrain the judge's reasoning)"

Compatibility and benchmark impact

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally.
  • Documentation added or updated (resources_servers/genrm_compare/README.md).
  • Pre-commit checks pass locally (pre-commit run --all-files).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s).

🤖 Generated with Claude Code

…ly when a rollout has none left

Since #3351 one failed judge comparison (no completed answer, judge HTTP
error or request deadline) fails the whole cohort, so all of its rollouts
are masked even when every other comparison succeeded. With 16 rollouts
and a long judge answer, one comparison that exhausts max_output_tokens on
both attempts discards 15 good comparisons.

Drop each comparison that raises a JudgeError and score every rollout
from the comparisons it has left; nothing is masked per rollout. As soon
as a rollout has no comparison left that could succeed, fail the cohort
as a judge failure exactly as before (every member masked through the
judge failsafe) and cancel the comparisons still running, so an outage
still ends the group early. Under circular comparisons only two adjacent
failures leave a rollout uncovered. Non-judge errors still fail the group
with 503, malformed completed answers keep the default-score fallback,
and cohort_evaluation_timeout_s still bounds the group. The fixed-baseline
path and /compare follow the same rule; /compare leaves dropped
comparisons out of comparison_results. There is no configuration key.

Report genrm_failed_comparison_rate_per_group and log each drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yfw yfw added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work labels Oct 29, 2026
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f6d207d

…_pairs

- Log a dropped comparison when it fails, not after the loop. A later
  failure, a non-judge error or the group deadline can end the group
  first, and the group error carries only the last failure's text.
- README: a connection error or a non-JSON judge reply is dropped too.
  A request deadline drops a comparison only if it fires before the group
  deadline, and by default the group deadline fires first. HTTP 503
  applies to /verify.
- README: name the position effect of a dropped circular comparison.
- Test all_pairs, where each rollout has three comparisons, and test
  that an earlier failure is logged when the group fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a66283d

…comparison

A failed judge comparison is still dropped. A rollout whose comparisons
all failed is now masked alone through the judge failsafe (reward 0.0,
mask_sample, failure_kind judge_failed), and its siblings keep their
scores. Before, it failed the whole group. The masked rollout is left out
of the group's length and style adjustments. When every rollout is left
uncovered, the whole group fails as a judge failure, as before.

- The server now waits for every comparison to finish or fail, within
  cohort_evaluation_timeout_s. A non-judge error still fails the group at
  once and cancels the comparisons still running.
- The fixed_baseline path follows the same rule. /compare has no
  per-response mask, so it still returns 503 when a response keeps no
  comparison.
- Checkpoints save and restore a member masked alone (masked_alone).
- New metric: genrm_masked_rollout_rate_per_group.

The masked rollout's reward is a placeholder, so a trainer must keep it
out of the group statistics: in NeMo RL, grpo.masked_reward_policy
exclude_infra (NVIDIA-NeMo/RL#4968) or exclude.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d94066c

A failed judge comparison now masks its two rollouts, and every other
rollout is scored from all of its comparisons. Before, the two rollouts
were scored from their remaining comparison. In a group-relative trainer
that mixed rewards averaged over two comparisons with rewards from one,
and a judge that favors one position moved the two scores in opposite
directions.

- A masked rollout's successful comparisons still score its neighbors.
  The group's length and style adjustments see only the scored rollouts.
- The same rule covers all_pairs, several judge passes and the
  fixed_baseline path: a rollout with any failed comparison or pass is
  masked. The whole group fails only when every rollout is masked.
- /compare has no per-response mask, so any failed comparison returns 503
  there, as on main.
- The masked-member reason now reads "GenRM rollout_index=N lost a
  comparison to a judge failure (k of n comparisons failed): ...".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6aeff06

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 30, 2026
…d answer

A resume regenerates a masked member's answer. The completed group now answers it with the recorded
judge failure, as a failed group does, instead of HTTP 409, which aborted collection by default.

Also: when every rollout is masked, the group failure says so; the README names
env.should_mask_flagged_samples in the consumer requirement and restores the /compare sentence; the
_collect_comparisons return type holds JudgeError; tests check that a failed comparison is logged
before a later error ends the group, and that a neighbor of a masked rollout keeps both comparisons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9ee8593

Resolve the import conflict in tests/test_http_lifecycle.py: use main's
new module paths and import JudgeError from nemo_gym.verification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1f4ccf8

@yuhezhang-ai yuhezhang-ai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @yfw! I left one inline comment about returning promptly after a judge failure in /compare.

I also have an earlier performance PR, #3625, that touches some of the same scoring and checkpoint code. It already incorporates main’s newer failure reporting, so I’d recommend landing that first, then rebasing this PR and rechecking the combined changes. I think that would make the overlap easier to reconcile.

Would that order work for you, or would you prefer a different approach? Happy to coordinate. We’d still keep NVIDIA-NeMo/RL#4968 ahead of this PR, as you’ve documented.

Comment thread resources_servers/genrm_compare/app.py Resolved
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 4, 2026
yfw and others added 2 commits November 16, 2026 01:38
…ed-comparisons

Main added failure reports (#5072 and its follow-ups), which reach the same
GenRM code. The conflicts kept both sides: masked_alone next to
failure_report in the saved member state, member_failures next to
failure_report when a cohort is published, and the cohort failure next to
the restored report.

A member masked alone now fails with its failed comparison's Gym failure as
its cause, as a failed group does. The cohort's report keeps that failure, so
the cause survives a checkpoint restore: the member's saved report names it,
and the restore clears it for the cohort. When every member is masked, the
group's JudgeError is caused by the first failed comparison.

Tests: main's one_failed report test now expects two masked rows and two
scored ones; the masked-alone restore test uses main's export signature and
checks the restored report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
/compare has no per-response mask, so it cannot use a partial result. Since
this PR, it waited for every comparison before it failed the request, so one
quick judge failure and one slow judge call delayed the 503 until the slow
call finished, with no group deadline. Now its first failed comparison is
raised at once and the comparisons still running are cancelled, as before
this PR. /verify still waits for every comparison and masks only the
members of a failed one.

The regression test has one failing and one blocked judge call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@yuhezhang-ai Thanks for the review. Landing #3625 first works for me, and NVIDIA-NeMo/RL#4968 still goes before this PR.

I checked the overlap at the new head. I merged main into this branch (35ac9ba, a merge, not a rebase). Then I did a trial merge of #3625 at bdc6376 on top:

  • There are two small conflicts in app.py, and both keep both sides. In verify, your digest-scheme check goes before the masked-alone check. In _finished_member_state, the member's own digest scheme goes next to its own failure report.
  • With those two resolved, all 400 genrm_compare tests pass.
  • One note for perf(genrm): reduce response copying and group cleanup overhead #3625: 13 of its test cases call export(None) or export_session_states(sessions). Current main requires checkpoint_id= there, so these cases fail after a merge of main.

The main merge also connects main's failure reports to masking. A masked member's failure has its failed comparison's Gym failure as its cause, and a checkpoint keeps that cause. When #3625 lands, I will merge main into this branch again, resolve the two conflicts and run the tests again.

@yfw yfw changed the title feat(genrm_compare): drop failed judge comparisons; fail the group only when a rollout has none left feat(genrm_compare): mask only the two rollouts of a failed judge comparison Nov 17, 2026
@yfw

yfw commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5bf85a5

@yuhezhang-ai

Copy link
Copy Markdown
Contributor

@yfw Thanks for catching the missing checkpoint IDs. Fixed in #3625 at 0a1981d8460a42804687638ed966ba2f5f1b9a7e after merging current main (505a2afa; merge 9b8390aae).

I reproduced all 13 failures, then updated the nine export call sites to pass checkpoint_id=. The repeated-checkpoint digest test now uses distinct IDs for successive saves; its replay and saved-scheme assertions are unchanged. All 380 GenRM tests, example-data validation, and 182 focused core checkpoint/failure-report tests pass on the pushed tree. All-files pre-commit passed; the private guardwords catalog was unavailable locally. CI has been requested for this exact head.

Agreed on the order: #3625 lands before #5330, and NVIDIA-NeMo/RL#4968 must also land before #5330. The merge here preserves the digest-scheme check and each member's saved scheme for the two overlap points you identified. Please merge main and rerun the combined suite once #3625 lands; your 400-test trial remains separate from this validation.

This branch was previously deployed

1 inactive (outdated) deployment
public — 1f4ccf83 Deployed Oct 30, 2026 by copy-pr-bot[bot] via release / finalize / notify #8805
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants