Skip to content

feat(checkpoint): declare twenty training verifiers stateless with replayable verification - #3887

Merged
ananthsub merged 2 commits into
mainfrom
ananthsub/partial-ckpt-verifier-declarations
Oct 15, 2026
Merged

ananthsub merged 2 commits into
mainfrom
ananthsub/partial-ckpt-verifier-declarations

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

A resources server that declares nothing is restart-only, so the rollouts that use it start over from their input after a crash. These twenty training verifiers keep no session state and their verification can run again after a crash, so they declare checkpoint_mode = "stateless" and checkpoint_verify = "replay":

  • code_gen, competitive_coding_challenges, equivalence_llm_judge, and math_with_judge verify each request on its own.
  • abstention, calendar, equivalence_rule, ether0, format_verification, instruction_following, inverse_if, jailbreak_detection, mcqa, multichallenge, nvarc, reasoning_gym, string_match, structured_outputs, and terminus_judge also verify each request on its own, from the request body, with a judge model call, or with a subprocess that ends with the call (from feat(checkpoint): declare 15 more training verifiers stateless and replayable #4540, by macandro96). A replayed judge call can return a different verdict; the declaration says that repeating it is safe, not that the result is identical.
  • genrm_compare keys its cohorts by prompt, not by session. Its verification must replay: a member waits for its siblings, so waiting on it would deadlock a checkpoint while siblings are still generating. After a crash, members that had not recorded their reward re-verify and rebuild the cohort. A crash between a cohort's result and every member recording it leaves the remaining members waiting; they fail after cohort_collection_timeout_s and are retried from input.

The declarations are class attributes, not config, because they describe what each server's code can do. The test reads them from source, because each server's dependencies live in its own environment.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR adds.

flowchart LR
  C["Controller<br/>NeMo RL, or rollout collection"]
  CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"]
  K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"]
  subgraph G["One participant per Gym server"]
    E["Environment server<br/>episode steps"]
    M["Policy model<br/>held responses, generation cuts"]
    A["Agent<br/>sessions parked at boundaries"]
    R["Resources server<br/>session state"]
  end
  W["Inference worker<br/>stages cut prefixes"]
  D[("Checkpoint directory<br/>records, then manifest")]
  L[("Capture ledger<br/>retire and delete from #3938, #3939")]
  C --> CO --> K
  K --> E & M & A & R
  M --> W
  M --> L
  G --> D
  V["Five training verifiers<br/>stateless, replayable verify"]
  R --- V
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class V this
Loading

One checkpoint, a crash, and the restore, end to end:

sequenceDiagram
  participant C as Controller
  participant G as Gym participants
  participant D as Checkpoint directory
  C->>G: prepare, in order environment, model, agent, resources
  Note over G: admission closes, in-flight work parks at a boundary,<br/>undelivered model responses are held
  G-->>C: prepared, or blockers at the deadline
  C->>G: commit with the episodes the controller continues
  G->>D: each participant writes its records, then its manifest
  Note over G: restored state the commit's scope leaves out is deleted
  C->>C: publish the checkpoint with the controller's own state
  C->>G: resume, in order resources, agent, model, environment
  Note over C,G: crash - every Gym process dies
  C->>G: restore the checkpoint in fresh processes, all or nothing
  D-->>G: records installed under attempt N+1, attempt N's capture ledger retired
  C->>G: resume
  C->>G: /run as attempt N+1 continues each episode from its boundary
  Note over C,G: retiring an episode, only while no checkpoint is open
  C->>G: retire - environment, then agent, then model and resources
  Note over G: each server marks the attempt, stops its work, waits, frees its state, then replies
  C->>G: forget, once the rollout is done for good - same order
  Note over G: the mark refusing the retired attempt goes away
Loading

This PR

These five servers declare that verification can run again after a crash, so a checkpoint never waits for a slow judge.

flowchart LR
  E["Environment server"] -->|"/seed_session"| R["code_gen, competitive_coding_challenges,<br/>equivalence_llm_judge, math_with_judge, genrm_compare"]
  R -->|"x-ng-checkpoint-verify: replay"| E
  E -->|"verify in flight at a checkpoint"| N["The checkpoint does not wait.<br/>Verify runs again after a crash."]
Loading

Where this sits in the stack

This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.

The stack builds on the token-capture cleanup PRs #3938 (capture ledger retire and delete) and #3939 (complete-record retire and delete), which have merged. #3882 to #3885 have merged too, so #3886's base is main. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.

  1. feat(checkpoint): add the participant control plane, episode steps, and coordination #3882 (ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)
  2. feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883 (ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)
  3. feat(token-capture): add worker staging helpers for generation cuts #3884 (ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)
  4. feat(checkpoint): continue environment server episodes from their boundaries #3885 (ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)
  5. feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 (ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks
  6. feat(checkpoint): declare twenty training verifiers stateless with replayable verification #3887 (ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verification (this PR)
  7. feat(checkpoint): add the agent session participant and Simple Agent continuation #3888 (ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuation
  8. test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 (ananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordination
  9. feat(checkpoint): checkpoint evaluation runs from rollout collection #3893 (ananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collection
  10. feat(checkpoint): spans and metrics for partial-rollout checkpoints #3903 (ananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpoints
  11. feat(checkpoint): resources, agent, and environment servers with several workers #3909 (ananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workers
  12. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 (ananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operations

Partial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the checkpoint: block unset, no server installs any checkpoint routes or behavior.

Relationship to the old stack (#2939 to #2946)

None of #2939 to #2946 declared these servers. The declarations come from #3349 (durable turn-level recovery), where they were config on the agent; here they are class attributes on the resources servers.

Issue

No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.

Validation

Run on this branch, on the current main:

  • tests/unit_tests/test_checkpoint_server_declarations.py: each of the twenty servers declares stateless and replay, read from its source.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/code_gen/tests: passed.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/competitive_coding_challenges/tests: passed.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/equivalence_llm_judge/tests: passed.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/genrm_compare/tests: passed.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/math_with_judge/tests: not run in this environment. It needs the server's math-verify==0.8.0 requirement, which this environment does not install. It passed on an earlier version of this branch with that requirement installed.
  • The full core unit suite (tests/unit_tests, eight processes, without test_opensandbox_compose.py, which needs the optional opensandbox package): passed except a sandbox retry test, which fails the same way on main in this development environment.
  • pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.
  • Every commit carries a Signed-off-by line.

Rollout evidence

  • These are declarations only: with checkpointing off, nothing changes for these servers. With it on, they no longer force their rollouts to be retired at each checkpoint.
  • A checkpointed training run on these environments with a real model: pending.

Compatibility

  • Nothing changes unless the global checkpoint: block sets enabled: true. With it on, these five servers no longer block checkpoints, and their /verify may run again after a crash.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub requested a review from zyzhou5 October 1, 2026 19:04
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from 77bfad7 to f91e01f Compare October 1, 2026 19:38
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from f91e01f to 39f45c2 Compare October 1, 2026 21:53
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from 39f45c2 to 2ef25f4 Compare October 1, 2026 22:05
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from 2ef25f4 to 82f0814 Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch 2 times, most recently from 31ca329 to ae94453 Compare October 2, 2026 21:17
@pthombre
pthombre added this pull request to stack #3962 October 2, 2026 22:50
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from bf536e9 to 5be2651 Compare October 10, 2026 05:09
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from 5be2651 to 114aa79 Compare October 10, 2026 05:30
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-verifier-declarations branch from 114aa79 to 06570d4 Compare October 12, 2026 14:56
@macandro96

Copy link
Copy Markdown
Contributor

@ananthsub added some more training verifiers as stateless #4540. Can we merge it?

@ananthsub

Copy link
Copy Markdown
Contributor Author

Yes. The 15 declarations from #4540 are now in this PR's commit, with you as co-author, so #4540 can be closed.

  • The title and description now say twenty verifiers, and tests/unit_tests/test_checkpoint_server_declarations.py covers all of them.
  • Each new declaration has the same two-line comment as the original five, naming what that server's /verify does.
  • ray_enabled = False from the feat(checkpoint): declare 15 more training verifiers stateless and replayable #4540 branch is left out, because main has since removed it from every server.

macandro96
macandro96 previously approved these changes Oct 14, 2026
ananthsub and others added 2 commits October 15, 2026 08:11
…ous session hooks

A resources server declares a checkpoint mode: stateless, exported, or
restart-only. In exported mode its session state is exported and
restored through asynchronous hooks. A checkpoint waits for requests in
flight; a request that arrives during one waits for resume instead of
being refused; tools called over MCP are covered; and retire frees a
session once its requests have stopped.

Nothing of a restart-only server is in a checkpoint. Its admission
never closes, its requests never hold up prepare, and its live sessions
are reported as restarts. Its seed reply says so, so the episode that
seeded the session starts over from its input after a crash, and keeps
running through the checkpoint if nothing crashes.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…playable verification

These verifiers keep no session state, so they declare themselves
stateless, and their /verify may run again after a crash instead of
holding up a checkpoint: abstention, calendar, code generation,
competitive coding, equivalence by rule, LLM-judge equivalence, ether0,
format verification, GenRM comparison, instruction following, inverse
instruction following, jailbreak detection, math with judge, multiple
choice, MultiChallenge, NVARC, Reasoning Gym, string match, structured
outputs, and Terminus judge.

Co-authored-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38766272125

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38768165705

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38770338048

This branch was previously deployed

1 inactive deployment
public — cb97276f Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5507
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants