Repository navigation
feat(checkpoint): declare twenty training verifiers stateless with replayable verification - #3887
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
77bfad7 to
f91e01f
Compare
f91e01f to
39f45c2
Compare
39f45c2 to
2ef25f4
Compare
2ef25f4 to
82f0814
Compare
31ca329 to
ae94453
Compare
bf536e9 to
5be2651
Compare
5be2651 to
114aa79
Compare
114aa79 to
06570d4
Compare
|
@ananthsub added some more training verifiers as stateless #4540. Can we merge it? |
|
Yes. The 15 declarations from #4540 are now in this PR's commit, with you as co-author, so #4540 can be closed.
|
…ous session hooks A resources server declares a checkpoint mode: stateless, exported, or restart-only. In exported mode its session state is exported and restored through asynchronous hooks. A checkpoint waits for requests in flight; a request that arrives during one waits for resume instead of being refused; tools called over MCP are covered; and retire frees a session once its requests have stopped. Nothing of a restart-only server is in a checkpoint. Its admission never closes, its requests never hold up prepare, and its live sessions are reported as restarts. Its seed reply says so, so the episode that seeded the session starts over from its input after a crash, and keeps running through the checkpoint if nothing crashes. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…playable verification These verifiers keep no session state, so they declare themselves stateless, and their /verify may run again after a crash instead of holding up a checkpoint: abstention, calendar, code generation, competitive coding, equivalence by rule, LLM-judge equivalence, ether0, format verification, GenRM comparison, instruction following, inverse instruction following, jailbreak detection, math with judge, multiple choice, MultiChallenge, NVARC, Reasoning Gym, string match, structured outputs, and Terminus judge. Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38766272125 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38768165705 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38770338048 |
What changed and why
A resources server that declares nothing is restart-only, so the rollouts that use it start over from their input after a crash. These twenty training verifiers keep no session state and their verification can run again after a crash, so they declare
checkpoint_mode = "stateless"andcheckpoint_verify = "replay":code_gen,competitive_coding_challenges,equivalence_llm_judge, andmath_with_judgeverify each request on its own.abstention,calendar,equivalence_rule,ether0,format_verification,instruction_following,inverse_if,jailbreak_detection,mcqa,multichallenge,nvarc,reasoning_gym,string_match,structured_outputs, andterminus_judgealso verify each request on its own, from the request body, with a judge model call, or with a subprocess that ends with the call (from feat(checkpoint): declare 15 more training verifiers stateless and replayable #4540, by macandro96). A replayed judge call can return a different verdict; the declaration says that repeating it is safe, not that the result is identical.genrm_comparekeys its cohorts by prompt, not by session. Its verification must replay: a member waits for its siblings, so waiting on it would deadlock a checkpoint while siblings are still generating. After a crash, members that had not recorded their reward re-verify and rebuild the cohort. A crash between a cohort's result and every member recording it leaves the remaining members waiting; they fail aftercohort_collection_timeout_sand are retried from input.The declarations are class attributes, not config, because they describe what each server's code can do. The test reads them from source, because each server's dependencies live in its own environment.
How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D V["Five training verifiers<br/>stateless, replayable verify"] R --- V classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class V thisOne checkpoint, a crash, and the restore, end to end:
This PR
These five servers declare that verification can run again after a crash, so a checkpoint never waits for a slow judge.
Where this sits in the stack
This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.
The stack builds on the token-capture cleanup PRs #3938 (capture ledger
retireanddelete) and #3939 (complete-recordretireanddelete), which have merged. #3882 to #3885 have merged too, so #3886's base ismain. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooksananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verification (this PR)ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuationananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordinationananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collectionananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpointsananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workersananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operationsPartial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the
checkpoint:block unset, no server installs any checkpoint routes or behavior.Relationship to the old stack (#2939 to #2946)
None of #2939 to #2946 declared these servers. The declarations come from #3349 (durable turn-level recovery), where they were config on the agent; here they are class attributes on the resources servers.
Issue
No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.
Validation
Run on this branch, on the current
main:tests/unit_tests/test_checkpoint_server_declarations.py: each of the twenty servers declaresstatelessandreplay, read from its source.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/code_gen/tests: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/competitive_coding_challenges/tests: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/equivalence_llm_judge/tests: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/genrm_compare/tests: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/math_with_judge/tests: not run in this environment. It needs the server'smath-verify==0.8.0requirement, which this environment does not install. It passed on an earlier version of this branch with that requirement installed.tests/unit_tests, eight processes, withouttest_opensandbox_compose.py, which needs the optionalopensandboxpackage): passed except a sandbox retry test, which fails the same way onmainin this development environment.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Signed-off-byline.Rollout evidence
Compatibility
checkpoint:block setsenabled: true. With it on, these five servers no longer block checkpoints, and their/verifymay run again after a crash.