Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
b5ffb47 to
22a07dd
Compare
5f8b5af to
87e1f0c
Compare
22a07dd to
324d650
Compare
87e1f0c to
207f588
Compare
207f588 to
afee845
Compare
324d650 to
29a2feb
Compare
29a2feb to
d1bac74
Compare
afee845 to
6a3c667
Compare
173032c to
b63ade8
Compare
54ffae4 to
4b775b8
Compare
4b775b8 to
e4f30fa
Compare
a014331 to
8575f2b
Compare
e4f30fa to
1d706d5
Compare
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Four correctness findings below, plus one non-blocking checkpoint-retention suggestion.
Validation: 562 focused existing tests passed, along with reduced reproductions and changed-file lint/format checks, at e24a8bc3dc4b. The checkpoint patch is unchanged after the rebase to 6b3f363347e4; I rechecked the intervening code changes and current comment locations. These reproductions exercise the collector and checkpoint-control components, not a real-model or multi-host workload. The current unit-test check passes.
| self._collection = collection | ||
| loop = asyncio.get_running_loop() | ||
| loop.add_signal_handler(signal.SIGUSR1, self._spawn, lambda: self.checkpoint()) | ||
| loop.add_signal_handler(signal.SIGTERM, self._spawn, self._checkpoint_and_stop) |
There was a problem hiding this comment.
[P2] Compose the checkpoint and timing-report SIGTERM handlers
When checkpoint_dir and dispatch_start_report_within_s are both set, _run_from_config() subsequently calls _print_on_sigterm(), replacing this event-loop handler. That helper saves signal.getsignal(SIGTERM), which is asyncio's no-op stub rather than this callback. On SIGTERM it prints the timing table, removes the event-loop handler and re-sends the signal to that stub; no checkpoint or stop occurs. I reproduced the combination yielding no coordination calls and a still-running collection, while the same check without timing reports published and stopped. Could the two features share/compose signal ownership, with a regression test enabling both?
There was a problem hiding this comment.
Fixed. With checkpoint_dir set, the checkpoint is the only owner of SIGTERM on the event loop.
The timing report no longer installs _print_on_sigterm(); it registers its table with the checkpointer instead.
On SIGTERM the handler prints the table first, because printing is quick and must not be lost if the checkpoint runs out the grace period, then checkpoints and stops.
start() now records the SIGUSR1 and SIGTERM handlers it replaces, and close() puts them back when collection ends.
Without checkpoint_dir, the timing report keeps its own handler as before.
Covered by test_sigterm_with_the_timing_report_on_checkpoints_stops_and_prints_the_table, which runs run_from_config() with both features on and sends SIGTERM while a row is in flight.
It checks that prepare and commit ran, the checkpoint was published, collection stopped with CollectionStopped, the table was printed before the checkpoint published, and the previous SIGTERM handler is back.
On the previous head it timed out with no coordination calls and collection still running.
There was a problem hiding this comment.
Thanks, the signal composition works at d9ac070, but the new regression test currently fails at line 800: it expects [prepare, commit] and receives [prepare, commit, retire, forget]. With #4029 now in the base, collector shutdown awaits cancellation of the pending row, which runs the cleanup hooks. I reproduced this in isolation; updating only that expected call list in a scratch copy lets all the remaining checks pass, including the published checkpoint, CollectionStopped, timing output order, and restored signal handler. Could you update the assertion to account for that cleanup? The unmodified checkpoint/collector/eval-command suite is 545 passed, 1 failed.
There was a problem hiding this comment.
Thanks, reproduced at d9ac070: with #4029 in the base, shutdown now cancels the pending row, and its after_dispatch(failed=True) retired the attempt and close() then forgot it.
Rather than accept those calls in the assertion, the run no longer sends them, so the test keeps expecting only prepare and commit.
After a stopping checkpoint the servers stay committed, and the control routes refuse a retire or a forget outside the idle and restored phases.
Real servers would have refused both, and the run would have logged that the rows' state stays on the servers.
Fixed in 6ed373f:
- once a run has stopped,
after_dispatch()records no failure, and the background retire and forget tasks return without sending; close()of a stopped run skips its final retire and forget;- the next run's restore already retires every attempt the stopped run sent, so nothing is left unfenced.
The new path also exposed a hang in close().
asyncio.gather() over tasks that have all finished returns without yielding, so a background task whose done callback had not run yet stayed in the set and the loop never ended.
close() now removes the tasks it waited for itself.
Covered by test_a_stopped_run_sends_no_retire_or_forget_to_its_committed_servers, where one row is retired before the stopping checkpoint, one fails while the checkpoint holds the lock, and one is cancelled after it.
It checks that nothing follows prepare and commit, and that the commit continues only the row still in flight.
On the previous head it fails with a retire and a forget after the commit; with only the close() change reverted it hangs.
The checkpoint, collector and evaluation command suites pass, including the SIGTERM test you ran.
|
/ok to test 3fa6d23 |
@ananthsub, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Rollout collection checkpoints the rows in flight on a signal or a timer, and continues them when the run resumes from its cache. Rows whose /run got no reply are retired everywhere, and the rollouts it retired are forgotten once the run dispatches nothing more. Rows that some server cannot capture, such as a restart-only agent's, are left out of the commit and recorded as restarts; they keep running through the checkpoint. After a crash, the run retires each restart's attempt everywhere, so a harness that outlived the crash cannot write into the rerun, and starts the row over as its next attempt. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
What changed and why
The rest of the stack lets a controller checkpoint Gym with rollouts in flight. The controller has to be the process that sends episodes: only it knows which episodes it will continue, and only it can send their replacements after a restore. A training framework such as NeMo RL plays that role for training. For an evaluation run it is rollout collection, which so far could only restart unfinished rows from their inputs (
resume_from_cache). This PR makes rollout collection the controller of its own rows, so a long evaluation task survives a preemption without starting over.With
checkpoint_dirset (and the globalcheckpoint:block enabled):Triggers.
SIGUSR1checkpoints the rows in flight and continues.SIGTERMcheckpoints them and stops the run. Slurm can send it ahead of a preemption, for example with--signal=B:TERM@120.checkpoint_every_scheckpoints on a timer, which covers crashes that give no warning.checkpoint_dir/collector.pid, for sending the signals.SIGTERMfor the run. Withdispatch_start_report_within_salso set, its handler prints the timing table first, then checkpoints and stops. The previousSIGUSR1andSIGTERMhandlers are restored when the run ends.A checkpoint:
/runhas not replied, except restarts, rows some server cannot capture, which keep running and are listed in the manifest;checkpoint_dir/LATESTat it;The checkpoint just published is always kept, plus the newest of the others, so a clock that steps back never deletes the checkpoint
LATESTnames.A checkpoint that does not prepare in time publishes nothing, and the run continues from its previous checkpoint.
A checkpoint the timer started is finished before the run closes, and a checkpoint that is cancelled still resumes every participant.
Dispatch log. Every run keeps
checkpoint_dir/dispatched.jsonl: its run ID, then one line for each attempt it sends, written just before the/runthat sends it, plus the attempts a restore installs and the retires it carries over from a crashed run. The log is flushed to the kernel but never synced. A killed process leaves its lines behind, and a crashed host takes its Gym servers down with it, so nothing on it is left to retire.Rows refused while a checkpoint is open. A row can pass the dispatch gate and still reach the environment server after prepare closed admission, for example while it waits for a connection. The server refuses it with 409
checkpoint_admission_closedbefore anything of the episode runs. Rollout collection sends the same attempt again after the resume; the row stays in flight and is not counted as a failure. Any other 409 is handled as before.Restart. A run restarted with
resume_from_cache=truereads the log and the latest checkpoint before dispatching anything.resume_from_cache=false) forgets the checkpoints of an earlier run in the same directory, as it already does with the output file, and starts a new log.Code.
nemo_gym/_checkpoint/collection.pyholds the controller.rollout_collection.py: three config fields, a gate and an in-flight count around each row's/run, and the restore step after the cache load.gym evaland the collection CLI print a message when a preemption signal stopped the run, and exit with a SIGTERM kill's status, 143, asmaindoes for a run a SIGTERM interrupts without a checkpoint. A scheduler then never takes a stopped run for a finished one.Cleanup.
/rungot no reply is retired everywhere, in batches under the checkpoint lock, so a retire never lands inside a checkpoint. Otherwise state it left behind, such as a restored record nothing claimed, would be exported again by every later checkpoint. A reply, even a failure, means the environment server already ended the episode and freed what it held.retire_incomplete, because it still holds a session of the rollout (for example one a protocol route started after the retire), is answered by retiring the rollout again at the attempt it retired and then forgetting it again, before the next checkpoint and when the run closes, so that session never holds up a later checkpoint. Every participant's refusal of the retired attempts is bounded by the run.Without
checkpoint_dir, rollout collection behaves exactly as before. The dispatch hook is only passed when it is set, so subclasses that override the internal dispatch method keep working.How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Rollout collection, this PR<br/>SIGTERM, SIGUSR1, or a timer"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class C thisOne checkpoint, a crash, and the restore, end to end:
This PR
Checkpointing an evaluation run, and continuing it after the job restarts.
sequenceDiagram participant S as Slurm or an operator participant RC as Rollout collection participant G as Gym servers participant D as checkpoint_dir S->>RC: SIGTERM, SIGUSR1, or the timer fires RC->>RC: stop starting rows RC->>G: prepare RC->>G: commit the rows whose /run has not replied RC->>D: collection.json, then LATEST alt SIGTERM RC-->>S: exit, the job is requeued else SIGUSR1 or the timer RC->>G: resume, rows start again end Note over RC,G: restarted job with resume_from_cache=true RC->>D: read dispatched.jsonl and LATEST RC->>G: retire the logged attempt of every unfinished row it does not restore RC->>G: restore, then resume, if the log's run published LATEST RC->>G: continued rows as attempt N+1, other unfinished rows from their input Note over RC: restored rows not dispatched yet stay in the next checkpoint's scope Note over RC: a row whose /run got no reply is retired, between checkpointsWhere this sits in the stack
This PR is independent of the rest of the checkpoint chain: #3909 (several workers) has merged, and #4197 (fault tests) does not touch rollout collection. It builds on #4997, which has merged and gives each checkpoint concept one name before the feature is released, including the
checkpoint_prefix on every checkpoint error code. Everything else it builds on (#3882 to #3889) has merged.Once #3903 merges, this PR also gets its two spans,
gym.checkpoint.collection.checkpointandgym.checkpoint.collection.restore; if this PR merges first, they come in a small follow-up instead.#3909 added the end-to-end test helper
reward_of()that this PR's collection tests use, so this PR uses that one instead of adding its own.Relationship to the old stack (#2939 to #2946)
New. The old stack left the controller to the training framework and had no evaluation path.
Issue
No tracking issue exists. This came out of design discussion of the stack: how a single long evaluation task would be checkpointed without a training controller.
Validation
Run on this branch, on the current
main, which includes #4997 and #3909:RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_collection.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_cli_eval.py tests/unit_tests/test_cli_eval_submit.py: passed.test_cli_eval.pyadds: a run that checkpointed and stopped exits with status 143; without the change it exits as a success.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed.test_checkpoint_collection.pycovers the controller with coordination faked:checkpoint_retire_incompleteis retired and forgotten again before the next checkpoint prepares, and when the run closes;/rungot no reply are retired, and rows that replied are not;checkpoint_dir;checkpoint_admission_closedbecause it reached the environment server after prepare runs again after the resume and finishes with its reward, with the failure sidecar off and on, using the real environment participant and control routes;SIGTERMwith the timing report on prints the table, checkpoints, publishes and stops, and the previous handler is restored;LATESTnames when the clock steps back.SIGTERMon the flat-row wrapper and on the typedsingle_agent_turnrequest path, andSIGUSR1) pass in the full e2e suite at the top of the stack.tests/unit_tests, eight processes, withouttest_opensandbox_compose.py, which needs the optionalopensandboxpackage): passed except a sandbox retry test, which fails the same way onmainin this development environment.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Signed-off-byline.Rollout evidence
The process-level tests above run real Gym server processes against the fake inference backend. This PR also adds an opt-in real-model test (
NEMO_GYM_CHECKPOINT_VLLM_URL) that kills the collector without warning after a checkpoint, while the Gym servers keep running, and checks that every row finishes exactly once after the resume.Qwen/Qwen3-4B-Instruct-2507served by vLLM 0.24.0 on one RTX 6000 Ada (--enable-auto-tool-choice --tool-call-parser hermes), driving real Gym server processes that the opt-in tests in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889, feat(checkpoint): checkpoint evaluation runs from rollout collection #3893 and feat(checkpoint): resources, agent, and environment servers with several workers #3909 checkpoint mid-run, kill and restore:single_agent_turnrequest path, sent 20 ms apart until the first one finished. The checkpoint caught about 20 of 26 inside their agent turn. After every Gym process group was killed and the checkpoint restored into fresh processes, every continued rollout finished with reward 1.0, as did every rollout that finished before the checkpoint.restart_onlyresources server. Prepare listed every restart still in flight and no saved rollout, and the commit left the restarts out. After the kill, the restarts were retired and started over from their input while the saved rollouts continued, and every rollout finished with reward 1.0.SIGUSR1checkpoint, four more rows sent, then the collector killed withSIGKILLwhile the Gym servers kept running. The resumed run retired the four rows sent after the checkpoint and sent them again as attempt 1, and all 48 rows finished exactly once with reward 1.0.NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_MULTIWORKER_E2E=1 NEMO_GYM_CHECKPOINT_VLLM_URL=http://127.0.0.1:8100/v1 NEMO_GYM_CHECKPOINT_VLLM_MODEL=Qwen/Qwen3-4B-Instruct-2507 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint -k real_modelCompatibility
checkpoint_dir(default unset);checkpoint_every_s(needscheckpoint_dir);checkpoint_timeout_s(default 300).checkpoint_diralso holds the run's dispatch log,dispatched.jsonl, which the next run replaces.checkpoint_dirset, rollout collection installsSIGTERMandSIGUSR1handlers for the length of the run and restores the previous ones afterwards. The timing report then prints its table from that handler instead of installing its own.checkpoint_dirset, a/runrefused with 409checkpoint_admission_closedis sent again as the same attempt after the resume.SIGTERMexits with status 143, like a run aSIGTERMinterrupts without a checkpoint.checkpoint_timeout_sfor a checkpoint that is already under way.