Repository navigation
feat(checkpoint): resources, agent, and environment servers with several workers - #3909
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
d64723f to
a92580a
Compare
a92580a to
6647f1c
Compare
6647f1c to
ec6b311
Compare
51dc9c8 to
bd1073e
Compare
bd1073e to
8b7a26d
Compare
8b7a26d to
c059299
Compare
macandro96
left a comment
There was a problem hiding this comment.
Thanks for this. I reviewed the multi-worker checkpoint changes and verified the main findings against the code; several were reproduced with small tests.
Blockers
- Restored records are removed before the worker confirms the claim (
participant_workers.py:183). - Agent records are de-duplicated by the wrong identity, and agent stale copies are never cleaned up (
participant_workers.py:225). - Cookie-less agent seed/close are not serialized across workers (
session_routing.py:209). - Cancelling a worker RPC doesn't stop the worker's handler, so retries overlap (
workers.py:143).
Fix before merge
5. Restore across workers is not atomic (needs confirmation of the intended contract) (participant_workers.py:318).
6. Retire uses a fixed 30s timeout instead of the request deadline (workers.py:450).
Minor / suggestions: frozen /status with several workers, serial stale-copy cleanup, agent owner vs cookie owner.
Perf follow-up: large synchronous JSON frames.
Relation to #4690: #3909 passes the merged worker records to the agent participant's commit_reply(), so #4690 can produce run_episode_ids with several workers. Finding 2 occurs before that hook: if two distinct agent-owned /run records share a cookie session ID, _one_copy_per_session() drops one, so run_episode_ids also omits that run. This is a #3909 de-duplication bug exposed by the combined stack, not a bug introduced by #4690.
| return None | ||
| if not self.accepting: | ||
| raise AdmissionClosedError(f"{self.kind} admission is closed for a checkpoint") | ||
| record = self.restored.pop(body["key"], None) |
There was a problem hiding this comment.
[Blocker] Restored record is removed before the worker confirms the claim.
handle_now pops the record from self.restored as soon as a worker asks for it. If the reply is lost (timeout/disconnect), the worker crashes, or the agent's restore_agent_sessions hook fails (agent.py _claim removes the session on error), the checkpoint state is gone for good: the coordinator no longer holds it and no worker exports it. A single process restarts from the checkpoint on a crash, so this loss only exists with several workers.
Suggestion: lease the record to the claiming worker and drop it only after an explicit acknowledgement that it was installed; return it to restored on failure or worker loss.
There was a problem hiding this comment.
Fixed. A claim now leases the record to the claiming worker instead of removing it.
The worker installs the record inside the claim and then settles the lease: installed, or not, if the install or the agent's restore_agent_sessions hook raised.
A record that was not installed goes back to restored, where any worker can claim it again and the next commit exports it if nobody does.
If the claim's reply is lost or times out after the coordinator leased the record, the worker gives it back the same way.
A settlement that cannot be sent is retried in the background until it is sent or the connection closes.
A lost worker's leases go back to restored when its connection closes, and a retire of the attempt drops the lease, so a lease ends on settlement, on worker loss, or on retire.
While a lease is open, prepare reports the replacement attempt as a blocker, so a commit never starts with a record half installed.
Covered by:
test_a_claimed_record_whose_install_fails_goes_back_and_another_worker_claims_it;test_a_claimed_record_whose_reply_is_lost_goes_back;test_a_claimed_record_of_a_lost_worker_goes_back_and_the_next_checkpoint_exports_it;test_an_unsettled_claim_holds_up_prepare_until_the_install_finishes.
On the previous head the record never came back in the first three, and prepare reportedpreparedwhile the install was still running in the last.
| for raw in reply["records"]: | ||
| # Built and checked on the worker, then parsed from its message: its payloads are JSON. | ||
| record = self.record_model.model_validate(raw, context=PARSED_JSON_CONTEXT) | ||
| session_id = getattr(record, "session_id", None) |
There was a problem hiding this comment.
[Blocker] Agent records are de-duplicated by the wrong identity.
Records are keyed by the cookie session_id, but agent sessions are identified by session_key. This fails both ways:
- Different
session_keys, one cookie: two agent-owned/runepisodes sent with the same agent cookie collapse into one record, and the other episode is silently dropped (only an error is logged). Gym's current first-party callers do not reuse cookies this way, but an external controller with a cookie jar can. - Same
session_key, different cookies: after a restore, a cookie-less re-seed can land on another worker and get a new session ID. Both the stale restored copy and the live copy are exported, giving two records for one episode. AgentSessionParticipanthas nounclaimed_sessions()/delete_unclaimed_session(), and thegetattrfallback at L450 silently reports[]. So agent stale copies are never classified or freed.
Suggestion: let the participant define the logical identity (session_key for agents, session_id for resources), and make stale-copy reporting and cleanup an explicit participant contract implemented for agents.
There was a problem hiding this comment.
Fixed. Each participant kind now names a record's identity with record_identity(): the session key for agent records, the session ID for resources records, and none for environment records.
De-duplication uses it, so two agent /run episodes sent with one cookie are both kept, and a stale restored copy and a live copy of one agent session are exported once.
Stale-copy reporting and cleanup are now an explicit part of CheckpointParticipant: unclaimed_sessions() and delete_unclaimed_session() raise NotImplementedError unless the kind implements them, and the worker calls them directly, with no getattr fallback.
The agent participant implements them over its restored sessions that no activation or request has used; the environment and restart-only agent participants implement them as empty, since nothing of theirs is restored on a worker.
Covered by test_two_agent_run_episodes_sent_with_one_cookie_are_both_exported, which runs two /run episodes under one cookie session on two workers and checks both the written records and the records the agent kind's commit_reply() receives.
That is the list #4690's run_episode_ids would be built from, so both runs would be in it.
On the previous head only run:a was kept.
And by test_a_stale_restored_agent_session_and_its_live_copy_export_once_and_the_stale_copy_is_freed, which re-seeds a restored session on the other worker under a new cookie session. On the previous head both copies were exported, and the stale one was never freed.
| return | ||
| owner = self._owner(HTTPConnection(scope)) | ||
| owner = self.aliases.get(owner, owner) | ||
| if owner is None or owner == self.routing_id: |
There was a problem hiding this comment.
[Blocker] Cookie-less agent operations are not serialized across workers.
A request with no cookie/owner is served by whichever worker receives it. Seed idempotency (_locked_agent_session by agent_session_id) and the unknown-ID close receipt in base_responses_api_agent.py (seed_agent_session / close_agent_session) only exist inside one worker. With several workers:
- a retried identical seed can reach a second worker and create a duplicate session/harness state;
- a close for an unknown ID on worker A doesn't stop a delayed seed on worker B, so the close incorrectly "succeeds".
Suggestion: reserve agent_session_id ownership (and close tombstones) at the coordinator before the operation is served, then route it to that owner.
There was a problem hiding this comment.
Fixed in #3905, which this PR carries unchanged: a seed or close without a cookie naming its owner is routed to the first worker that saw its agent_session_id, and a close of an unknown ID claims the ID, so a delayed seed reaches the same worker and is refused.
The covering tests are in tests/unit_tests/test_session_routing.py; the reply on #3905 lists them.
Restored agent sessions now agree with those claims: a restore claims each agent session it installs on its worker before running the agent's hook, and releases the claims if the hook fails.
test_a_restored_agent_session_is_seeded_and_closed_on_its_worker_without_a_cookie runs two real agent servers with session routing and checkpointing, restores a session onto one worker, and sends the replacement attempt's seed and close without a cookie through the other worker. Both reach the worker that holds the session.
On the previous head the other worker claimed the ID itself and closed a second copy, and the restored session was never closed.
| # Typed, so callers that fall back on a lost coordinator, such as a cut claim, do so here too. | ||
| raise CoordinatorUnavailableError(f"checkpoint message {kind!r} failed: {error!r}") from error | ||
| finally: | ||
| self._pending.pop(message_id, None) |
There was a problem hiding this comment.
[Blocker] A timeout/cancel here doesn't stop the worker's handler.
On timeout or cancellation, only the local reply future is dropped. The worker's _serve task keeps running, so a retry can overlap the original operation. Verified with a repro: a retire whose deadline passed, retried immediately, ran retire_session_state twice concurrently for the same session. In a single process, _concurrently cancels each stop and waits for it to finish before releasing the lock; that invariant is lost here. The same applies to other control messages (e.g. a slow export finishing after admission reopens).
Suggestion: track inbound tasks by message ID and send a cancel/abort that the worker acknowledges, or coalesce retries onto the in-flight operation.
There was a problem hiding this comment.
Fixed by cancelling the other side's handler.
When a caller of Channel.call stops waiting, at its timeout or because it was cancelled, it sends a cancel naming the message, and the other side cancels that handler and answers once the handler has stopped.
The caller returns only then, so each stop is cancelled and awaited before the control lock is released, as _concurrently does in one process.
It waits as long as the handler takes while the other side keeps answering a ping, and gives up, with an error in the log, only if the other side stops answering or the connection closes.
A cancel that arrives after the handler finished is answered at once.
This applies to every message kind, export and install included.
Cancellation was chosen over coalescing a retry onto the operation in flight because it covers every kind in one place and keeps the single-process behavior, where a cut-short operation has stopped before its retry starts; coalescing needs an identity per operation and would still leave an export running after the checkpoint resumed.
Covered by:
test_a_retire_cut_short_by_its_deadline_stops_the_worker_s_teardown_before_a_retry_starts_it, your repro: on the previous head the teardown was still running when the cut-short retire returned, and the immediate retry ran it a second time alongside;test_a_slow_export_is_stopped_on_every_worker_before_resume_returns: on the previous head the worker's export was still running after the resume;test_a_cancel_that_arrives_after_the_handler_finished_is_answered_at_once.
| ) | ||
| for worker_index, placed in installs.items() | ||
| } | ||
| await self.call_each(messages, timeout=None) |
There was a problem hiding this comment.
[Needs confirmation] Restore across workers is not atomic.
If one worker's install fails after others succeed, the coordinator never records its aliases/placements/restored, but the successful workers keep their installed state. The failed-restore path in control.py (_open_after_failed_restore) then reopens admission without rolling those workers back. The comment there says the controller is expected to retire what the install brought. If that's the contract, could you confirm and document it? Otherwise this needs prepare/install/commit-or-abort or a rollback of the workers that succeeded before reopening.
There was a problem hiding this comment.
Confirmed, and closed the gap rather than relying on the controller.
The contract on one process is that a failed restore leaves nothing usable, and rollout collection's coordination.restore() then retires the restored attempts everywhere.
With several workers, a failed install on worker B used to leave worker A's installed state, and admission reopened before that retire.
Now the coordinator's install() collects every worker's outcome; if any worker failed or was lost, the workers that installed theirs delete it again and reset their routing tables to the coordinator's, and only then does the restore fail.
Admission reopens after that, so nothing of the failed restore can be used, and the controller's retire finds nothing left to free.
The contract is in the docstrings of the coordinator's install() and of participant_workers.
Covered by test_a_restore_that_fails_on_one_worker_leaves_nothing_on_the_others, where B's hook raises after A installed: A holds nothing, both routing tables are empty, admission is open, and the controller's retire succeeds. On the previous head A still held its session.
And by test_a_worker_lost_while_it_installs_fails_the_restore_and_leaves_nothing_on_the_others.
|
|
||
| async def retire(self, episode_id: EpisodeId) -> None: | ||
| self.retire_restored(episode_id) | ||
| await self.broadcast( |
There was a problem hiding this comment.
Retire uses a fixed 30s timeout instead of the request deadline.
The single-process retire is bounded only by the RetireRequest deadline (_within(request, ...) in control.py). Here it's MESSAGE_TIMEOUT_SECONDS, so a teardown hook slower than 30s returns a 503 checkpoint_coordinator_unavailable even with minutes of deadline left, while the worker keeps retiring. Verified with a scaled-down repro. delete_restored has the same issue (it can await retire_session_state too). Suggest passing the request deadline through, together with the cancellation fix on Channel.call.
There was a problem hiding this comment.
Fixed together with the cancellation above.
Retire, delete_restored, forget and its check no longer have a fixed timeout: the control call's deadline bounds them, and when it passes the workers' handlers are cancelled and awaited.
A teardown slower than 30 seconds now finishes within a longer deadline instead of returning a 503 while it keeps running.
Covered by test_a_teardown_slower_than_the_message_timeout_finishes_within_the_retire_deadline, scaled down with the fixed timeout patched below the teardown's time; it also deletes a restored session through a commit's cleanup. On the previous head the retire failed with checkpoint_coordinator_unavailable.
| app, | ||
| participant, | ||
| num_workers=self.config.num_workers or 1, | ||
| status=participant.status_extra(), |
There was a problem hiding this comment.
Minor: /status is a frozen snapshot with several workers.
participant.status_extra() is evaluated once at setup in the main process, whose participant never serves sessions. So retired_sessions always reports 0 with num_workers > 1 (verified). The e2e assertion that it's 0 passes regardless. Suggest aggregating status from the workers' reports.
There was a problem hiding this comment.
Fixed. A status request now asks every worker for its participant's status_extra() and readiness, waiting up to two seconds for each, and merges the answers: numbers add up, lists join, and flags hold only if every worker's does.
The status's report merges the workers' readiness the same way outside a checkpoint; while one is open it is the readiness prepare reads, as before.
status_workers says how many workers answered, so a status taken while a worker is unreachable still answers and says so.
Covered by test_status_adds_up_what_every_worker_reports and test_status_answers_without_a_worker_that_does_not; on the previous head retired_sessions was missing from the coordinator's status.
The end-to-end retire test now runs with one and with two workers per server.
It first sends the resources server a retire whose deadline has passed, which only fences the episode's session, and checks that the status counts that session as retired; after the full retire and after the forget it checks for none.
The old assertion, retired_sessions == 0 after the forget, was true whatever the workers held, so it could not catch this; on the previous head the new one fails with two workers per server.
| ) | ||
| if len(live) > 1: | ||
| LOGGER.error("%s session %s is live on several workers; exporting one copy", self.kind, key) | ||
| for worker_index, session_id in stale: |
There was a problem hiding this comment.
Minor: stale-copy deletes are serialized on the export critical path.
Each delete_unclaimed is awaited in sequence (each up to 30s, and each may call a user teardown hook), so export latency is the sum of all deletions. The export task is shielded: after a restore with many re-seeded sessions, the commit caller can hit its deadline while cleanup continues, and a retry then waits for the same export. Suggest batching per worker and running workers concurrently, or otherwise using bounded concurrency.
There was a problem hiding this comment.
Fixed. The coordinator sends each worker one message with all of its stale copies, to every worker at once, and each worker deletes up to 16 at a time.
The export is bounded by the commit's deadline as before.
Covered by test_stale_resources_copies_on_several_workers_are_deleted_at_once and its agent twin, with six teardowns of 0.3 seconds on two workers: the commit takes about one teardown's time. On the previous head it took 1.8 seconds.
| if not self.accepting and not (seed and session_key in self._seeding): | ||
| raise AdmissionClosedError("agent admission is closed for a checkpoint") | ||
| self._add(_Session(key=session_key, episode_id=episode_id)) | ||
| self._add(_Session(key=session_key, episode_id=episode_id, owner=self.owner, session_id=current_session_id())) |
There was a problem hiding this comment.
Suggestion: agent records the serving worker, not the cookie's owner.
Resources sessions keep the owner named in the session cookie; agent sessions record self.owner. If a client keeps an aliased pre-restore cookie and seeds here, the record says Y while the cookie still says X, so after the next restore X may not be aliased and the cookie gets a 410. Current first-party flows do not appear to reuse aliased cookies this way, so this is low reach, not a blocker.
There was a problem hiding this comment.
Done. An agent session now records the owner its cookie names, as a resources session does, and falls back to the serving worker for a cookie without one.
Covered by test_an_agent_session_seeded_with_a_cookie_from_before_a_restore_keeps_the_cookie_s_owner, with real agent servers: a client keeps a cookie from before a restore, seeds a new agent session with it, and the next checkpoint records the pre-restore owner, which its restore aliases again. On the previous head the record named the serving worker.
| # The same encoder as the checkpoint writer: | ||
| # orjson would refuse integers beyond 64 bits and silently turn NaN and infinities into null, | ||
| # so a record could fail to reach a worker, or arrive changed. | ||
| data = json.dumps(message, separators=(",", ":")).encode() |
There was a problem hiding this comment.
Perf follow-up: large JSON frames on the event loop.
Frames up to 1 GB are encoded/decoded with json.dumps/json.loads synchronously on the loop, and the commit collects every worker's export reply before processing any. (The choice of stdlib json over orjson is reasonable per the comment above.) Consider chunking/batching records, moving (de)serialization off the loop, and processing worker replies incrementally.
There was a problem hiding this comment.
Done, keeping the stdlib json encoder.
A worker now encodes its export once, in a thread, into messages of about 2 MiB, and the coordinator checks each one in a thread as it arrives, while the other workers are still exporting.
Frames over 256 KiB are decoded in a thread, and a restore's install messages are encoded in one.
A worker lost between chunks fails the commit, and the coordinator keeps nothing of that export.
Measured with two workers exporting 200 sessions of about 1 MB each (185 MB of records), all in one process so that a stall anywhere shows: the longest event-loop stall during the commit fell from about 760 ms to between 75 and 130 ms, and the commit took about the same time, 2.5 to 2.8 seconds.
With 4,000 small sessions (17 MB), the longest stall fell from about 200 ms to about 160 ms; what remains is per-record work in the participants, which runs on the loop on one process too.
Covered by test_a_large_export_arrives_in_chunks_with_every_record and test_a_worker_lost_between_export_chunks_fails_the_commit_and_keeps_nothing.
|
Thanks for the review. Every item is fixed in this round, in the rebuilt commit of this PR ( Blockers
Fix before merge
Minor and suggestions
Performance
|
| @asynccontextmanager | ||
| async def legacy_run(self, session_key: str, episode_id: EpisodeId) -> AsyncIterator[LegacyRun]: | ||
| """Hold one legacy ``/run`` episode and its turn-loop session.""" | ||
| await self._claim(session_key, episode_id) |
There was a problem hiding this comment.
[BLOCKER] Roll back the claim when open_session() fails
_claim() settles the restored record before open_session() runs. If open_session() rejects the request, execution never enters the try/finally, so the claimed session remains registered, is exported at every later checkpoint, and every retry fails the same way. Please include open_session() in rollback protection and remove/return the claimed session on every failed-open path.
There was a problem hiding this comment.
Fixed. _claim now opens the session inside the claim block, after the agent's restore hook ran.
If open_session() refuses, for example because the /run is of another attempt than the one the record continues, the session is removed, the state the hook installed is released through retire_agent_session, and the record goes back to the coordinator, so the attempt it continues can claim it.
A failed hook already gave the record back; that path is unchanged.
test_a_restored_run_whose_session_cannot_open_goes_back_and_leaves_nothing_on_the_worker runs a /run of attempt 2 against a record that continues attempt 1, then the attempt-1 /run on the other worker.
On the previous head the refused session stayed registered on the first worker and the record was settled as installed, so the record never came back (the wait for it timed out).
Now the first worker keeps nothing, its hook state is released, and the second worker continues from the saved boundary.
Restored agent sessions installed at restore time take the same care with their routing claims: a worker rewrites a restored session's claim only after its install succeeded, and a worker whose install or claim rewrite fails deletes what it installed and releases the claims it rewrote.
| async def open_admission(self) -> None: | ||
| self.accepting = True | ||
| self.request = None | ||
| self.lost_worker = False |
There was a problem hiding this comment.
[BLOCKER] Do not clear restored-worker loss before validating reopen
A worker lost while admission is closed sets lost_worker, but this clears it before open is broadcast and suppresses broadcast failures. If that worker held restored sessions and dies after install but before resume, resume succeeds and silently loses their state. Keep the loss/generation until every expected restored worker has acknowledged reopen, or fail recovery so the affected episodes can restart.
There was a problem hiding this comment.
Fixed. The coordinator now remembers which workers the last restore installed sessions on, until the resume that reopens them.
If one of them is lost before that resume, or does not acknowledge reopening, the resume fails with the new checkpoint_worker_lost error and admission stays closed, and every later resume fails the same way.
Those sessions' state is gone, so the restore has to be repeated on a restarted server; the error says so.
A worker lost meanwhile that held no restored session is handled as before, since it held nothing a checkpoint needs.
test_a_worker_lost_with_restored_sessions_before_the_resume_fails_the_resume restores two agent sessions onto two workers, closes the connection of one holder, and resumes twice.
On the previous head the first resume succeeded (DID NOT RAISE); now both fail with checkpoint_worker_lost.
With the routing redesign in #3905, the requests of a session whose worker died get a 410 from the router rather than running against empty state, so after a successful resume a later loss is visible to the caller too.
|
|
||
| @asynccontextmanager | ||
| async def lifespan_with_coordinator(application: FastAPI) -> AsyncIterator[Any]: | ||
| await link.connect() |
There was a problem hiding this comment.
[RISK] Register only after the private routing socket is ready
This registers the worker before entering the wrapped lifespan, while the outer session-routing lifespan starts its Unix socket only after this lifespan yields. A restore can place sessions on this worker during that window, and callers then receive a misleading 410 because its socket is not listening. Start the socket first or add a routing-ready handshake before registration.
There was a problem hiding this comment.
Fixed by starting the socket first.
A worker of a server that routes sessions now joins its coordinator from inside session routing's lifespan, after its private socket listens and before uvicorn hands it any request.
Session routing enters the lifespans listed in the app state under ROUTED_LIFESPANS_STATE at that point, and the checkpoint link adds itself there. A worker without a private socket, such as an environment server's, still joins before its own startup.
test_a_worker_joins_its_coordinator_only_once_its_private_socket_accepts checks, at the moment each of two real counter server workers registers, that its socket accepts a connection.
On the previous head neither did ([False, False]).
| # With several workers, the token names the worker that holds the session, | ||
| # so a tool call that lands on another worker is routed back here (see nemo_gym.session_routing). | ||
| # Single-worker tokens are unchanged. | ||
| owner = getattr(request.app.state, "nemo_gym_routing_id", None) |
There was a problem hiding this comment.
[RISK] Encode the session's stable owner, not merely the serving worker
After a restore, a request may be served by worker Y while its cookie/session record intentionally preserves pre-restore owner X so the next restore aliases X to the new holder. This token records Y instead; after another restore no alias for Y is guaranteed, so MCP tool calls can route to a dead or wrong worker. Use the cookie/restored owner when present, falling back to this routing ID only for a genuinely new session.
There was a problem hiding this comment.
Resolved by the routing redesign in #3905 rather than here: cookies and MCP session tokens no longer carry a worker ID at all.
They name only the session ID, and the router forwards by that ID's claim, so there is no stable owner to encode and no alias table to keep in step with it.
A restore installs each session on one live worker, which rewrites the session's claim to name itself; every other worker drops any claim it held on that session.
A cookie or token from before any number of restores therefore reaches whichever worker holds the session now.
test_an_agent_session_cookie_from_before_a_restore_reaches_the_worker_that_holds_the_session_now seeds an agent session, checkpoints, restores onto two fresh workers, and sends the old cookie through each worker: both requests are served by the holder.
test_a_restored_resources_session_serves_a_request_with_its_old_cookie_on_any_worker does the same for counter resources sessions, test_after_a_restore_onto_several_workers_every_worker_refuses_the_replaced_attempt_s_mcp_token for MCP tokens, and test_an_old_cookie_follows_its_session_to_the_worker_a_restore_moved_it_to in test_session_routing.py for the claim rewrite itself.
|
|
||
| async def _send_reports(self) -> None: | ||
| while self._report_pending and not self.accepting: | ||
| self._report_pending = False |
There was a problem hiding this comment.
[RISK] Preserve a failed readiness report for retry
_report_pending is cleared before sending, and the participant's _numbered_report() records _last_report before the send succeeds. On failure this returns with neither a pending report nor a changed value for the polling fallback, so prepare can wait until its deadline even though the worker is ready. Mark the report acknowledged only after a successful call, or restore both pending/last-report state and retry with backoff.
There was a problem hiding this comment.
Fixed both halves.
A report that does not reach the coordinator stays pending and is sent again, waiting 0.1 s and doubling up to 2 s, until it arrives or admission reopens.
The worker records a report as the last one sent only after the coordinator received it, so the poll that catches readiness changing without a notification also sends it again.
test_a_readiness_report_that_did_not_arrive_is_sent_again loses the report a worker sends when its last request ends.
On the previous head the following prepare stayed preparing until its deadline; now it is prepared.
| # Bounded by the forget request's deadline. | ||
| await self.broadcast("check_forget", {"rollout_ids": rollout_ids}, timeout=None) | ||
| self.retired.forget(rollout_ids) | ||
| await self.broadcast("forget", {"rollout_ids": rollout_ids}, timeout=None) |
There was a problem hiding this comment.
[RISK] Recover from a partially applied forget
The coordinator clears its mark before broadcasting the mutation. If cancellation arrives after one worker has completed forget but before another has, the deployment is split and the unfenced worker can admit a late request for that rollout. A clean retry normally repairs the state, but the unsafe window remains. Re-mark/reconcile on failure, or make the mutation transactional across workers.
There was a problem hiding this comment.
Fixed. The coordinator now keeps its own marks until every worker has forgotten.
If the broadcast fails or is cut short, it sends every worker the marks again, as a retire does, so a worker that had already forgotten refuses the attempts again.
That repair runs to the end even if the forget is cancelled again, and a retried forget starts from where every worker refuses.
test_a_forget_cut_short_leaves_every_worker_refusing_the_attempts blocks one worker's forget past the request's deadline.
On the previous head the other worker and the coordinator had forgotten ([0, 1, 0]); now all three still refuse ([1, 1, 1]) until a later forget reaches every worker.
| for worker_index, (kind, result) in outcomes.items(): | ||
| if isinstance(result, ControlError): | ||
| # Keep the worker's status and code, such as invalid_phase, so the controller can act on it. | ||
| raise _RemoteControlError( |
There was a problem hiding this comment.
[NOTE] Report worker loss as worker loss, not coordinator unavailability
When a coordinator-to-worker channel closes, Channel.call() produces checkpoint_coordinator_unavailable; this wrapper preserves that 503 code even though the coordinator itself is healthy and a worker was lost. The previous model-worker path returned a lost-worker conflict. Map this direction of channel failure to a dedicated lost-worker error so callers get an accurate, actionable response.
There was a problem hiding this comment.
Fixed. A connection to a worker that closes or times out on the coordinator's side is now WorkerLostError, with code checkpoint_worker_lost, status 409, and the worker's index in the detail.
An error a worker itself reports, including its own loss of the coordinator, keeps its status and code as before.
test_a_worker_lost_in_the_middle_of_a_message_is_reported_as_a_lost_worker closes a worker's connection while it handles delete_restored.
On the previous head the error was checkpoint_coordinator_unavailable.
| # The participant and this worker share one set of retired attempts, which the coordinator keeps current. | ||
| participant.retired = self.retired | ||
| participant.on_change = self.changed | ||
| if hasattr(participant, "claim_restored"): |
There was a problem hiding this comment.
[NOTE] Make participant capabilities explicit
claim_restored (and owner below) is wired only when an attribute happens to exist. A typo, rename, or incomplete participant then silently disables restore/routing behavior. Please make these optional hooks explicit on the participant contract or use typed capability protocols so a missing implementation fails visibly.
There was a problem hiding this comment.
Fixed. claim_restored, on_session_freed (new, below) and fence_retired are now part of CheckpointParticipant itself, with defaults of None and a no-op, and the worker link sets the first two on every participant without checking for them.
The owner hook is gone with the worker IDs in cookies.
The participants that need claim_restored (environment, and agent for its own /run episodes) use the base attribute, so a rename now fails type checks and tests rather than silently disabling the claim.
on_session_freed tells session routing when the resources participant frees a session outside a request, such as a retire or a commit that drops a restored episode, so that session's claim ends too.
test_every_participant_takes_the_hooks_a_worker_link_sets builds a link for a resources, an environment and an agent participant.
On the previous head the resources participant had no claim_restored (AttributeError).
|
|
||
| with pytest.raises(ValueError, match="num_workers=1"): | ||
| agent.setup_webserver() | ||
| app = agent.setup_webserver() |
There was a problem hiding this comment.
[NOTE] Stop the coordinator created by this test
setup_webserver() starts WorkerCoordinator on a daemon thread with its own event loop. Unlinking the socket path does not stop that thread, so the test leaves a live coordinator behind. Add a coordinator shutdown API/fixture cleanup and await or join it here; monkeypatch already handles restoring the environment variable.
There was a problem hiding this comment.
Fixed. WorkerCoordinator.stop() closes the coordinator's server, stops its event loop, joins its thread and removes its socket.
The test now calls it in a finally, and checks that the thread has ended and the socket is gone; the environment variable is left to monkeypatch as you say.
No other test starts a coordinator in the background: the others run their workers with the worker flag set, so they link to a coordinator the test serves and closes itself.
| def _client(self, owner: str) -> ClientSession: | ||
| client = self.clients.get(owner) | ||
| if client is None: | ||
| client = self.clients[owner] = ClientSession( |
There was a problem hiding this comment.
[NOTE] Document why this client is intentionally not shared
This per-owner ClientSession is the correct exception to the usual shared-client rule because it requires a distinct UnixConnector for each worker socket. A short comment stating that constraint would prevent a future refactor from incorrectly replacing it with the application's shared HTTP client.
There was a problem hiding this comment.
Agreed: each forwarding client holds a UnixConnector bound to one worker's private socket, so the application's shared HTTP client cannot replace it.
That client lives in #3905's session_routing.py, which this PR carries unchanged, so the comment went into #3905, next to the clients pools, and this PR carries it from there:
# Not Gym's shared aiohttp client: each pool needs a UnixConnector bound to its claimant's socket.
|
Thanks for the second pass. Every item is addressed in the rebuilt commit of this PR, and each inline reply names its tests and what they showed on the previous head. #3905 now routes by one key, and this PR follows it. Cookies and MCP session tokens no longer carry a worker ID: the router finds a request's session ID and forwards it to the worker that claims that ID. That removed three things from this PR:
The manifest's A restored resources session is open on its holder, as after a seed, so its claim lasts until its close or verify. When the participant frees a session outside a request, such as by a retire or a commit that drops a restored episode, the session's claim ends a window later as well. The inline items
#4690's A new opt-in stress test checkpoints a thousand counter rollouts in flight across 8 resources and 4 agent workers, kills every Gym process, and restores onto 4 and 8 workers, and the reverse. Every rollout finished before the checkpoint or continued with reward 1.0, and no claim or session was left once the close window passed. |
|
🌿 Preview your docs: https://nvidia-preview-pr-3909-04578d21e9fa.docs.buildwithfern.com/nemo/gym |
| f"{self.kind} worker {worker_index} did not complete install: {error.detail}" | ||
| ) from error | ||
| raise ControlError(f"{self.kind} worker {worker_index} did not complete install: {error}") from error | ||
| self._restored_workers = set(placement.values()) |
There was a problem hiding this comment.
[BLOCKER] Record restored holders before dispatching installs
_restored_workers is populated only after every install call has settled. If a worker acknowledges its install and then disconnects while another worker is still installing, leave() cannot recognize it as a restored holder; this assignment then clears the observed loss, and resume can succeed with that worker's state gone. Please clear stale loss state at the start, populate the intended holders before sending, preserve losses observed during installation, and reset tracking appropriately after rollback.
There was a problem hiding this comment.
Fixed. install() now records the workers it is about to install sessions on, and clears any loss left over from an earlier restore, before it sends the first install.
So leave() recognizes a holder lost at any point after that, including one that already acknowledged its install, and the resume that follows fails with checkpoint_worker_lost.
A loss seen during the installation is kept: a successful install no longer resets the tracking.
A failed restore rolls back on every worker and then clears both, since nothing it installed is left to lose.
test_a_holder_lost_after_its_install_while_another_worker_still_installs_fails_the_resume holds one worker's install, lets the other worker acknowledge its own, and then disconnects that worker.
On the previous head the restore succeeded and the resume did too (DID NOT RAISE), with the lost worker's session gone.
| lost = failed & self._restored_workers | ||
| if lost: | ||
| self._restored_lost |= lost | ||
| self.accepting = False |
There was a problem hiding this comment.
[BLOCKER] Roll back a partially reopened resume
If a restored holder fails during reopen_workers(), reachable workers have already reopened, while this branch closes admission only in the coordinator. Those workers continue serving even though resume failed. The checkpoint request was also cleared, so a replacement worker joining the closed coordinator validates None as its close request. Preserve the request until reopen succeeds and re-close or roll back workers that already acknowledged open.
There was a problem hiding this comment.
Fixed. When a worker holding restored sessions does not acknowledge open, open_admission() now closes every worker again under the checkpoint request they were closed for, then raises checkpoint_worker_lost.
That puts the coordinator back in its closed state with the real request, so a replacement worker that joins afterwards closes under that request instead of validating None.
Later resumes keep failing, as before, until the server restarts and the restore is repeated.
A worker that cannot be closed again is logged; it is lost, or the next prepare finds it.
test_a_resume_whose_restored_holder_is_lost_while_reopening_closes_the_other_workers_again makes the holder die while it reopens.
On the previous head the other worker kept serving after the failed resume, and the replacement worker's join raised a ValidationError on None as its CheckpointRequest.
| self.open_session(session_key, episode_id) | ||
| except BaseException: | ||
| if installed: | ||
| await self.hooks.retire_agent_session(session_key) |
There was a problem hiding this comment.
[BLOCKER] Do not return the claim when rollback teardown fails
If retire_agent_session() raises or is cancelled here, _remove() and the restored-episode pop are skipped. The claim context gives the record back while this worker still retains the installed session, so a later checkpoint can save both copies. Guarantee local cleanup, but make the record claimable elsewhere only after external teardown succeeds; otherwise retain or fence ownership, or fail recovery.
There was a problem hiding this comment.
Fixed by keeping ownership of the record until the external teardown succeeds.
The agent's rollback in _claim() now does only the local cleanup, _remove() and popping the restored episode, in its except with no await, so a cancellation cannot skip it.
The teardown of the agent's copy moved into the claim: claim_restored(key, teardown=...).
When the block raises, the worker keeps the coordinator's lease and runs the teardown in the background, retrying with a wait that doubles up to 30 seconds.
Only once the teardown has succeeded does the worker settle the claim, and only then can another worker claim the record.
While the lease is held, prepare waits for it and other claims are told the record is still settling (the next comment), so no checkpoint can contain the record and also find the installed copy.
Why this option: it keeps state bounded and keeps the record.
There is at most one background task per failed claim, and it ends when the teardown succeeds or the worker disconnects.
If the worker is lost or stops, the coordinator returns the record, which is then correct because the copy went with the worker.
A retire of the attempt ends the lease as before, and the teardown still runs to free the copy.
Failing the recovery would strand an episode whose record is fine, and returning the record at once is the bug you found.
The cost is that a teardown that never succeeds blocks prepare, with the attempt named as a blocker, until the controller retires that attempt.
One more case came up while fixing it: with the teardown in the background, a later /run of the same session key on the same worker could open a new session that the late teardown would then free.
A claim of a key whose rolled-back install this worker is still freeing now waits for it, at most MESSAGE_TIMEOUT_SECONDS, and is refused as still settling after that.
test_a_restored_run_whose_teardown_fails_keeps_its_record_until_a_teardown_succeeds runs with a teardown that raises and with a request cancelled during the teardown.
On the previous head both left the session on the worker and returned the record before the teardown had finished.
test_a_run_that_reuses_a_key_whose_rolled_back_install_is_still_being_freed_waits_for_it fails on the previous head because its teardown ran inside the request and never returned.
| if not self.accepting: | ||
| raise AdmissionClosedError(f"{self.kind} admission is closed for a checkpoint") | ||
| key = body["key"] | ||
| record = self.restored.pop(key, None) |
There was a problem hiding this comment.
[Correctness] Treat a leased record as still settling, not absent
A failed claimant returns its record through asynchronous settle_claim, while a new claim is handled immediately. A retry can reach this line while the key is still leased, receive record=None, start fresh, and then have the old record returned. If the key is leased, return a retryable “claim still settling” result instead of treating it as no restored record.
There was a problem hiding this comment.
Fixed. The coordinator now refuses a claim of a leased key with ClaimSettlingError (checkpoint_claim_settling, status 503), instead of answering that there is no restored record.
The worker sends the claim again after 50 ms, doubling the wait up to 1 second, for as long as its caller waits and at most MESSAGE_TIMEOUT_SECONDS; an environment server's claim runs inside the episode's deadline.
After that the retryable error reaches the caller.
Each claim attempt also names its lease with a fresh ID, and settle_claim ends a lease only when the ID matches.
A claim whose reply was lost gives back a lease it may not have got. Now that a lease is kept through a teardown, that settlement could otherwise end a lease the same worker holds for an earlier failed claim still being torn down.
test_a_claim_that_arrives_while_a_failed_claim_settles_waits_and_continues_from_the_record holds the failed claim's settlement while the other worker's /run claims.
On the previous head that /run returned at once and started over without the restored boundary; now it waits and continues from it.
| # The participant freed the session outside a request, which the router would otherwise never see end. | ||
| lifetimes = self._lifetimes() | ||
| if lifetimes is not None: | ||
| lifetimes.ended(session_id) |
There was a problem hiding this comment.
[Correctness] Keep retired-session refusal routable until forget
After retire frees a resources session, this ends its routing lifetime, so its claim is released after the close window. A later cookie-only request can land on another worker, which has no session-ID tombstone and handles it as unclaimed instead of retired. Retain the routing claim until forget, or distribute the retired session-ID tombstone to every worker.
There was a problem hiding this comment.
Fixed by keeping the claim until forget, as you suggested.
When a retire frees a fenced resources session, the participant now calls on_session_freed(session_id, until_forget=True).
The worker then holds the session's claim with ResourcesSessionLifetimes.retain(), whatever later requests for the session do, including the harness's cleanup close.
The forget that removes the session's refusal calls on_session_freed(session_id) again, which ends the session, and the sweeper releases its claim one close window later.
So the claim has the same end of life as the refusal it routes to.
A forget cut short works as round 25 arranged: the coordinator marks every worker again, and a retried forget finishes on the rest.
A worker that had already forgotten has also ended the claim, so after its window a cookie-only request is served like any request after a forget, as on one worker.
test_a_retired_session_s_cookie_is_refused_on_every_worker_until_forget uses two counter server workers with a short close window.
After the retire and well past the window, a cookie-only request through the other worker is forwarded to the holder and refused with stale_attempt, and after the forget the claim is gone and no state is left on either worker.
On the previous head the claim was already gone after the retire, so the request was served on the other worker, which created a new counter for the retired session.
| return | ||
| try: | ||
| os.unlink(self._path(session_id)) | ||
| os.unlink(path) |
There was a problem hiding this comment.
[Correctness] Serialize release with restore rewrite
This ownership check and unlink are a TOCTOU pair. Another worker can rewrite() the claim after the check but before unlink(), allowing the former owner to delete the restored holder's new claim. Another check remains racy; use cross-process serialization or ownership-specific claim files.
There was a problem hiding this comment.
Fixed with serialization across processes.
SessionClaims.release() now checks the claimant and unlinks while it holds an fcntl.flock on a lock file beside the claims directory, and rewrite() replaces the claim file under the same lock.
The lock is held only for those file operations, never across an await.
claim() takes no lock: its hard-link create fails whenever any claim exists, so it cannot replace a rewritten claim.
Cost, over 20,000 releases on one workstation: release went from 15.2 to 15.5 µs at p50 to 19.8 to 20.4 µs.
Only a release that actually removes this worker's claim takes the lock; the check after each request returns before it.
This changes nemo_gym/session_routing.py, which came with #3905.
It belongs in this PR because the race only exists with this PR's rewrite().
test_a_release_cannot_delete_a_claim_a_restore_rewrites_while_it_releases runs the other worker's rewrite() in a thread right after the release's check.
On the previous head the release then deleted the rewritten claim, leaving none.
| self.worker_label, | ||
| exc_info=True, | ||
| ) | ||
| self._channel.close() |
There was a problem hiding this comment.
[Cleanup] Stop worker-link background tasks on clean disconnect
The channel remains non-null after this close, so report and claim-settlement loops continue retrying against it; a readiness poll can also remain alive while the worker is locally closed. Clear the channel and cancel and await the report, poll, and settlement tasks during disconnect.
There was a problem hiding this comment.
Fixed. disconnect() now clears the link's channel first, tells the coordinator it is stopping over its own reference, closes the connection and waits for the reader.
It then cancels and awaits background_tasks(): the report task, and on a participant link also the readiness poll and every claim settlement, including any teardown retry from the claim fix above.
With the channel cleared, a task that wakes before its cancellation finds no connection and stops.
test_a_clean_disconnect_stops_the_worker_link_s_background_tasks disconnects a worker while a checkpoint is open, its report keeps failing, and a claim settlement keeps retrying.
On the previous head all three tasks were still running after the disconnect.
| # Converted and encoded once, off the event loop, into messages of bounded size, | ||
| # which the coordinator checks as they arrive. | ||
| # The exported state does not change until the checkpoint resumes. | ||
| chunks = await asyncio.to_thread(encode_records, records, chunk_bytes=EXPORT_CHUNK_BYTES) |
There was a problem hiding this comment.
[Perf, non-blocking] Stream export chunks as they are encoded
Although encoding is off the event loop, encode_records() builds the complete serialized export before this loop sends the first chunk. A large export therefore exists as both record objects and a full encoded checkpoint. A bounded producer/consumer pipeline would keep only a few chunks in memory and overlap encoding, IPC, and validation.
There was a problem hiding this comment.
Done. encode_records() is now a generator, and the worker encodes one message at a time in a thread.
It starts encoding the next message before it sends the current one, and the coordinator checks each message before it answers, so encoding, sending and checking overlap.
The worker holds at most two encoded messages instead of the whole export.
Measured with round 24's synthetic export (1,000 agent sessions, 177 MB of records, two workers and the coordinator in one process), five alternating runs each:
- peak resident memory above the level before the commit: 783 to 788 MB before, 635 to 649 MB now;
- commit time: median 4.44 s before (2.64 to 6.47 s), median 3.23 s now (3.12 to 6.88 s), on a shared machine whose load varied a lot;
- longest event-loop stall: median 114 ms before, 60 ms now.
In separate processes the saving lands on the worker, which no longer builds a full encoded copy of its export.
test_an_export_sends_its_first_chunk_before_it_has_encoded_every_record counts the records encoded when the first message goes out.
On the previous head all 20 were; now at most a few are.
|
@macandro96 thanks for the third pass. All eight items are addressed in the rebuilt commit, which is now on the current
|
| for worker_index, placed in installs.items() | ||
| } | ||
| outcomes = await self.call_each_settled(messages, timeout=None, bulk=True) | ||
| failed = {index: result for index, (_, result) in outcomes.items() if isinstance(result, BaseException)} |
There was a problem hiding this comment.
[BLOCKER] Roll back a restore whose holder was lost during installation
A holder can acknowledge its install and then disconnect while another worker is still installing. leave() records that loss, but if every install RPC eventually reports success, this path leaves _restored_lost set. When the controller has already timed out the restore, its failed-restore reopen and every later resume then raises WorkerLostError, leaving the server closed until restart. Treat a holder recorded lost during installation as an install failure, uninstall the surviving workers, and clear the restore-tracking sets before reopening; simply clearing the flag and exposing partial state would not be safe.
There was a problem hiding this comment.
Fixed. install() now treats a holder that leave() recorded as lost during the installation as an install failure, even when every install message reported success.
The workers still connected delete what they installed and release the claims they rewrote, the restore-tracking sets are cleared, and the restore fails with checkpoint_worker_lost. The controller's failed-restore reopen then finds nothing left to lose, so admission reopens and later resumes succeed.
One window remained: a restore that ran past its deadline, whose install then finished on every worker, and whose holder was lost after that but before the failed restore's reopen ran. install() has already returned there, so the control plane now calls a new participant hook, discard_failed_restore(), before it reopens after a failed restore. The coordinated participant rolls the restore back the same way when a holder was lost, and drops the records it holds. Without such a loss it keeps the installed state, which the controller's retire frees, as on one process. The default hook does nothing, so single-process participants are unchanged.
Tests, each failing on the previous head:
test_a_holder_lost_after_its_install_while_another_worker_still_installs_fails_the_restore: the previous head's restore succeeded (DID NOT RAISE).test_a_restore_past_its_deadline_whose_holder_was_lost_while_installing_reopens_admissionis your scenario. On the previous head the failed-restore reopen raisedWorkerLostError, and the surviving worker never reopened.test_a_holder_lost_after_an_install_that_outlived_its_restore_does_not_keep_admission_closedholds the control lock while the install finishes and the holder is lost. It failed the same way on the previous head.
|
|
||
| async def reopen_workers(self) -> set[int]: | ||
| """Reopen admission here and on every worker; return the workers that did not acknowledge it.""" | ||
| self.accepting = True |
There was a problem hiding this comment.
[BLOCKER] Keep admission closed until every worker is ready to reopen
This opens the coordinator before all open calls have succeeded, and each worker's open_local() starts accepting before it sends its acknowledgment. If another restored holder then fails, _close_again() eventually closes the healthy workers, but requests can be admitted and complete during that window. Please use a two-phase reopen or a shared admission gate so workers become externally available only after every required acknowledgment succeeds.
There was a problem hiding this comment.
Fixed with a two-phase reopen in CoordinatedParticipant.reopen_workers().
In the first phase every worker answers a prepare_open message and stays closed. The coordinator stays closed too, and keeps the checkpoint request, so a worker that joins meanwhile closes under it. A new check_reopen() hook then decides on the workers that did not answer. The server participant raises checkpoint_worker_lost there if a worker holding restored sessions did not answer, or was recorded lost, and no worker has reopened at that point. Only after that does the coordinator open and send open to every worker connected by then. So a worker's open_local() never runs before every required acknowledgment has arrived, and _close_again() is gone.
Each phase is bounded by MESSAGE_TIMEOUT_SECONDS. The first phase is also cancelled by the deadline of the resume it serves, which leaves every worker closed. Once the second phase has started, it runs to completion even if its caller is cancelled. Otherwise some workers would serve while others stay closed. A holder lost during the second phase, after every worker said it was ready, is lost as it would be just after the resume: the other workers serve, and that worker's sessions get a 410.
Tests:
test_no_worker_reopens_until_every_worker_holding_restored_sessions_canmakes the holder wait for the other worker to start serving before it dies. On the previous head the other worker was already serving at that moment; now it stays closed, and so does a replacement that joins.test_a_holder_lost_after_every_worker_is_ready_to_reopen_is_lost_as_after_the_resumecovers a loss in the second phase. On the previous head the resume failed and closed the other worker again.
| wait = _SETTLE_RETRY_SECONDS | ||
| while teardown is not None: | ||
| try: | ||
| await teardown() |
There was a problem hiding this comment.
[Resilience, non-blocking] Bound a stuck teardown without releasing its lease
If this hook never returns, the settlement task and its lease remain forever. That lease keeps every checkpoint unready, while later /run requests for the same key repeatedly time out in _wait_torn_down(). Consider a bounded, cancellation-safe retry or a policy that fences/restarts the worker after a teardown exceeds its limit. The record must remain unclaimable until teardown is confirmed, so timing out must not simply return the lease.
There was a problem hiding this comment.
Done as a bounded retry that keeps the lease.
ParticipantWorkerLink._tear_down() cancels each teardown attempt after _TEARDOWN_ATTEMPT_SECONDS (60 s), and tries again after a wait that grows to 30 s. A failure or a timeout counts as a failed attempt, and so does a hook that raises CancelledError by itself. Only cancelling the settlement task stops it, as a disconnect does.
The worker never gives the lease back while the teardown has not succeeded. So the record stays unclaimable, and prepare keeps naming the attempt as a blocker, until a teardown succeeds, the controller retires the attempt, or the worker leaves. The worker is not restarted.
A later /run of the same key on that worker waits in _wait_torn_down() for up to MESSAGE_TIMEOUT_SECONDS. Then it fails with the retryable checkpoint_claim_settling (503) instead of hanging. Gym's /run carries no deadline of its own, so that timeout and the caller's cancellation are what bound it. The behavior is documented in the claim() and _tear_down() docstrings and in the module docstring.
This turned up a related gap. A reopen sent the workers only the unleased restored keys, so a record that was leased during a checkpoint, and that went back after its teardown succeeded, could no longer be claimed on any other worker. The replacement then started over without its restored state. Leased keys now stay in the claimable set that a reopen or a join hands out.
test_a_stuck_teardown_keeps_its_record_blocks_prepare_by_name_and_a_run_of_its_key_gives_up keeps the teardown hanging. It checks that the teardown is cancelled and retried, that prepare reports r-a1 as a blocker, that a /run of the key gets ClaimSettlingError, and that the record cannot be claimed until the teardown succeeds, after which the other worker continues from it. On the previous head the first attempt never ended.
There was a problem hiding this comment.
A follow-up on the end of life of a stuck teardown. The teardown used to keep retrying even after the controller's retire had dropped the record's lease, which left one task per stuck teardown running for the life of the worker.
While the lease is held, nothing changes: the worker keeps retrying, the record cannot be claimed, and prepare names the attempt. After each failed attempt, the worker now asks the coordinator whether it still holds the lease. Once the lease is gone, for example after a retire, the teardown gets _TEARDOWN_ATTEMPTS_WITHOUT_LEASE (3) more attempts. If none of them succeeds, the worker logs an error that names the session and the attempt and says that the external resource may be orphaned, then drops the task and the session's entry. A teardown that succeeds within those attempts settles its claim as before.
A cancelled attempt now gets _TEARDOWN_CANCEL_GRACE_SECONDS (10 s) to stop. If the hook ignores the cancellation, the worker logs it and stops waiting. While that hook call is still running, the worker does not start another one: the next retry waits for the same call again. When the attempts run out, the worker drops its reference to the call.
Tests:
test_a_teardown_whose_lease_a_retire_ended_stops_within_its_cap[always_fails]makes every teardown attempt fail, then retires the attempt. It checks that the teardown stops within the cap, logs the error, sends no settlement, and leaves no task or entry behind. On the previous head it kept retrying.test_a_teardown_that_ignores_cancellation_is_no_longer_awaited_once_its_lease_is_goneuses a hook that swallows its cancellation. On the previous head the worker waited for that hook forever. Now the worker stops waiting once its attempts run out, and it never starts a second copy of the call.[succeeds_after_the_lease_ends]checks that a teardown that succeeds after the retire still settles its claim. It passes on the previous head too, which retried until success. It guards against the cap cutting a success short.
| lease = self.leases.get(body["key"]) | ||
| # Only the claim that holds the lease settles it. | ||
| # A claim whose reply was lost settles a lease it never got, which changes nothing. | ||
| if lease is not None and (lease.worker_index, lease.lease_id) == (worker_index, body["lease"]): |
There was a problem hiding this comment.
[Test] Exercise a stale settlement from an older lease
This lease-ID comparison is important but has no targeted regression test. Please cover the ordering where settlement A is processed but its reply is lost, another worker obtains lease B, and the retry of settlement A arrives afterward. The test should verify that stale settlement A cannot settle or return lease B.
There was a problem hiding this comment.
Added test_a_settlement_retried_after_another_claim_leased_the_record_changes_nothing, parametrized on whether another worker or the same worker takes the second lease.
The first claim fails, and its settlement reaches the coordinator but its reply is lost. The worker's retry is held until the second claim holds lease B. The test then checks that the retry leaves lease B in place and does not return the record, and that the second claim's own settlement ends lease B with the record installed.
It passes on the previous head too, since the lease-ID check was already there. To show that it guards the check: comparing only the worker index fails the same-worker case, and dropping the comparison fails both.
|
|
||
| run = asyncio.create_task(run_of_a_later_attempt()) | ||
| await tearing_down.wait() | ||
| if failure == "is_cancelled": |
There was a problem hiding this comment.
[Test] This branch does not cancel the teardown
By the time tearing_down is set, teardown is running in an independent _settling task and the foreground run task is already unwinding or complete. Cancelling run therefore does not cancel the teardown, and this variant makes no assertion that distinguishes cancellation. Please explicitly cancel/disconnect the tracked teardown task, or make the hook raise CancelledError, and assert the intended lease and cleanup behavior.
There was a problem hiding this comment.
Agreed, that variant only cancelled the /run, after the teardown had moved to its own task. It is replaced by two variants that reach the teardown itself:
- the hook raises
CancelledErroron its own. On the previous head this ended the settlement task, so the record was never returned while the worker stayed connected (the test timed out). Now it counts as a failed attempt and is retried; - the hook hangs, and the attempt timeout cancels it. The test checks that the hook saw the cancellation and that the next attempt ran. On the previous head the first attempt never ended.
In both, the record cannot be claimed and the session is gone from the worker until a later attempt succeeds. The other worker then continues from the record. Explicitly cancelling the tracked task is covered by test_a_clean_disconnect_waits_a_bounded_grace_for_a_teardown_in_flight[outlasts_the_grace], which checks that the hook is cancelled and that the coordinator takes the record back as the worker leaves.
| await asyncio.gather(self._reader_task, return_exceptions=True) | ||
| tasks = self.background_tasks() | ||
| for task in tasks: | ||
| task.cancel() |
There was a problem hiding this comment.
[Resilience, optional] Give external teardown a graceful-shutdown window
For ParticipantWorkerLink, these background tasks include a claim's external teardown. Cancelling them immediately on a clean disconnect can leave a sandbox or other external resource alive, while leave() returns the record for installation elsewhere. Checkpoint ownership remains consistent, but the external resource may be orphaned or duplicated. If graceful shutdown is expected to clean such resources, await teardown for a bounded grace period before cancelling, or document and provide recovery for this tradeoff.
There was a problem hiding this comment.
Done. ParticipantWorkerLink.disconnect() now waits up to _TEARDOWN_GRACE_SECONDS (30 s, a documented constant) for the teardowns still running before the base disconnect cancels the background tasks.
The wait comes before the worker announces it is stopping and before the connection closes. So a teardown that finishes within the grace sends its settlement over the open connection, and the record goes back with no lease left for leave(). A teardown still running when the grace ends is cancelled, and leave() then returns the record as for any departing worker. The docstring states the tradeoff: the checkpoint stays consistent, but what the install built outside the participant, such as a sandbox, may outlive the worker alongside the copy that the next claim installs elsewhere.
test_a_clean_disconnect_waits_a_bounded_grace_for_a_teardown_in_flight records the leases the coordinator holds when the worker leaves. In the variant that finishes within the grace, the teardown completes and no lease is left. In the variant that outlasts it, the hook is cancelled at the grace, and the lease is still there for leave() to return. On the previous head the teardown was cancelled at once in both.
|
@macandro96 thanks for the thorough review.
|
macandro96
left a comment
There was a problem hiding this comment.
[P1, pre-existing prerequisite] Finish failed-restore recovery before admitting another checkpoint
The missing _reopen_pending guard already exists on main; this PR did not introduce it. However, the new multi-worker lost-holder handling adds a permanent-closure failure mode: a new prepare can overtake pending failed-restore recovery, after which a lost restored-state holder can make every subsequent resume fail while surviving workers remain closed.
Please either fix this here or land a prerequisite fix before merging this PR. While _reopen_pending is true, a new checkpoint must not begin; resume should finish the failed-restore recovery first. As defense in depth, _reopen() should discard pending failed-restore state before reopening admission.
| {worker_index: ("uninstall", {"scope": scope}) for worker_index in workers}, timeout=None | ||
| ) | ||
| for worker_index, (_, result) in outcomes.items(): | ||
| if isinstance(result, BaseException): |
There was a problem hiding this comment.
[P1] Keep admission closed when rollback does not complete
_uninstall() only logs worker cleanup failures. install() then clears the restore tracking and failed-restore recovery can reopen admission, even though a surviving worker may still hold state installed by the failed restore. I reproduced this with one worker failing its install and another worker failing its rollback hook: the restore returned an error, but the latter worker reopened with the restored session still present.
Please propagate and retain rollback failures instead of calling _end_restore() unconditionally. Admission should remain closed, and cleanup should be retried, until every installed copy is removed or the affected worker/server is replaced.
There was a problem hiding this comment.
Fixed. A failed rollback is now kept rather than only logged, and admission stays closed until it is repeated successfully.
Before the coordinator asks the workers to delete what a failed restore installed, it records each of them as still holding that state, with the restore's scope. It clears a worker only when that worker answers that it deleted everything. So a worker whose rollback hook fails stays recorded, and so does every worker if the rollback itself is cancelled.
Every reopen now starts by repeating the rollback on the recorded workers: CoordinatedServerParticipant.open_admission() calls _finish_rollback() before the two-phase reopen. It retries with a wait that grows from 0.1 s to 2 s, for at most 30 seconds or the caller's deadline, whichever comes first. If a rollback is still incomplete then, it raises the new retryable checkpoint_rollback_incomplete (503), and no worker has reopened. That covers the failed restore's own reopen, a resume that retries it, and a lease expiry. discard_failed_restore() repeats it too, so a restore retried after the failed one does not install anything until the earlier rollback has finished.
The record ends when the rollback succeeds on that worker, or when the worker is lost, since what it held went with its process. A replacement worker joins with nothing installed. If neither happens, admission stays closed and each resume tries again.
Tests, each failing on the previous head:
test_a_rollback_that_fails_on_a_worker_keeps_every_worker_closed_until_it_is_repeatedis your scenario: one worker fails its install and the other fails its rollback hook. The test checks that neither worker serves, the session and its claim are still on the second worker, and the resume fails withcheckpoint_rollback_incomplete. Once the hook works, the next resume rolls back and reopens every worker, with no session or claim left. On the previous head the resume returned (DID NOT RAISE), and the second worker was already serving with the restored session.test_a_worker_lost_while_its_rollback_is_retried_lets_admission_reopen: the worker that cannot roll back is lost, and the next resume reopens the other one. On the previous head the other worker was already serving before the loss.
An adversarial pass, outside the PR, also covered rollbacks failing on two of three workers and recovering one at a time, a worker lost in the middle of a repeated rollback, and a restore retried while a rollback is incomplete, which fails until the rollback succeeds.
| key, | ||
| episode, | ||
| ) | ||
| return False |
There was a problem hiding this comment.
[P2] Keep the key fenced while an abandoned teardown is still running
When a teardown ignores cancellation, this returns False after the lease ends even though the child attempt is still running. _settle() then removes the key from _tearing_down, so another /run can open the same key. The abandoned teardown can subsequently finish and free the new session because hooks such as retire_agent_session() operate only on session_key.
Please retain a per-key fence until the child task actually terminates. If it never terminates, the key must remain unavailable or the worker must be replaced.
There was a problem hiding this comment.
Fixed. When a teardown is given up while its call is still running, because the call ignored its cancellation, the worker keeps the key fenced until that call actually ends.
The fence is a per-key entry, _fenced, that holds the call's task, and the task's done callback removes it. claim() waits for both a teardown still running and a fence on the key, for at most MESSAGE_TIMEOUT_SECONDS in total, as before. After that it raises the retryable checkpoint_claim_settling. So a late teardown can no longer free a session that a later /run of the same key opened on that worker.
When a teardown is given up while its call still runs, the worker logs an error that names the key and says it cannot be claimed on this worker until that call ends. A call that never ends keeps the key unavailable on that worker for the worker's life. That is the chosen tradeoff over restarting the worker, and _fence() documents it. A disconnect that cancels a teardown fences the key the same way, but does not wait for the call. The cap from the last round is unchanged.
test_a_key_whose_given_up_teardown_still_runs_cannot_be_claimed_on_its_worker_until_that_call_ends gives up a teardown whose call swallows its cancellation. A /run of the key on that worker then gets checkpoint_claim_settling. Once the call ends, a /run of the key opens normally, and the fence is gone. On the previous head the /run opened a new session while the old call was still running (DID NOT RAISE).
| await asyncio.shield(remark) | ||
| raise | ||
| # Kept here until every worker forgot, so a worker that joins meanwhile still refuses the attempts. | ||
| self.retired.forget(rollout_ids) |
There was a problem hiding this comment.
[P2] Make forget linearizable with worker joins and older repairs
The forget broadcast uses a snapshot of workers. A worker that joins after that snapshot can inherit the current coordinator mark but miss the forget; this line then clears the coordinator mark, leaving only the new worker refusing the rollout. Separately, if a failed forget is cancelled again while its shielded repair is pending, a retry can complete successfully before the older repair marks the workers again. I reproduced both coordinator/worker divergences.
Please version mark/forget state, or otherwise serialize membership changes and overlapping repair tasks, so a successful forget cannot be missed by a joining worker or overwritten by an older repair.
There was a problem hiding this comment.
Fixed by ordering forget against both, rather than by versioning the marks.
Joins: the coordinator sends the forget to the workers it knows, then again to any worker that joined since, until a round finds no new worker. It then drops its own marks with no await in between. A worker that joins before that point adopts the marks and is sent the forget, and one that joins after it finds no mark. A new worker holds nothing of those rollouts, so it can always forget them, and the loop ends once workers stop joining.
Repairs: a forget first waits for every repair still running from an earlier forget that was cut short. Each repair is bounded by MESSAGE_TIMEOUT_SECONDS per worker, and the wait is bounded by the forget's own deadline. Forgets already run one at a time under the control lock, so with this no older repair can land after a later forget. Repairs and retire marks only ever raise a mark, so their order among themselves does not matter.
Versions would have to travel to every worker and be checked there, for the same guarantee.
Tests, each failing on the previous head:
test_a_worker_that_joins_while_a_forget_runs_forgets_too: a worker joins while another worker is still forgetting. On the previous head it kept refusing the rollout after the coordinator forgot it ([0, 0, 1, 0]).test_a_forget_retried_while_the_repair_of_one_cut_short_runs_waits_for_that_repairreproduces your second case. The repair is held on the worker that had already forgotten, and the forget is cancelled again. On the previous head the retry finished first, and the repair then left that worker refusing ([1, 0, 0]).
The adversarial pass also covered a worker that joins while a repair runs, and a forget of another rollout while a repair runs. That forget waits for the repair, and the repaired rollout stays refused everywhere.
| claims.disown(session_id) | ||
| lifetimes = self._lifetimes() | ||
| for session_id in sorted(installed): | ||
| claims.rewrite(session_id) |
There was a problem hiding this comment.
[P3] Batch restored-session routing rewrites
This performs a synchronous temporary-file write, lock acquisition, and atomic replacement for every restored session on the event loop. In a local 12,000-session microbenchmark, _route_restored() took about 3.1 seconds, during which this worker could not promptly serve other asynchronous work.
Please consider a batched SessionClaims rewrite operation that holds the shared lock once for the batch, or move the complete batch off the event loop with carefully managed ownership of _mine.
There was a problem hiding this comment.
Done, both ways. A new SessionClaims.rewrite_all() writes the temporary claim files and replaces them in a thread. It holds the shared lock once per 256 renames, so a release on another worker waits for at most one batch, not the whole restore. _route_restored() now awaits it.
Ownership of _mine:
- The thread touches only files.
_minechanges only on the event loop, after the thread has ended, and then gains exactly the IDs the thread replaced. If the caller is cancelled meanwhile, it still waits for the thread, updates_mine, and then passes the cancellation on, so the rollback that follows releases every claim the thread wrote. - Before the batch starts,
_route_restored()marks each installed resources session open. An agent session is already held by its installed record. Soholds()is true for every batched ID, and no release on this worker removes its claim, during the batch or after it. - A release on this worker of an ID that is not in
_mineyet does nothing. A release on another worker still checks, under the same lock, that the file names it. - A claim on this worker during the batch either finds the old claim and forwards there, as before the restore, or creates one naming this worker, which the rename keeps.
Measured with _route_restored() alone at 12,000 resources sessions, on this workstation's ext4 /tmp, five runs each:
_route_restored() (s) |
Longest event-loop stall (ms) | |
|---|---|---|
| Previous head | 0.44 to 0.47 | 441 to 469 |
| This change | 0.41 to 0.43 | 6.7 to 8.5 |
This filesystem is faster than the one you measured on, where the same work took about 3.1 s. The stall before the change is the whole call either way, because it ran on the event loop.
The total stays close, since it is file work. What changes is that the event loop keeps running during it.
Tests, each failing on the previous head:
test_restoring_many_sessions_rewrites_their_claims_without_holding_up_the_event_looprestores 200 sessions with each rename slowed to 5 ms. It checks that every claim names its holder and that the event loop never stalls for 200 ms. On the previous head the loop stalled for 1.05 s.TestCheckpointRestores::test_a_batched_rewrite_whose_caller_is_cancelled_finishes_before_the_worker_counts_its_claimscancels the caller part way through. No claim counts as this worker's while the thread runs, and every claim does once it ends. The previous head had no batched rewrite.
The adversarial pass also raced a batch against a former claimant releasing the same IDs from a thread, and against claims and releases on the holder's own event loop. Every claim ended up naming the holder.
A failed restore's rollback still releases its claims one at a time on the event loop. That happens only on the failure path, while admission is closed, and is not batched here.
|
@macandro96 thanks for the fifth pass. All five items, including the review summary's prerequisite on
The stack is rebuilt on the current |
macandro96
left a comment
There was a problem hiding this comment.
mostly lgtm! just one low priority race that is cheap to fix
| batch: list[tuple[str, str, str]] = [] | ||
| for session_id in session_ids[start : start + _REWRITE_BATCH]: | ||
| path = self._path(session_id) | ||
| temporary = f"{path}.{self.routing_id}.tmp" |
There was a problem hiding this comment.
[P3] rewrite_all and claim() share one scratch file name, so a request during restore can abort the restore.
This temporary name is f"{path}.{self.routing_id}.tmp", the same name claim() uses (L264) for the same worker and session. Now that _replace_claims runs in a thread, the two can overlap:
- The restore thread writes
X.<worker>.tmpfor a restored sessionX. - A seed or close naming
Xin its body reaches this worker. The router callsclaim(X)before any admission check, andXis not in_mineuntil the thread ends.claim()truncates the same file, and itsfinallyunlinks it. - The thread's
os.replaceraisesFileNotFoundError,_route_restoredfails, and the whole restore rolls back.
The trigger is narrow: it needs a stale or out-of-band seed or close for a restored session during the install. The impact is a failed restore, not corrupted state, and a retry recovers. As a side effect, claim() can also link a scratch file the thread just truncated, so a reader briefly sees an empty claim.
Suggested fix: give each operation its own scratch file, for example f"{path}.{self.routing_id}.{uuid4().hex}.tmp" in both places, and unlink any leftover in a finally in _replace_claims. The final steps stay atomic and agree on the owner (os.replace vs os.link from the same worker), so nothing else changes.
There was a problem hiding this comment.
Fixed in 989f513a33, as suggested.
claim() and _replace_claims() each write their own scratch file, <claim path>.<worker>.<random suffix>.tmp, so a claim of a restored session on this worker during a restore's batched rewrite can no longer truncate or remove the rewrite's file.
_replace_claims() also removes any of its scratch files that a batch cut short left behind.
The final steps are unchanged: os.link for a claim and os.replace for a rewrite, both naming the same worker.
The suffix comes from secrets.token_hex rather than uuid4, so the scratch names stay independent of how the router mints session IDs.
Covered by test_a_claim_on_this_worker_during_a_batched_rewrite_of_the_same_id_does_not_fail_the_rewrite.
It claims the session on the same worker at the moment the rewrite takes the claims lock, which is the interleaving you described.
It checks that the rewrite completes, that the claim names this worker, and that no scratch file is left.
On the previous head it fails with FileNotFoundError from the rewrite's os.replace.
…ral workers Resources, agent, and environment servers may run several uvicorn workers with checkpointing on. Each session's requests reach the worker that holds it, and each server is one participant across its workers through a coordinator in the main process, shared with the policy model. The worker count may change between checkpoint and restore. Gym servers listen with a configurable uvicorn_backlog, default 16384. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
What changed and why
This PR builds on #3905's session routing, which has merged.
Until now, resources, agent, and environment servers refused
num_workers > 1when checkpointing was on. uvicorn's workers share one port, so a control call reaches one arbitrary worker. If each worker ran its own participant, a checkpoint would close, export, and restore only that worker's share of the episodes and sessions. Production servers run several workers (swe_rebenchruns 8, several sandboxed agents run 4), so this guard kept them out of partial-rollout checkpoints.This PR makes each of these servers one participant across its workers:
A generic worker coordinator. The policy model server already runs as one participant across workers. The parts of it that do not depend on the policy model move to
nemo_gym/_checkpoint/workers.py:jsonencoder so NaN, infinities, and integers beyond 64 bits arrive unchanged, and answers a reply it cannot frame with a typed error. A message's send and reply share one timeout. A timeout or transport error on a worker's side isCoordinatorUnavailableError; on the coordinator's side it means the worker was lost, and isWorkerLostError(checkpoint_worker_lost), while an error a worker reports keeps its own status and code. Retire,delete_restoredandforgetare bounded by the request's deadline instead of that fixed timeout. A caller that stops waiting, on a timeout or a cancellation, sends the worker a cancel and waits until the worker's handler has stopped, so a retry never overlaps the operation it replaces. Exports travel in chunks of about 2 MiB, each encoded off the event loop while the one before it is sent and checked, and large frames are encoded and decoded off the event loop;delete_restoredinstead of a retire, so a replacement's sessions on other workers are not freed and no fence is left behind;WorkerCoordinator.stop(), for a test that starts a coordinator in the background;The policy model keeps only its own state, and its behavior and tests are unchanged.
Prepare closes every worker. It is ready once every worker reports ready, and its restarts are the union of every worker's. While a checkpoint is open, a worker re-reports its readiness on a short poll, so readiness that changes without a notification still reaches the coordinator. One example is a resources session that ends.
Commit asks every worker for its records. A worker whose readiness regressed after its last report refuses with
checkpoint_invalid_phase, and a worker's typed error keeps its status and code through the coordinator. The coordinator writes the participant's one records file and manifest, so the store format and the controller's view do not change. Every session whose cookie may outlive a restore exports a record: a stateless resources session exports one without state, so a restore places it on a worker and tracks it until it ends.Restore of sessions that span requests (resources sessions and agent sessions on the typed
single_agent_turnrequest path). fix(server): route a session's requests to the worker that holds it #3905 routes every request by one key, the session ID, through the session's claim, and cookies and MCP session tokens carry no worker ID. A restore therefore only has to move claims:SessionClaims.rewrite_all), holding the shared lock once per 256 renames, so a restore of thousands of sessions does not hold up the worker's event loop. Every other worker drops any claim it held on a restored session (SessionClaims.disown), for example one a cleanup close made before the restore. A worker releasing a claim checks that the file still names it and removes it while holding a lock file the rewrite also holds, so a release never removes a claim a restore moved, even one moved between the check and the removal.resources_session_id, so its seed and close reach the same worker.on_session_freed, so the session's claim ends a window later too. A session a retire frees stays refused until the controller forgets its rollout, so its claim lasts until that forget, and a late request that carries only its cookie still reaches the worker that refuses it.checkpoint_worker_lost. Those sessions' state is gone, so the restore must be repeated on a restarted server, rather than continue those episodes without it. Because the workers reopen in two phases, such a worker lost before every worker confirmed it is ready fails the resume with no worker reopened, and a worker that joins meanwhile closes too. One lost after every worker confirmed is lost as it would be just after the resume: its sessions get a 410.Restore of state that lives inside one request (environment episodes and an agent's own
/runepisodes):/runclaims its record, and exactly one claim succeeds.Restored records are leased until installed. A worker's claim leases a restored record. The coordinator drops it only when the worker confirms the install, and takes it back on a failed install, a session that cannot open (for example a
/runof another attempt than the one the record continues), a lost reply or a lost worker, so the next commit still exports it. Prepare waits while a lease is open./runof the key opens there. A call that never ends keeps the key unavailable on that worker for the worker's life. A later/runof the same key on that worker waits up to 30 seconds for the teardown, then gets the retryablecheckpoint_restored_claim_settling. A worker that stops cleanly waits up to 30 seconds for the teardown. Past that it cancels the teardown, and the coordinator takes the record back as the worker leaves, so what the install built outside the agent, such as a sandbox, may be left behind.checkpoint_restored_claim_settling, since the record may be on its way back, and the worker claims again with a growing wait, bounded by its caller's deadline.A restore is all or nothing across workers. If one worker's install fails, or a worker the restore installs sessions on is lost before every install has finished, even after it acknowledged its own, the others delete what they installed, and release the claims they rewrote, before the restore fails and admission reopens. A restore that failed while its install still ran, and whose install then finished but lost such a worker, is rolled back the same way, through the participant's
discard_failed_restore()hook, before admission reopens. A worker whose rollback fails is kept on record as still holding the failed restore's state, and so is every worker if the rollback is cancelled. Every reopen first repeats the rollback on those workers, for at most 30 seconds or the caller's deadline. While one is still incomplete, no worker reopens, and the resume fails with the retryablecheckpoint_rollback_incomplete. The record ends when the rollback succeeds there or that worker is lost. The contract is in the coordinator'sinstall()docstring.Status covers every worker.
/statusmerges every worker's participant status and readiness, instead of the main process's idle participant.Retire and forget reach every worker. The coordinator marks a retired attempt on every worker before any of them stops it, and relays
forget, which also clears a resources worker's retired sessions. Forget is all or nothing across workers: the coordinator first asks every worker whether it can forget, and a worker that still tracks a retired session, or still has a seed of a retired attempt in flight, refuses withcheckpoint_retire_incompletebefore any worker or the coordinator forgets anything. The coordinator keeps its own marks until every worker has forgotten. A worker that joins while the forget runs adopts those marks, so it is sent the forget too, and the coordinator drops its marks only once no new worker has joined. If the forget fails or is cut short part way, the coordinator marks every worker again, so a worker that had already forgotten refuses the attempts until a forget reaches every worker. The next forget waits for that repair to finish, so the repair cannot refuse the attempts again after it. A worker that registers later receives the current marks. A worker applies a mark through a newWorkerLink.mark_localhook, which records it in the worker's retired attempts. The policy worker (PolicyWorkerLink) overrides it to also refuse, for good, its calls of those attempts that are waiting at closed admission, as feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883's policy gate does in one process. Marking is its own message, so the commit's cleanup of unclaimed restored state, which retires without marking, leaves no refusal on the workers. The coordinator retires the unclaimed records of retired attempts, and exports unclaimed records only while the commit's scope names their replacement attempt.One record per session. Each participant kind defines a record's identity: the session key for agent sessions and the session ID for resources sessions. Two agent
/runepisodes sent with one cookie therefore stay two records. Reporting and freeing stale restored copies is part of the participant contract, which the agent participant now implements too, and stale copies are deleted in one batch per worker, on all workers at once, a bounded number at a time. Routing sends a restored session's seed to its holder, so a second copy should not arise; if one does, a commit exports one record per session ID, preferring the copy a request has used, and frees a stale restored copy that no request has used on its worker.Restored state no longer continued is retired on every worker. The coordinator gathers the unclaimed restored records and each worker's unclaimed restored sessions, so a commit whose scope leaves them out retires them.
Carried restarts reach every worker. The coordinator passes the restarts an earlier prepare stage reported (feat(checkpoint): add the resources server participant with asynchronous session hooks #3886) to every worker, including one that joins while the checkpoint is open, and each worker clears them when it reopens.
The commit reply keeps each participant kind's own fields. With several workers the coordinator answers the commit, so it builds the reply from the participant kind it stands in for over the merged records. A field a kind adds to its reply, such as the agent's list of the episodes its own
/rundrives (feat(checkpoint): name the episodes an agent's own /run drives in the agent commit reply #4690), comes through as it does with one worker.Participant capabilities are explicit.
claim_restored,fence_retiredandon_session_freedare part ofCheckpointParticipant, with defaults, and the worker link sets them on every participant, rather than only where an attribute happens to exist.Fail closed. Fewer registered workers than
num_workersblocks prepare. So does a worker lost while a checkpoint is open, until the controller resumes. A worker can also be lost after the commit checked readiness but before its records are collected, and it takes what it held with it. Every coordinated participant's export, the policy model's included, therefore goes through a sharedcollect()on the coordinator, which fails the commit withcheckpoint_export_failedif a worker was lost or fewer workers answered than expected. The failure is final for that checkpoint, because a later export could take a later cut.A failed restore's recovery finishes before another checkpoint begins. This closes a gap that is on
maintoo. A failed restore leaves admission closed until its recovery reopens it, either once an install that outlived the restore has finished, or on the controller's resume. Before, a prepare could begin a new checkpoint in that window. The recovery then never ran, and with several workers, a lost worker that held the failed restore's sessions made every later resume fail. Now a prepare is refused with the retryablecheckpoint_restore_recovery_pendingwhile the recovery is pending. The controller resumes after a failed prepare, and that resume finishes the recovery. A restore retried meanwhile first rolls back what the failed one left. Any reopen of a checkpoint also discards what a failed restore left before it reopens, and a restore that succeeds ends the pending recovery.Three smaller changes support this:
/runcalls its/v1/responsesfor its turn loop. That call carries no session ID, so it now names its worker in a newx-ng-session-ownerheader, and the router forwards a request without a session ID that names another live worker there, or answers 410 if that worker is gone. The turn loop's activation therefore runs on the worker that tracks the episode. The call also carries the activation index from feat(checkpoint): add the agent session participant and Simple Agent continuation #3888, and the worker that claims a restored agent/runepisode restores the turn loop's saved reply with it.ROUTED_LIFESPANS_STATEat that point, and the checkpoint link adds itself there. Before, a restore could place sessions on a worker whose socket was not yet listening, and forwarded requests got a 410. Simple Agent and the weather resources server gain the module-levelappthat multi-worker uvicorn imports.uvicorn_backlog, a new top-level setting that defaults to 16384 instead of uvicorn's 2048. The head server and each worker's private routing socket use it too, and the kernel caps it atnet.core.somaxconn. In the 12,000-rollout scale test, a burst of replacement/runrequests opened thousands of connections to the agent server at once and filled its queue of 2048. A session close that opened a new connection then waited on TCP retransmission backoff, past the environment server's 30-second cleanup timeout.How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class E,A,R thisOne checkpoint, a crash, and the restore, end to end:
This PR
One server with several uvicorn workers is still one participant. Its coordinator, in the main process, drives each worker's own participant; the router sends a session's requests to the worker that claims it.
Restoring sessions into new workers. A cookie or MCP session token names only the session ID; the restore rewrites that ID's claim to name the worker that holds the session now.
Whole-episode state, such as an environment episode or an agent's own
/run, waits in the coordinator. Exactly one worker claims it, whichever receives the replacement attempt's/run.Where this sits in the stack
This is one PR in a stack of draft PRs for partial-rollout checkpointing. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.
The stack builds on the token-capture cleanup PRs #3938 (capture ledger
retireanddelete) and #3939 (complete-recordretireanddelete), which have merged. #3882 to #3889, #3903 and #4690 have merged too. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.This PR builds on #4997, which has merged and gives each checkpoint concept one name before the feature is released. This PR uses those names, and its own new names follow them: its error codes carry the
checkpoint_prefix, a participant marks rather than fences, and a restored record a failed claim is still returning is refused withcheckpoint_restored_claim_settling.#4887, which has merged, fixes a Simple Agent prepare race in the turn loop's policy call. This PR changes Simple Agent's
/runcall to its own/v1/responses, and the two changes compose.ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks (merged)ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verification (merged)ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuation (merged)ananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordination (merged)ananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpoints (merged)ananthsub/fix-session-worker-routing): fix(server): route a session's requests to the worker that holds it, (merged)ananthsub/simple-agent-held-reply-boundary): fix(simple_agent): stay at the boundary until a held policy reply's body arrives (merged)ananthsub/checkpoint-naming): refactor(checkpoint): one name per checkpoint concept before release (merged)ananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workers (this PR)ananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operations#3893 (
ananthsub/partial-ckpt-rollout-collection, checkpoint evaluation runs from rollout collection) is independent of this chain and targetsmain. Its two spans,gym.checkpoint.collection.checkpointandgym.checkpoint.collection.restore, are added to it once #3903 merges, or in a small follow-up if #3893 merges first.Server ports that build on the stack: #3894 (Workplace Assistant), #3895 (Gymnasium), #3896 (Blackjack), #3897 (indirect prompt injection), #3898 (proof refinement). Session routing for several workers is #3905, which has merged. This PR's own additions to
nemo_gym/session_routing.pyare the claim rewrite for restores and its batched form, the lock that keeps a release from removing a claim a rewrite has just written, claims that last until a retired session's forget, thex-ng-session-ownerheader, and the hook that lets a worker join its coordinator once its private socket listens.Relationship to the old stack
The old stack (#2939 to #2946) required one worker for every checkpointing server except the policy model. The policy model already had multi-worker support in #3883, as a coordinator in the main process with worker gates. This PR generalizes that coordinator rather than adding a second mechanism. It replaces the three
num_workers=1guards the re-cut stack kept for resources, agent, and environment servers.Issue
No tracking issue exists. Multi-worker support for these servers was a stack requirement. Session routing, which it depends on, is #3905, which has merged.
Validation
Run on this branch, on the current
main, which includes #4997:RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider -n 16 tests/unit_tests/test_checkpoint_*.py tests/unit_tests/test_session_routing.py: passed.test_checkpoint_participant_workers.pyruns each scenario with a real coordinator socket and two or three worker links:checkpoint_worker_lost;/runepisode are each claimed by exactly one worker, and an unclaimed one is exported by the next checkpoint while the scope continues it, and retired, on every worker, when it does not;CancelledErrorby itself, or hangs past its attempt timeout, the worker forgets the session at once and the record goes back only after a later attempt succeeds; a later/runof the same key on that worker waits for the teardown;/runof its key on that worker getscheckpoint_restored_claim_settling; once a retire ends the lease, a teardown that keeps failing, or one that ignores its cancellation, stops within its cap and leaves nothing behind, and one that succeeds within the cap still settles its claim;/runepisodes sent with one cookie jar, on two workers, and an unclaimed restored one are all exported and named in the commit reply'srun_episode_ids;checkpoint_rollback_incomplete, until the rollback is repeated successfully or that worker is lost, and nothing is left afterwards;checkpoint_restore_recovery_pending, and once the recovery has run, a worker that held the failed restore's sessions and was lost leaves no worker closed; a resume of a checkpoint that began during the recovery anyway still rolls the failed restore back, and a holder lost after a retried restore succeeded still fails its resume;/statusadds up every worker's report and answers without a worker that does not;test_session_routing.pyadds: an old cookie follows its session to the worker a restore moved it to, a release after a restore moved the claim leaves the new claim, a rewrite that lands between a release's check and its removal survives, a batched rewrite whose caller is cancelled finishes before the worker counts its claims, and a request without a session goes to the worker itsx-ng-session-ownerheader names, or gets a 410 if that worker is gone.test_server_utils.pyandtest_session_routing.pyalso check the listen backlog: it defaults to 16384 with one worker and with several, followsuvicorn_backlog, refuses a value below 1 before uvicorn starts, reaches the head server, and reaches each worker's private routing socket. These checks fail without the setting.test_checkpoint_agent.py: the coordinator the multi-worker agent test starts is stopped, with its thread and socket gone.test_checkpoint_control.pyadds: a prepare while a failed restore's install still runs, or after its recovery could not reopen, is refused withcheckpoint_restore_recovery_pendinguntil admission reopens, and then succeeds.tests/unit_tests, sixteen processes, in a development environment with CI's extras): passed, except one benchmark-listing check that fails the same way onmainon this workstation, whose host name resolves to127.0.1.1.simple_agent,example_session_state_mgmtandsingle_agent_turntests, each run on its own: passed. Thesimple_agenttests include fix(simple_agent): stay at the boundary until a held policy reply's body arrives #4887's check that a held reply's body keeps the session at its boundary.NEMO_GYM_MULTIWORKER_E2E=1 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/session_routing: passed.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Signed-off-byline.Rollout evidence
The real-model and scale runs below ran on commit
16713dafec, #4197's tip on top of this PR'sffa3646fa6. The many-worker stress rows ran again on3fea619731, #4197's tip on top of this PR'se495d7b05b, which adds the changes to failed-restore recovery, rollback, forget, teardown key blocks and batched claim rewrites. They ran before this PR was rebuilt on #4997; the rebuild renames names and error codes and changes no behavior. The workstation was shared with other jobs during every run, with a load average of 26 to 55 on 24 cores, so the times below are slower and noisier than on an idle machine.Real model, on one workstation:
Qwen/Qwen3-4B-Instruct-2507served by vLLM 0.29.0 on one RTX 6000 Ada (--enable-auto-tool-choice --tool-call-parser hermes --max-model-len 32768), driving real Gym server processes that the opt-in tests in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 and this PR checkpoint mid-run, kill and restore. Each scenario ran with one and two policy workers and with one and two workers per server, and the whole set passed eight runs in a row.single_agent_turnrequest path, sent 20 ms apart until the first one finished. Each checkpoint caught 20 to 41 rollouts in flight, most inside their agent turn and the rest parked before closing the agent session, before verification, or before returning, while 2 to 13 had finished before it. After every Gym process group was killed and the checkpoint restored into fresh processes, every continued rollout finished with reward 1.0, as did every rollout that finished before the checkpoint.restart_onlyresources server. Prepare listed every restart still in flight and no saved rollout, and the commit left the restarts out. After the kill, the 10 to 21 restarts in each run were retired and started over from their input while the 10 to 16 saved rollouts continued, and every rollout finished with reward 1.0./runclaims its restored episode on whichever worker receives it. The runs with two workers per server go through that path.NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_CHECKPOINT_VLLM_URL=http://127.0.0.1:8129/v1 NEMO_GYM_CHECKPOINT_VLLM_MODEL=Qwen/Qwen3-4B-Instruct-2507 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint -k real_modelRunning the real-model tests with two workers per server found a gap in the e2e harness: uvicorn's workers import a server by the path its config names, so a test server that subclasses a repository server ran its parent's code with several workers, and the restart-only test server was not restart-only. Test servers now always run one worker, and the fake-backend restart scenarios also run with two workers per server; the two-worker restart scenario fails without the harness fix.
Scale, on one workstation with every Gym server and the fake inference backend. The run used weather rollouts on the typed
single_agent_turnrequest path, with token capture and generation cuts on. Each rollout's first model call returns and its second is held, so every rollout is in flight at the checkpoint. At most 500 rollouts open connections at once, the scale test's default. The test checkpoints, kills every Gym process group, restarts, restores, and runs the replacement attempts. Every server ran with two workers, and each row is the faster and slower of two runs:/runat once. Repeated at 12,000 rollouts, it failed in about 3 of 10 runs, with 1 to 509 replacement rollouts failing their session cleanup, on this PR and on its earlier version alike. Two causes add up. With uvicorn's listen backlog of 2048, the agent server's accept queue overflowed, and a close on a new connection waited on TCP retransmission backoff; this PR'suvicorn_backlograises that limit. Each worker also runs uvloop, which accepts about one new connection per event-loop iteration, so a busy worker can leave hundreds of connections unaccepted for tens of seconds. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 therefore bounds how many rollouts the scale test has opening connections at once.NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_CHECKPOINT_SCALE=12000 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint/test_checkpoint_e2e.py -k "test_checkpoint_at_scale and servers2-2"Many workers, on this version, with test(checkpoint): check checkpoint invariants over random sequences of operations #4197's opt-in stress test: the
/runrelay over Simple Agent and the counter resources server, 100 rollouts finished first and 1,000 held mid-episode at the checkpoint, every Gym process killed, and the checkpoint restored onto other worker counts:Every rollout that finished first scored 1.0 too, and each continued rollout made only the model calls it still needed.
Compatibility
num_workers > 1with checkpointing on. Single-worker servers behave as before, and their records do not change.ownerandsession_idfields the record models no longer accept. Checkpoints frommainload as before. The earlier version never shipped./runepisode whose replacement/runarrives while a checkpoint is open is refused withcheckpoint_admission_closed, as a new episode is, and the caller retries it. With one worker, it is admitted and parks at its first boundary.uvicorn_backlog, default 16384. Every Gym server, including the head server and each worker's private routing socket, now listens with this backlog instead of uvicorn's 2048. The kernel caps it atnet.core.somaxconn, so a host with a lower value gets that value.x-ng-session-owner: on a server with several workers, a request without a session ID that names another worker in it is forwarded to that worker, or gets a 410 if that worker is gone.checkpoint_worker_lost(409), for a worker lost during a coordinator's message, for a restore that lost a worker it installed sessions on, and for a resume after a worker holding restored sessions was lost.checkpoint_restored_claim_settling(503), retryable, for a replacement/runwhose restored record a failed claim is still returning, after the worker's own retries ran out.checkpoint_rollback_incomplete(503), retryable, for a resume while a worker could not yet delete what a failed restore installed on it. Admission stays closed until it can.checkpoint_restore_recovery_pending(503), retryable, for a prepare while a failed restore's recovery has not reopened admission yet, on every participant, with one worker or several. The controller resumes, which finishes the recovery, then prepares again.checkpoint_worker_lostorcheckpoint_deadline_exceeded, instead of the genericcheckpoint_error.