Repository navigation
feat(checkpoint): add the resources server participant with asynchronous session hooks - #3886
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
aa41669 to
defe5d3
Compare
defe5d3 to
f481624
Compare
f481624 to
4e9f81d
Compare
4e9f81d to
b72fe50
Compare
b72fe50 to
7f1cfb6
Compare
7f1cfb6 to
412e854
Compare
69300c6 to
98bdc0c
Compare
|
|
||
| def readiness(self) -> PrepareReport: | ||
| restarts = ( | ||
| sorted({episode_id.capture_key for _, episode_id in self._checkpointed_sessions()}) |
There was a problem hiding this comment.
restart_only: a seed in flight at prepare changes restarts mid-checkpoint, so the commit fails with restart_in_scope. restarts only lists registered sessions. A /seed_session admitted before close_admission isn't in it at prepare, so the controller keeps that episode in scope. The seed then finishes and registers, the commit's re-computed readiness() lists the episode as a restart, and the commit fails with RestartInScopeError on every retry of that checkpoint.
Repro on 98bdc0c (slow seed_session on a RestartOnlyServer, prepare while it's in flight, release it, commit with episode_ids=[{"rollout_id": "r"}]): prepare → prepared, restarts=[]; seed → 200 with x-ng-checkpoint-restart: 1; commit → 409 restart_in_scope: ['r'].
Suggest reporting in-flight seeds as restarts too, without waiting for them, so the server still never holds up prepare. Its seed reply always says restart, so the outcome is already known:
restarts = (
sorted(
{episode_id.capture_key for _, episode_id in self._checkpointed_sessions()}
| {episode_id.capture_key for episode_id in self._seeding}
)
if self.mode == "restart_only"
else []
)With this, the same repro gives prepare restarts=['r'] and the commit succeeds; the existing resources tests still pass. A restart_only variant of test_a_seed_in_flight_when_admission_closes_holds_up_prepare_and_is_exported would cover it.
There was a problem hiding this comment.
Fixed as you suggested. In restart_only mode, the restarts now include the episodes of seeds in flight, without waiting for them, so the server still never holds up prepare. Covered by test_a_restart_only_seed_in_flight_when_admission_closes_is_reported_as_a_restart, your reproduction: prepare reports the seed's episode as a restart, the seed reply says restart, and the commit of the remaining scope succeeds.
| await self.participant.wait_open() | ||
| if path == _SESSION_START: | ||
| # A restart_only server stays open during a checkpoint, but a new session would add a restart to it. | ||
| await self.participant.wait_open() |
There was a problem hiding this comment.
Nit, same invariant: this is a single wait_open(), not a loop. In restart_only, if a new checkpoint clears _open between open_admission() setting it and this task resuming, the seed is admitted during that checkpoint and adds a restart after prepare reported. Suggest looping until open, like the accepting loop above.
There was a problem hiding this comment.
Fixed. wait_open() now checks again each time it wakes, like the accepting loop, so a seed whose wait ended just before a new checkpoint closed admission keeps waiting. Covered by test_a_restart_only_seed_waits_again_when_a_new_checkpoint_closes_before_it_resumes, which reopens and closes admission before the waiting seed runs again.
| counted = path not in self.participant.replayable_paths | ||
| self.participant.request_started(session_id, counted=counted) | ||
| try: | ||
| await self.app(scope, receive, capture_status) |
There was a problem hiding this comment.
A disconnect can lose the session change. seeded() / session_ended_by() run only after await self.app(...) returns. Every resources server is wrapped in ClientDisconnectCancellationMiddleware, which cancels the request when the client disconnects (server_utils.py:1261). If that happens after the handler has changed state but before the response finishes sending, the CancelledError skips this block: a created seed is left untracked, or a session the server already closed/verified stays tracked. That's exactly what an environment retire does when it cancels its in-flight calls.
Suggest applying the change in capture_status on the first successful http.response.start, before awaiting the outer send, guarded to run once and re-reading the final session ID from scope["session"] there. A regression where the request is cancelled (or send raises) right after response start would cover it.
There was a problem hiding this comment.
Fixed. capture_status applies seeded() or session_ended_by() on the first successful http.response.start, before it awaits the outer send, once per request, and reads the final session ID from scope["session"] there. The session is still registered while the request counts as in flight. Covered by test_a_session_change_survives_a_disconnect_once_the_reply_has_started, where send is cancelled right after the response starts: the seed is tracked, and the closed or verified session is not.
| return | ||
| if (self.config.num_workers or 1) != 1: | ||
| raise ValueError("resources checkpointing requires num_workers=1: sessions live in one process") | ||
| if self.checkpoint_mode == "exported": |
There was a problem hiding this comment.
checkpoint_mode and checkpoint_verify are ClassVar[Literal[...]], so nothing checks them at runtime. A typo like "exportd" starts fine and acts as a fourth mode: nothing is exported and no restarts are reported, so session state is silently dropped. Suggest validating both against their allowed values here at startup.
There was a problem hiding this comment.
Fixed. Startup now checks checkpoint_mode and checkpoint_verify against their allowed values and raises a ValueError that names the attribute, the value, and the allowed values. The check runs whether or not checkpointing is enabled, so a typo fails before anyone turns it on. Covered by test_an_unknown_checkpoint_declaration_fails_at_startup.
| session_id_to_counter: Dict[str, int] = Field(default_factory=dict) | ||
|
|
||
| async def export_session_states(self, session_ids: list[str]) -> dict[str, JsonValue]: | ||
| return {session_id: {"count": self.session_id_to_counter.get(session_id, 0)} for session_id in session_ids} |
There was a problem hiding this comment.
.get(session_id, 0) makes up state for sessions the server no longer holds. The hook contract says to leave those out, so the participant can stop tracking them; with this, lost state is exported as count=0 and a restore installs a wrong counter. Since this is the example people will copy, suggest only returning IDs present in session_id_to_counter.
There was a problem hiding this comment.
Fixed. The example now returns only the sessions present in session_id_to_counter, and says why in a comment, since people copy it. Covered by test_export_leaves_out_a_session_the_server_no_longer_holds.
| # Inside an episode every resources call carries the rollout prefix, native seeds included. | ||
| episode_id = current_episode_id() | ||
| if episode_id is not None: | ||
| self.participant.seeded(session_id, episode_id) |
There was a problem hiding this comment.
Follow-up: a successful retire now handles an in-flight seed, but the path where the retire is cut short is still unsafe. retire() waits for _seeding (line 241) before it adds any session to _retired_sessions / _retired_session_rollouts. If the deadline cancels it during that wait, the seed finishes afterwards and seeded() binds the session as live. Then:
- MCP calls that carry only the session token (no rollout prefix) are admitted;
- the session is in
_checkpointed_sessions(), so a checkpoint taken before the retry exports the retired attempt's state; check_forget()letsforgetthrough because_retired_sessions[rollout]is empty, and the session stays live after the rollout mark is cleared.
Slow seeds (sandbox/container start) make this realistic. Suggest having seeded() check whether self.retired already covers the EpisodeId and, if so, add the new session to the retired maps immediately (or install an equivalent rollout cutoff before waiting). A gated-seed / short-deadline regression would cover it.
98bdc0c to
b2cf5f2
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-3886-71c32bf1633f.docs.buildwithfern.com/nemo/gym |
| """Refuse the attempts' sessions, wait for their requests in flight, then release them.""" | ||
| # A seed admitted before the retire marked its attempt registers its session when it finishes; wait for it, | ||
| # so that session is freed here and not left behind. | ||
| while any( |
There was a problem hiding this comment.
Blocker: a retire cut short leaves the attempt's existing sessions unfenced. retire() waits for _seeding before it adds the existing sessions to _retired_sessions / _retired_session_rollouts. If the deadline cancels that wait, sessions the attempt already had are never fenced. The rollout mark doesn't cover them for requests without a prefix, such as MCP calls.
Repro on fb5c857: one existing session for r/0, a second seed in flight, retire with a short deadline (409), then the late seed fails:
- an unprefixed request on the existing session → 200;
ready()→ True;forget(["r"])→ 200, and the session stays live with the mark gone.
(If the late seed succeeds, its own session gets fenced and happens to block prepare and forget; the existing session is still admitted.) The commit only exports the retired attempt when its scope is None; with a normal scope, export filters it out.
Suggest fencing the sessions already known before the first await, then collecting and fencing again once the seeds finish. As a safety net, check_forget could use the rollout index (any tracked session with attempt ≤ the retired mark) rather than only the fence maps, so a missed fence can't make a session live again.
There was a problem hiding this comment.
Agreed, a retire that never got past the seed wait left the attempt's existing sessions admitted.
The participant now refuses those sessions as the attempt is marked, before anything awaits.
So a retire that is cut short, or that arrives after its deadline and only marks the attempt, still leaves them refused, holding up readiness, and out of every export until a retried retire frees them.
The retire itself collects the rollout's sessions again once the in-flight seeds finish, so it frees the sessions those seeds created as well.
With several workers, each worker fences its own sessions when the coordinator marks the attempt.
As the safety net you suggest, check_forget now looks up the rollout's tracked sessions through the rollout index and refuses while any of them belongs to an attempt at or below the retired mark, whether or not a fence reached it.
Covered by test_a_retire_cut_short_while_a_seed_is_in_flight_still_fences_the_attempt_s_existing_session, your reproduction: an existing session, a second seed in flight, a retire cut short, and the late seed failing.
The unprefixed request is now refused with stale_attempt, ready() is false, and forget is refused with retire_incomplete until a retried retire frees the session.
On the previous code the unprefixed request returned 200.
test_a_retire_that_arrives_after_its_deadline_still_fences_the_attempt_s_sessions covers a retire that only marks; on the previous code its late request also returned 200.
|
|
||
| def admit(self, session_id: Optional[str], path: str) -> None: | ||
| # A close is never refused: it is how the server frees a session's state. | ||
| if path != _SESSION_CLOSE: |
There was a problem hiding this comment.
Blocker: a stale /close_session can delete a restored replacement. Closes skip every stale-attempt check. After a restore, the session is bound to attempt N+1, but a close from attempt N (with N marked retired by the restore) is still admitted. restore() marks the replaced attempts precisely because "a harness of a crashed attempt that outlived the crash may still call this server" (control.py:786).
Repro on fb5c857 (commit r/0, restore with episode_ids=[r], resume):
- stale
POST /ng-rollout/r/increment→ 409 (correctly refused); - stale
POST /ng-rollout/r/close_session→ 200; - tracked sessions → 0, and the next commit writes 0 records.
Suggest checking ownership instead of exempting closes: in admit(), refuse any non-seed request whose EpisodeId (from the prefix) differs from the session's bound episode. That covers /verify too, and the close body already carries episode_id. A close from the owning attempt, including a retired one freeing its own session, still goes through. Requiring the rollout prefix on session-ending routes, as seeds already do, would make this complete.
There was a problem hiding this comment.
Agreed, closes skipped every check, so a replaced attempt could end its replacement's session.
admit() now checks ownership: a request whose rollout prefix names an episode other than the one the session is bound to is refused with stale_attempt.
Seeds are the exception, since a seed binds the session.
This covers /close_session and /verify alike, and a request from another rollout on the same cookie.
A close from the owning attempt still goes through, including a retired attempt freeing its own session.
With checkpointing on, /close_session and /verify now need the rollout prefix, as session starts do, and are refused with rollout_id_required without it.
Every caller already sends it: the environment server runs each episode inside its rollout context, Simple Agent's /run does the same, and ServerClient adds the prefix to every resources call made in that context; rollout collection and the end-to-end tests reach resources servers only through those.
Covered by test_after_a_restore_a_close_from_the_replaced_attempt_is_refused_and_keeps_the_session, your reproduction: commit, restore, resume, a stale close, then the next commit.
The stale close is refused, the next commit writes the session's record, and the replacement's own close then ends it.
On the previous code the stale close returned 200.
test_a_request_on_a_session_from_another_rollout_is_refused and test_a_session_end_without_a_rollout_prefix_is_refused_before_its_handler_runs cover the other two rules.
| session_id = (scope.get("session") or {}).get(SESSION_ID_KEY) | ||
| is_mcp = self.mcp_session_id is not None and (path == self.mcp_path or path.startswith(self.mcp_path + "/")) | ||
| if is_mcp: | ||
| session_id = self.mcp_session_id(scope) or session_id |
There was a problem hiding this comment.
Blocker / design decision: old MCP tokens stay valid after a restore. The token holds only sid and tools. A restored session keeps its sid and is bound to attempt N+1, and an MCP call carries no rollout prefix and isn't in the fence maps. So a harness of attempt N that outlived the crash (the case restore() guards against at control.py:786) can still change the restored session: the original token moves it from count 1 to 2.
test_the_harness_keeps_its_token_across_a_restore requires exactly this: the replacement harness keeps the same bearer credential. Since the old and new harness hold the same token, the server can't tell them apart, so "retired attempts stay fenced" and "the replacement keeps its token" can't both hold.
This needs an explicit decision. Options: put the attempt (or a session generation that restore bumps) in the token and issue the replacement a new one during recovery; or, if that's out of scope here, document it as a known limitation with a follow-up issue and narrow the test's claim.
There was a problem hiding this comment.
Decided: the token now names the attempt whose seed minted it, and the resources server refuses a token whose attempt a retire or a restore has fenced.
With checkpointing on, _mint_session_metadata adds an episode claim holding the seed's capture key, such as r or r-a1.
With checkpointing off, the token is byte for byte what it was.
The checkpoint middleware reads the claim before the call reaches /mcp, and ResourcesParticipant.admit checks it the way it checks a rollout prefix: the session must belong to that attempt, and the attempt must not be retired.
A verified token without an episode claim is refused too, because a server with checkpointing on only mints tokens that have one, so such a token comes from a process without checkpointing, such as the one before a restart.
A refused call gets 409 stale_attempt before its tool runs, and it does not claim a restored session, so a commit that no longer continues the episode still deletes that session.
The replacement attempt never inherits the old token.
It gets its own by seeding under its own attempt, and a seed that carries the session's cookie claims the restored session, so the new token continues it.
No agent that holds an MCP token continues across a restore today.
Simple Agent, the only agent that continues from a checkpoint, refuses a required MCP grant and ignores an optional one.
Hermes, Codex, OpenCode, Pi and Haystack are restart-only, so their episodes start over from their input after a crash and seed a new session, which mints a new token.
To make that a rule instead of a coincidence, the environment server's _seed_agent now marks an episode as a restart when it grants MCP access.
So no checkpoint records such an episode past its seed while its agent holds a token for an earlier attempt.
The same fence closes a related gap: a harness that outlived a crash could call the fresh process with a token for a session that process never saw, and create state there that nothing would free.
Covered by:
test_a_restore_refuses_the_replaced_attempt_s_token_and_a_seed_of_the_replacement_gets_one_that_works, which replacestest_the_harness_keeps_its_token_across_a_restore;test_a_stale_harness_cannot_create_state_for_a_session_the_fresh_process_never_saw;test_a_token_without_an_attempt_is_refused_when_checkpointing_is_on;test_a_second_restore_refuses_the_first_replacement_s_token;test_a_refused_stale_call_does_not_claim_a_restored_session;test_a_token_minted_without_checkpointing_names_only_its_session_and_toolsandtest_a_token_minted_with_checkpointing_names_the_attempt_that_seeded_it;test_after_a_restore_onto_several_workers_every_worker_refuses_the_replaced_attempt_s_mcp_token, including a worker that joins after the restore;test_an_episode_that_grants_mcp_access_is_a_restartin thesingle_agent_turntests;test_mcp_tool_calls_wait_out_a_checkpoint_and_a_restore_hands_their_sessions_to_new_tokensin the end-to-end suite, across a restart onto three workers.
Ending a token's lifetime when its session closes is separate revocation work.
It will use the newResourcesParticipant.is_openwhile checkpointing is on.
| async def delete_restored(self, episode_id: EpisodeId) -> None: | ||
| """Free the restored sessions of ``episode_id`` that no request has used, without refusing anything later: | ||
| the controller may still start the rollout over as that attempt.""" | ||
| for session_id in [key for key in self._restored_pending if self._sessions.get(key) == episode_id]: |
There was a problem hiding this comment.
Perf: post-restore commit cleanup is quadratic. _delete_unscoped() calls delete_restored() once per unscoped restored episode, and each call scans all of _restored_pending, comparing pydantic EpisodeIds. Timing on fb5c857 (restore N sessions, then a commit whose scope drops them all):
| restored sessions dropped | time |
|---|---|
| 1,000 | 0.37 s |
| 5,000 | 9.7 s |
| 10,000 | 41 s |
This runs inside the commit's deadline. Each scan is synchronous, so it also stalls the event loop in ~4 ms chunks per episode. That can happen whenever the controller drops many restored rollouts after a restore.
Suggest using the rollout index, as retire() does:
for session_id in [
key
for key in self._rollout_sessions.get(episode_id.rollout_id, ())
if key in self._restored_pending and self._sessions[key] == episode_id
]:With this, the same probe takes 20 ms at 10,000 and 118 ms at 50,000, and the resources tests still pass.
There was a problem hiding this comment.
Applied as you suggest: delete_restored now walks the rollout's sessions through the rollout index, so each call costs as much as its own rollout.
Measured with your probe, restoring N sessions and then deleting them all as a commit whose scope drops them does:
- 1,000 sessions: 0.29 seconds before, 0.005 seconds after;
- 10,000 sessions: 31 seconds before, 0.05 seconds after;
- 50,000 sessions: 945 seconds before, 0.44 seconds after.
Covered bytest_deleting_one_rollout_s_restored_sessions_does_not_scan_every_restored_session, which makes iterating the restored sessions fail; on the previous code it did.
| episode_ids.append(mcp_token.episode_id) | ||
| bound = self._sessions.get(session_id) if session_id is not None else None | ||
| for episode_id in episode_ids: | ||
| if not starts_session and bound is not None and bound != episode_id: |
There was a problem hiding this comment.
Require a tracked, open session for every non-start session-scoped call.
This ownership check runs only when bound is not None. A valid MCP token is therefore still admitted after /close_session, or against a fresh process before restore/retire installs its session. A prefixed direct-HTTP request can likewise target an untracked session ID. The tool can then create or mutate state while the participant reports no session, so later checkpoints neither save nor retire it.
Please fail closed for every MCP or direct-HTTP non-start session call unless the session is currently tracked and open. /seed_session must remain able to create the binding; an unknown /close_session can be idempotent, but it should not invoke a destructive hook when ownership cannot be established. Genuinely sessionless routes such as /aggregate_metrics should be explicit exceptions.
Please cover MCP and prefixed direct-HTTP calls both after close and before restore/retire has installed the session.
There was a problem hiding this comment.
A tool call that creates or changes state on a closed or unknown session is a session-lifecycle bug, and it exists with checkpointing off as well.
It will be fixed on main for MCP and direct HTTP alike, together with revoking a session's MCP token when the session closes and binding the default seed's cookie session to resources_session_id, so the rule holds whether checkpointing is on or off.
That is why this PR does not add it.
What #3886 covers today is the attempt fence:
- an MCP token names the attempt whose seed minted it, and a token of a retired or replaced attempt, or one that names no attempt, is refused before its tool runs;
- a request whose rollout prefix names an attempt other than the one a session is bound to is refused, and so is a request on a session a restore installed that names no attempt (see the reply on the unprefixed direct HTTP call);
ResourcesParticipant.is_openreports whether a session is tracked and not retired, for the lifecycle work to use with checkpointing on.
| # Refusing it would fail that episode; its episode re-runs it after a crash anyway. | ||
| if not self.accepting and path not in self.replayable_paths: | ||
| raise AdmissionClosedError("resources admission is closed for a checkpoint") | ||
| self._restored_pending.discard(session_id) |
There was a problem hiding this comment.
An unprefixed direct-HTTP tool call can mutate and claim a restored session.
_seed_agent() gives direct-HTTP clients the bare Resources URL plus the session cookie. Such a request supplies session_id but no attempt in episode_ids, so it skips ownership validation and still removes the session from _restored_pending.
Reproduced after restoring r as r-a1: /ng-rollout/r/increment is correctly rejected, but /increment with the same cookie returns 200, mutates the restored state, and claims the session. A later commit that drops r-a1 writes no record but also fails to delete that external state.
Please propagate the current activation's capture key into direct-HTTP tool URLs and reject session-bearing requests with no attempt identity. These need to land together: requiring identity alone breaks current direct HTTP, while adding prefixes alone leaves cookie-only callers able to bypass ownership checks.
Please test that stale r is rejected, replacement r-a1 succeeds, and a refused stale call does not remove the session from _restored_pending.
There was a problem hiding this comment.
Fixed, with both halves together.
With checkpointing on, the environment server's direct HTTP grant now carries the seeding attempt's /ng-rollout/<capture key> prefix in its base URL.
Simple Agent sends each direct HTTP tool call under the rollout prefix of the activation it is serving, in place of the prefix in the grant, so after a restore the calls of the continued session name the replacement attempt.
Its legacy /v1/responses tool calls carry the same prefix.
No other agent with checkpoint support uses direct HTTP grants: every other agent rejects a required one, and restart-only agents seed again after a crash and get a fresh grant.
On the resources server, a request on a session a restore installed that names no attempt (no rollout prefix and no MCP token) is refused with 400 rollout_id_required, since the cookie alone cannot tell the replacement from the replaced attempt.
The refusal comes before the participant takes the session out of _restored_pending, so a refused call never claims a restored session.
Sessions seeded in the running process are not affected: a stale attempt can reach one only through a cookie, and a retire refuses the sessions of the attempts it retires.
Keeping it to restored sessions also leaves restart-only servers, which never restore a session, and agents whose tool calls carry only the cookie unaffected.
Covered by:
test_after_a_restore_only_the_replacement_s_prefixed_calls_reach_the_session: after restoringrasr-a1,/ng-rollout/r/incrementis refused withstale_attempt,/incrementwith the cookie alone is refused withrollout_id_required, neither changes the restored count or claims the session, and/ng-rollout/r-a1/incrementcontinues it;test_stale_and_cookie_only_calls_stay_refused_across_a_second_restore;test_with_checkpointing_on_a_direct_http_grant_names_the_seeding_attemptin thesingle_agent_turntests;test_a_restored_session_s_direct_http_tool_calls_reach_the_restored_resources_sessionin the Simple Agent tests, a continuation through Simple Agent and a real resources server, whose tool call reaches the restored session under the replacement's prefix;test_resources_state_is_restored_with_the_episodein the end-to-end suite, where the replacement's legacy tool call increments the restored counter.
|
|
||
| def session_ended_by(self, session_id: str, path: str) -> None: | ||
| """A request on ``path`` ended ``session_id``. A replayable /verify during a checkpoint ends it at resume.""" | ||
| if path in self.replayable_paths and not self.accepting: |
There was a problem hiding this comment.
Blocker: replayable /verify can end its session between environment and resources prepare.
session_ended_by() defers the end only after this resources participant has closed admission. Coordination prepares environment before resources. If the environment is inside a replayable /verify, environment prepare freezes a pre-verify boundary; if the request finishes before resources prepare reaches this server, accepting is still true and this branch calls ended(). Resources then commits no session record, but recovery replays /verify from the environment's pre-verify boundary with no restored state.
I reproduced this ordering: /verify returned 200, the resources commit wrote zero records, and the external counter remained present but untracked. The existing replayable-verify test starts /verify only after resources is already prepared, so it misses this cross-participant window.
For checkpoint_verify="replay", please do not make /verify the final resources-lifecycle end; keep the session tracked until an explicit /close_session or retire. Alternatively, introduce a coordinator-wide checkpoint epoch/fence visible to resources before environment prepare. Checking only not self.accepting cannot cover this window.
Please add a coordination-order regression: start replayable /verify → prepare environment while it is in flight → finish /verify → prepare and commit resources; assert that the session is exported and restore can replay verification.
There was a problem hiding this comment.
Fixed: a replayable /verify no longer ends the session when its caller closes the session afterwards.
The caller says so with a new x-ng-checkpoint-close-follows: 1 header on /verify.
With checkpoint_verify = "replay", the participant then keeps the session tracked, open and exported until the /close_session that follows, a declared end, or a retire.
So the order you reproduced no longer loses the session: whether the verification finishes before or after the environment and resources prepares, the session is still tracked when the resources server exports, and a restore runs the verification again against the restored state.
single_agent_turn (and so the legacy flat-row wrapper, which runs the same episode), the UserSim environment server and the sandboxed Hermes agent send the header, because each of them closes the session in its final cleanup.
Simple Agent's legacy /run now sends it after a replayable verification and then closes the resources session, including when a replacement is restored at its return step.
A /verify without the header still ends the session at once, as does a waited-for /verify with or without it.
A caller that never closes, such as a restart-only agent's own /run, would otherwise leave the session tracked forever, and no checkpoint continues its episode.
A checkpoint waits for a waited-for verification, so no boundary from before it can outlive it.
The deferred-end path (_deferred_ends) is gone.
Covered by:
test_a_replayable_verify_that_ends_between_environment_and_resources_prepare_is_exported_and_replays, your coordination order: the verification finishes before resources prepare, the commit exports the session, and after a restore the replacement verifies again and closes it;test_a_replayable_verify_whose_caller_closes_the_session_afterwards_leaves_it_open_until_the_close;test_a_replayable_verify_whose_caller_never_closes_ends_the_session_even_during_a_checkpointandtest_a_waited_for_verify_ends_the_session_whether_or_not_its_caller_closes_afterwards;test_verify_says_that_the_resources_session_is_closed_afterwardsin thesingle_agent_turntests;test_a_legacy_run_closes_its_resources_session_after_a_verification_a_checkpoint_does_not_wait_forin the Simple Agent tests.
…ous session hooks A resources server declares a checkpoint mode: stateless, exported, or restart-only. In exported mode its session state is exported and restored through asynchronous hooks. A checkpoint waits for requests in flight; a request that arrives during one waits for resume instead of being refused; tools called over MCP are covered; and retire frees a session once its requests have stopped. Nothing of a restart-only server is in a checkpoint. Its admission never closes, its requests never hold up prepare, and its live sessions are reported as restarts. Its seed reply says so, so the episode that seeded the session starts over from its input after a crash, and keeps running through the checkpoint if nothing crashes. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38766077417 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38768162309 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38770334609 |
What changed and why
Every resources server takes part in checkpoints once checkpointing is on, in one of three modes, declared with the
checkpoint_modeclass attribute:restart_only(the default): nothing of a session is saved, so its rollout is a restart. It never blocks prepare, keeps running through the checkpoint, and is reported in prepare'srestarts; the seed reply carriesx-ng-checkpoint-restartso the environment server knows. After a crash the controller retires the rollout and starts it over from its input. Only a new session start waits while a checkpoint is open, so the restarts a prepare reported cannot change before its commit. A seed already in flight when admission closes is reported as a restart too, without waiting for it, because its reply will say restart; otherwise it would register during the checkpoint and the commit would fail withrestart_in_scope. A waiting seed checks again each time it wakes, so a new checkpoint that closes admission before it runs keeps it waiting.stateless: tool results depend only on the request, so nothing is exported. The weather example server uses it.exported: the server implements the session hooks. The counter example server (example_session_state_mgmt) uses it. It exports only the sessions it still holds, so a dropped session is left out and the participant stops tracking it, instead of being exported as a new counter at zero.The session hooks are coroutines, and export takes every session at once:
export_session_states(session_ids),restore_session_states(states), andretire_session_state(session_id). A server whose session state lives outside the process, such as a sandbox per session, can snapshot all of it concurrently within the commit. A session missing from the export result is treated as already dropped (for example, after a failed verification) and is no longer tracked, instead of failing the commit. Withcheckpoint_mode = "exported", a server that leaves any of these hooks as the base stub fails at startup; hooks from an intermediate base class or mixin count.retire_session_stateis called in every mode, and its base version does nothing, so arestart_onlyorstatelessserver that keeps per-session state can free it on retire by overriding it.checkpoint_modeandcheckpoint_verifyare checked against their allowed values at startup, whether or not checkpointing is on. A typo such as"exportd"raises aValueErrornaming the attribute, the value, and the allowed values, instead of acting as a fourth mode that exports nothing and reports no restarts.The session lifecycle comes from the server's session-start routes,
/close_session, and/verify, keyed by the rollout attempt, so no request body is inspected.checkpoint_session_starts, which defaults to("/seed_session",); a Gymnasium-style server declares/reset. The checkpoint middleware treats every declared route as it treats/seed_session, before the handler runs: it waits out an open checkpoint in every mode,restart_onlyincluded, counts as a seed from admission, carries the seed reply headers, and binds the session when it succeeds. There is no other way to start a tracked session, so no start can slip in after a commit has fixed its scope. Sessions end through/close_session, orcheckpoint_session_endedfor a server that ends them elsewhere./ng-rollout/<id>prefix is refused with 400rollout_id_requiredbefore its handler runs, so no session can exist that the participant does not track.SimpleServer.setup_checkpoint_participant(app)runs for every server aftersetup_webserver. With checkpointing on, a resources server whose app did not install the checkpoint participant refuses to start, naming its class, instead of being silently left out of every checkpoint. Gymnasium servers onmainbuild their own app, so they refuse to start with checkpointing on until fix(gymnasium): build Gymnasium servers on the shared resources server setup #3895 moves them onto the shared setup; with checkpointing off nothing changes.stale_attempt, seeds excepted, so after a restore a replaced attempt's/close_sessionor/verifycannot end the replacement's session. A close from the owning attempt, including a retired attempt freeing its own session, still goes through. With checkpointing on,/close_sessionand/verifyneed the/ng-rollout/<id>prefix, as session starts do.rollout_id_required. The cookie alone cannot tell the replacement from the attempt the restore replaced. The refusal comes before the participant takes the session out of the restored sessions no request has used yet, so a refused call never claims one. Sessions seeded in the running process are not affected, and neither are restart-only servers, which never restore a session.forgetis refused while any tracked session of the rollout is at or below the retired mark.episodeclaim with the seeding attempt's capture key; with checkpointing off the token is unchanged. A call whose token names a fenced attempt, an attempt other than the session's, or no attempt gets 409stale_attemptbefore its tool runs, and never claims a restored session. A restore therefore no longer lets the replaced attempt's harness keep using its token: the replacement gets a token by seeding under its own attempt with the session's cookie.ResourcesParticipant.is_open(session_id)reports whether a session is tracked and not fenced, for revocation at session close.mainfor MCP and direct HTTP alike.restartsfield of the prepare request. The commit leaves those episodes out, so a participant keeps them incarried_restartsuntil the checkpoint resumes, aborts or expires, and leaves them out of its blockers and its export. The resources participant no longer waits for requests in flight on their sessions, so an episode the environment server reported as a restart, for example from an MCP grant, keeps running through the checkpoint. Coordination sends the field only when an earlier stage reported a restart, and a prepare without it behaves as before.The
checkpoint_verifyclass attribute declares whether/verifymay run again after a crash (replay) or must finish before a checkpoint (wait, the default). The server reports it in thex-ng-checkpoint-verifyheader on every/seed_sessionreply and in its checkpoint status. A replayable/verifyis admitted even after admission closes, because the checkpoint does not wait for the episode step that sends it. A/seed_sessionthat arrives while a checkpoint is open waits for resume instead, and a seed already in flight holds up prepare and is exported. If the seed handler replaces the session ID, the session is tracked under the new one. The participant applies a seed, close, or verify to its session tracking as soon as the response starts, before the rest of the response is sent. A client that disconnects after the handler ran cancels the request, as an environment server's retire does to its calls in flight, and that cancellation can no longer leave a created session untracked or an ended one tracked. The environment server seeds inside a replay step that can finish while the checkpoint is open, so the session must match whichever boundary the environment server records.Tools called over MCP (
expose_tools_over_mcp: true). A CLI agent harness calls tools over MCP with a signed session token and no session cookie. Before this change, such a server failed at startup with checkpointing on: MCP auto-exposure refuses middleware that direct MCP dispatch would skip, and the checkpoint middleware was one of them.episodeclaim decides whether its attempt may still use it after a restore./verifyis still admitted at once.Ending sessions.
/close_sessionis never refused. Closing is how a server frees a session's state. A close that arrives while a checkpoint is open, from an episode that ended during it, waits for resume as an MCP call does; a refusal would leave the session unfreed, since the environment server closes once.retire_session_state, in every mode. A session stays tracked until its state is freed: a retire cut short by its deadline leaves it for a retry, and meanwhile it blocks the next checkpoint and is not exported.retire_session_statecan therefore run again for the same session, and must tolerate state it already freed. The refusal lasts until the controller forgets the rollout:/seed_sessioncannot create state nothing frees;forget, and counted in status asretired_sessions.statelessandrestart_onlyincluded, records which attempt each session belongs to, so a retire drains and refuses their sessions too. Export and readiness still depend on the mode./seed_sessionof the same rollout, up to the retired attempt, that is already admitted, then frees the session that seed created. A seed is counted at its admission, with no await in between, so a seed still waiting for resume cannot hold up the retire, and the retire's mark refuses it once it is admitted. Retire visits only the rollout's own sessions, through an index from rollout to sessions.forgetis refused withretire_incomplete, changing nothing, while a retired session it targets is still tracked, or while a seed of a retired attempt of that rollout is still in flight; the controller retires again first. Forgetting during such a seed would clear the mark before the seed finished, and its session would then be bound as live./verifywhose caller closes the session afterwards does not end it. The caller says so with thex-ng-checkpoint-close-follows: 1header, and the session stays tracked and exported until the/close_sessionthat follows, a declared end, or a retire. A checkpoint can keep the environment server's boundary from before the verification even when the verification finishes before the resources server prepares, so the session must still be exported, and a restore runs the verification again against the restored state.single_agent_turn, the UserSim environment server, and the sandboxed Hermes agent send the header, since each closes the session in its final cleanup; Simple Agent's legacy/runsends it in feat(checkpoint): add the agent session participant and Simple Agent continuation #3888. A/verifywithout the header, and a wait-mode/verify, end the session at once, so a caller that never closes cannot leave it tracked.retire_session_statewithout refusing anything later, since the controller may still start the rollout over as that attempt. A restored session that the server drops before any request uses it does not break the commit.ready()without building its blocker list, and admission refusals are the sharedAdmissionClosedError. The MCP session token helpers live inbase_resources_server.pyand are shared with MCP auto-exposure.How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class R thisOne checkpoint, a crash, and the restore, end to end:
This PR
How a resources server takes part, chosen by a class attribute.
flowchart TD M{"checkpoint_mode"} -->|stateless| S["Nothing to export.<br/>Never blocks a checkpoint."] M -->|exported| X["export_session_states at commit<br/>restore_session_states after a crash<br/>retire_session_state on retire"] M -->|"restart_only, the default"| O["Its rollout is a restart: reported in prepare,<br/>never blocks, starts over after a crash"]A session's lifecycle, as the participant learns it from the routes that already define it.
Retire drains a session before freeing it.
Where this sits in the stack
This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.
The stack builds on the token-capture cleanup PRs #3938 (capture ledger
retireanddelete) and #3939 (complete-recordretireanddelete), which have merged. #3882 to #3885 have merged too, so #3886's base ismain. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks (this PR)ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verificationananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuationananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordinationananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collectionananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpointsananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workersananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operationsPartial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the
checkpoint:block unset, no server installs any checkpoint routes or behavior.Relationship to the old stack (#2939 to #2946)
Issue
No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.
Validation
Run on this branch, on the current
main:test_each_prepare_stage_carries_the_restarts_every_earlier_stage_reportedandtest_a_request_in_flight_for_a_restart_an_earlier_stage_reported_does_not_hold_up_prepare, each failing without its change: later stages get the earlier stages' restarts, the restarts are cleared at resume, and a resources request in flight for such an episode no longer holds up prepare.test_a_retire_cut_short_while_a_seed_is_in_flight_still_fences_the_attempt_s_existing_session,test_a_retire_that_arrives_after_its_deadline_still_fences_the_attempt_s_sessions,test_after_a_restore_a_close_from_the_replaced_attempt_is_refused_and_keeps_the_session,test_a_request_on_a_session_from_another_rollout_is_refused,test_a_session_end_without_a_rollout_prefix_is_refused_before_its_handler_runs,test_deleting_one_rollout_s_restored_sessions_does_not_scan_every_restored_session,test_a_restore_refuses_the_replaced_attempt_s_token_and_a_seed_of_the_replacement_gets_one_that_works,test_a_stale_harness_cannot_create_state_for_a_session_the_fresh_process_never_saw,test_a_token_without_an_attempt_is_refused_when_checkpointing_is_on,test_a_second_restore_refuses_the_first_replacement_s_token,test_a_refused_stale_call_does_not_claim_a_restored_session,test_a_token_minted_with_checkpointing_names_the_attempt_that_seeded_it, and theis_opentests;test_a_token_minted_without_checkpointing_names_only_its_session_and_toolsguards the unchanged token with checkpointing off.test_a_declared_session_start_during_a_commit_s_export_waits_for_resume, in all three modes: a declared/resetarriving while the commit is in export waits for resume, and the commit has no record for it (without the fix, therestart_onlyreset finished during the checkpoint);test_a_restart_only_declared_session_start_in_flight_is_reported_as_a_restart(without the fix, prepare reported no restart).test_a_session_start_without_a_rollout_prefix_is_refused_before_its_handler_runs, for/seed_sessionand/reset(without the fix, a 200 that created untracked state).test_a_resources_server_whose_app_left_out_the_checkpoint_participant_refuses_to_startandtest_a_resources_server_whose_app_installs_the_participant_passes_the_startup_check, including a server with its own app starting with checkpointing off.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed, including, each failing without its change:/close_sessionduring a checkpoint waits for resume, then ends the session;forgetis refused while a retire has not freed its sessions;/verifywhose caller closes afterwards keeps the session tracked and exported until the close, including one that finishes between environment and resources prepare, after which a restore verifies again and closes it;/verifywithout the close-follows header, and a wait-mode/verifywith or without it, end the session at once;stale_attemptand a cookie-only call withrollout_id_required, across two restores, and neither changes the restored state or claims the session;ready()does not list blockers;restart_onlymode, a seed in flight when admission closes is reported as a restart by prepare, its reply says restart, and the commit of the remaining scope succeeds;restart_onlymode, a seed waiting for resume waits again when a new checkpoint closes admission before it runs;checkpoint_modeorcheckpoint_verifyfails at startup;restart_onlyandstatelessservers throughretire_session_state;stale_attempt, a checkpoint does not export it,forgetis refused withretire_incomplete, and a retried retire frees it, after whichforgetsucceeds;forgetis refused withretire_incompletewhile a seed of a retired attempt is in flight, and the session that seed creates is refused afterwards.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_mcp_auto_exposure.py tests/unit_tests/test_checkpoint_resources_mcp.py: passed. The MCP checkpoint tests go through the real control routes and MCP JSON-RPC:RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/example_session_state_mgmt/tests: passed, including a new test that the counter example leaves a session it no longer holds out of its export.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider resources_servers/example_single_tool_call/tests: passed.environment_servers/single_agent_turn/tests,environment_servers/nemo_user_sim/tests(in its own environment, with the UserSim package) andresponses_api_agents/hermes_sandboxed_agent/tests: passed, including that/verifysays the session is closed afterwards.main.tests/unit_tests, eight processes, withouttest_opensandbox_compose.py, which needs the optionalopensandboxpackage): passed except a sandbox retry test, which fails the same way onmainin this development environment.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Signed-off-byline.Rollout evidence
Qwen/Qwen3-4B-Instruct-2507on vLLM: saved rollouts continued after a kill and restore, restarts started over, and every rollout finished with reward 1.0. Its restart test uses arestart_onlyresources server.Compatibility
checkpoint_mode,checkpoint_verify,export_session_states,restore_session_states,retire_session_state,checkpoint_session_starts,checkpoint_session_ended, and thesetup_checkpoint_participantstartup check. All are new; existing servers need no change and default torestart_onlywithwaitverification.checkpoint_modeorcheckpoint_verifyvalue raises aValueErrorat startup, even with checkpointing off.retire_session_statein every mode, not onlyexported, and the base hook does nothing. Arestart_onlyorstatelessserver that already overrides it must expect the call.restartsfield. A participant from an older Gym build would refuse a prepare that carries it, which coordination sends only when an earlier stage reported a restart; within one deployment every server runs the same build.episode_idsscope. A commit request withoutepisode_idsstill exports every live session./seed_sessionreplies carry a newx-ng-checkpoint-verifyheader when checkpointing is on./verifyand then closes the session should sendx-ng-checkpoint-close-follows: 1on/verify; without it the session ends at the verification, as before.rollout_id_required.checkpoint:block setsenabled: true.