Skip to content

feat(checkpoint): resources, agent, and environment servers with several workers - #3909

Merged
ananthsub merged 1 commit into
mainfrom
ananthsub/partial-ckpt-multi-worker
Oct 26, 2026
Merged

ananthsub merged 1 commit into
mainfrom
ananthsub/partial-ckpt-multi-worker

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

This PR builds on #3905's session routing, which has merged.

Until now, resources, agent, and environment servers refused num_workers > 1 when checkpointing was on. uvicorn's workers share one port, so a control call reaches one arbitrary worker. If each worker ran its own participant, a checkpoint would close, export, and restore only that worker's share of the episodes and sessions. Production servers run several workers (swe_rebench runs 8, several sandboxed agents run 4), so this guard kept them out of partial-rollout checkpoints.

This PR makes each of these servers one participant across its workers:

  • A generic worker coordinator. The policy model server already runs as one participant across workers. The parts of it that do not depend on the policy model move to nemo_gym/_checkpoint/workers.py:

    • the framed Unix-socket channel, which encodes with the checkpoint writer's json encoder so NaN, infinities, and integers beyond 64 bits arrive unchanged, and answers a reply it cannot frame with a typed error. A message's send and reply share one timeout. A timeout or transport error on a worker's side is CoordinatorUnavailableError; on the coordinator's side it means the worker was lost, and is WorkerLostError (checkpoint_worker_lost), while an error a worker reports keeps its own status and code. Retire, delete_restored and forget are bounded by the request's deadline instead of that fixed timeout. A caller that stops waiting, on a timeout or a cancellation, sends the worker a cancel and waits until the worker's handler has stopped, so a retry never overlaps the operation it replaces. Exports travel in chunks of about 2 MiB, each encoded off the event loop while the one before it is sent and checked, and large frames are encoded and decoded off the event loop;
    • worker registration, and the join, clean-stop and lost-worker log lines feat(token-capture): add worker staging helpers for generation cuts #3884 added for the policy model, so a resources, agent or environment worker that is lost is logged the same way. A worker that stops cleanly first gives a claim's teardown that is still running up to 30 seconds to finish and settle its claim, then cancels and awaits its report, readiness poll and claim-settlement tasks, so none of them retries against a closed connection;
    • marking an attempt retired fences each worker's sessions of that attempt at once, including on a worker that joins later, and a commit that drops a restored episode deletes it on each worker through that worker's own delete_restored instead of a retire, so a replacement's sessions on other workers are not freed and no fence is left behind;
    • MCP tokens from a replaced attempt are refused on every worker, since each worker checks the token's attempt against the marks the coordinator broadcasts;
    • closing every worker, and reopening them in two phases: every worker first confirms that it is ready while it, the coordinator and every other worker stay closed, and only then do the coordinator and the workers reopen;
    • sequenced readiness reports, each sent again with backoff until the coordinator has it, so prepare never waits out its deadline for a worker that is ready;
    • the blockers for unreported and lost workers;
    • a worker that loses the coordinator stops itself;
    • WorkerCoordinator.stop(), for a test that starts a coordinator in the background;
    • the coordinator socket, which lives in the server's runtime directory from feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883 with the routing sockets.

    The policy model keeps only its own state, and its behavior and tests are unchanged.

  • Prepare closes every worker. It is ready once every worker reports ready, and its restarts are the union of every worker's. While a checkpoint is open, a worker re-reports its readiness on a short poll, so readiness that changes without a notification still reaches the coordinator. One example is a resources session that ends.

  • Commit asks every worker for its records. A worker whose readiness regressed after its last report refuses with checkpoint_invalid_phase, and a worker's typed error keeps its status and code through the coordinator. The coordinator writes the participant's one records file and manifest, so the store format and the controller's view do not change. Every session whose cookie may outlive a restore exports a record: a stateless resources session exports one without state, so a restore places it on a worker and tracks it until it ends.

  • Restore of sessions that span requests (resources sessions and agent sessions on the typed single_agent_turn request path). fix(server): route a session's requests to the worker that holds it #3905 routes every request by one key, the session ID, through the session's claim, and cookies and MCP session tokens carry no worker ID. A restore therefore only has to move claims:

    • Each restored session is installed on the live worker with the fewest sessions so far.
    • That worker rewrites the session's claim to name itself, replacing the claim file with one atomic rename, and only after its install succeeded. It rewrites all of its restored sessions' claims in one batch in a thread (SessionClaims.rewrite_all), holding the shared lock once per 256 renames, so a restore of thousands of sessions does not hold up the worker's event loop. Every other worker drops any claim it held on a restored session (SessionClaims.disown), for example one a cleanup close made before the restore. A worker releasing a claim checks that the file still names it and removes it while holding a lock file the rewrite also holds, so a release never removes a claim a restore moved, even one moved between the check and the removal.
    • A cookie or MCP session token from before the crash, or from a single-process server, names only the session ID, so its requests reach the worker that holds the session now. A typed resources session's session ID is its resources_session_id, so its seed and close reach the same worker.
    • A restored resources session is open on its holder, as after a successful seed, so its claim lasts until its close or verify. A restored agent session is held by the agent's own record, as after its seed.
    • When the resources participant frees a session outside a request, by a retire or a commit that drops its episode, it tells session routing through the participant's on_session_freed, so the session's claim ends a window later too. A session a retire frees stays refused until the controller forgets its rollout, so its claim lasts until that forget, and a late request that carries only its cookie still reaches the worker that refuses it.
    • So the worker count can change between a checkpoint and a restore: 1 to N, N to M, and N to 1. With one worker there are no claims, and the single process holds every session.
    • A worker that holds restored sessions and is lost after the restore sends its install, and before the resume that reopens it, fails that resume, and every later one, with checkpoint_worker_lost. Those sessions' state is gone, so the restore must be repeated on a restarted server, rather than continue those episodes without it. Because the workers reopen in two phases, such a worker lost before every worker confirmed it is ready fails the resume with no worker reopened, and a worker that joins meanwhile closes too. One lost after every worker confirmed is lost as it would be just after the resume: its sessions get a 410.
  • Restore of state that lives inside one request (environment episodes and an agent's own /run episodes):

    • The coordinator holds the records.
    • The worker that receives the replacement attempt's /run claims its record, and exactly one claim succeeds.
    • A claim is answered before any later message to that worker. A claimed episode is therefore either exported by its worker or waited for by the next checkpoint, never missed.
    • Claims are refused while a checkpoint is open, with the same code a new episode gets, so the caller retries the episode the same way.
    • A record nobody has claimed is exported again by the next checkpoint.
  • Restored records are leased until installed. A worker's claim leases a restored record. The coordinator drops it only when the worker confirms the install, and takes it back on a failed install, a session that cannot open (for example a /run of another attempt than the one the record continues), a lost reply or a lost worker, so the next commit still exports it. Prepare waits while a lease is open.

    • When the agent's hook had already installed the session, the record goes back only after the hook's teardown of that copy succeeds. The worker keeps the lease and retries the teardown in the background until then, cancelling each attempt after 60 seconds, so no checkpoint can find the record and the installed copy, and prepare names the attempt as a blocker. Once a retire, or anything else, has ended the record's lease, the teardown gets three more attempts, and is then given up with an error that says what it built outside the agent may be orphaned. If the teardown call is still running when it is given up, because it ignored its cancellation, the key stays blocked on that worker until the call ends, so the late call cannot free a session that a later /run of the key opens there. A call that never ends keeps the key unavailable on that worker for the worker's life. A later /run of the same key on that worker waits up to 30 seconds for the teardown, then gets the retryable checkpoint_restored_claim_settling. A worker that stops cleanly waits up to 30 seconds for the teardown. Past that it cancels the teardown, and the coordinator takes the record back as the worker leaves, so what the install built outside the agent, such as a sandbox, may be left behind.
    • A claim of a leased record is refused with the retryable checkpoint_restored_claim_settling, since the record may be on its way back, and the worker claims again with a growing wait, bounded by its caller's deadline.
    • Each lease carries an ID, so only the claim that holds it settles it.
    • Every worker keeps a leased record's key among the keys it may claim after a reopen, so a record that goes back later can still be claimed by the attempt it continues.
  • A restore is all or nothing across workers. If one worker's install fails, or a worker the restore installs sessions on is lost before every install has finished, even after it acknowledged its own, the others delete what they installed, and release the claims they rewrote, before the restore fails and admission reopens. A restore that failed while its install still ran, and whose install then finished but lost such a worker, is rolled back the same way, through the participant's discard_failed_restore() hook, before admission reopens. A worker whose rollback fails is kept on record as still holding the failed restore's state, and so is every worker if the rollback is cancelled. Every reopen first repeats the rollback on those workers, for at most 30 seconds or the caller's deadline. While one is still incomplete, no worker reopens, and the resume fails with the retryable checkpoint_rollback_incomplete. The record ends when the rollback succeeds there or that worker is lost. The contract is in the coordinator's install() docstring.

  • Status covers every worker. /status merges every worker's participant status and readiness, instead of the main process's idle participant.

  • Retire and forget reach every worker. The coordinator marks a retired attempt on every worker before any of them stops it, and relays forget, which also clears a resources worker's retired sessions. Forget is all or nothing across workers: the coordinator first asks every worker whether it can forget, and a worker that still tracks a retired session, or still has a seed of a retired attempt in flight, refuses with checkpoint_retire_incomplete before any worker or the coordinator forgets anything. The coordinator keeps its own marks until every worker has forgotten. A worker that joins while the forget runs adopts those marks, so it is sent the forget too, and the coordinator drops its marks only once no new worker has joined. If the forget fails or is cut short part way, the coordinator marks every worker again, so a worker that had already forgotten refuses the attempts until a forget reaches every worker. The next forget waits for that repair to finish, so the repair cannot refuse the attempts again after it. A worker that registers later receives the current marks. A worker applies a mark through a new WorkerLink.mark_local hook, which records it in the worker's retired attempts. The policy worker (PolicyWorkerLink) overrides it to also refuse, for good, its calls of those attempts that are waiting at closed admission, as feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883's policy gate does in one process. Marking is its own message, so the commit's cleanup of unclaimed restored state, which retires without marking, leaves no refusal on the workers. The coordinator retires the unclaimed records of retired attempts, and exports unclaimed records only while the commit's scope names their replacement attempt.

  • One record per session. Each participant kind defines a record's identity: the session key for agent sessions and the session ID for resources sessions. Two agent /run episodes sent with one cookie therefore stay two records. Reporting and freeing stale restored copies is part of the participant contract, which the agent participant now implements too, and stale copies are deleted in one batch per worker, on all workers at once, a bounded number at a time. Routing sends a restored session's seed to its holder, so a second copy should not arise; if one does, a commit exports one record per session ID, preferring the copy a request has used, and frees a stale restored copy that no request has used on its worker.

  • Restored state no longer continued is retired on every worker. The coordinator gathers the unclaimed restored records and each worker's unclaimed restored sessions, so a commit whose scope leaves them out retires them.

  • Carried restarts reach every worker. The coordinator passes the restarts an earlier prepare stage reported (feat(checkpoint): add the resources server participant with asynchronous session hooks #3886) to every worker, including one that joins while the checkpoint is open, and each worker clears them when it reopens.

  • The commit reply keeps each participant kind's own fields. With several workers the coordinator answers the commit, so it builds the reply from the participant kind it stands in for over the merged records. A field a kind adds to its reply, such as the agent's list of the episodes its own /run drives (feat(checkpoint): name the episodes an agent's own /run drives in the agent commit reply #4690), comes through as it does with one worker.

  • Participant capabilities are explicit. claim_restored, fence_retired and on_session_freed are part of CheckpointParticipant, with defaults, and the worker link sets them on every participant, rather than only where an attribute happens to exist.

  • Fail closed. Fewer registered workers than num_workers blocks prepare. So does a worker lost while a checkpoint is open, until the controller resumes. A worker can also be lost after the commit checked readiness but before its records are collected, and it takes what it held with it. Every coordinated participant's export, the policy model's included, therefore goes through a shared collect() on the coordinator, which fails the commit with checkpoint_export_failed if a worker was lost or fewer workers answered than expected. The failure is final for that checkpoint, because a later export could take a later cut.

  • A failed restore's recovery finishes before another checkpoint begins. This closes a gap that is on main too. A failed restore leaves admission closed until its recovery reopens it, either once an install that outlived the restore has finished, or on the controller's resume. Before, a prepare could begin a new checkpoint in that window. The recovery then never ran, and with several workers, a lost worker that held the failed restore's sessions made every later resume fail. Now a prepare is refused with the retryable checkpoint_restore_recovery_pending while the recovery is pending. The controller resumes after a failed prepare, and that resume finishes the recovery. A restore retried meanwhile first rolls back what the failed one left. Any reopen of a checkpoint also discards what a failed restore left before it reopens, and a restore that succeeds ends the pending recovery.

Three smaller changes support this:

  • An agent's own /run calls its /v1/responses for its turn loop. That call carries no session ID, so it now names its worker in a new x-ng-session-owner header, and the router forwards a request without a session ID that names another live worker there, or answers 410 if that worker is gone. The turn loop's activation therefore runs on the worker that tracks the episode. The call also carries the activation index from feat(checkpoint): add the agent session participant and Simple Agent continuation #3888, and the worker that claims a restored agent /run episode restores the turn loop's saved reply with it.
  • A worker of a server that routes sessions joins its coordinator only once its private routing socket listens, and before it serves any request: session routing enters the lifespans the app lists under ROUTED_LIFESPANS_STATE at that point, and the checkpoint link adds itself there. Before, a restore could place sessions on a worker whose socket was not yet listening, and forwarded requests got a 410. Simple Agent and the weather resources server gain the module-level app that multi-worker uvicorn imports.
  • Every Gym server now listens with a backlog of uvicorn_backlog, a new top-level setting that defaults to 16384 instead of uvicorn's 2048. The head server and each worker's private routing socket use it too, and the kernel caps it at net.core.somaxconn. In the 12,000-rollout scale test, a burst of replacement /run requests opened thousands of connections to the agent server at once and filled its queue of 2048. A session close that opened a new connection then waited on TCP retransmission backoff, past the environment server's 30-second cleanup timeout.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR adds.

flowchart LR
  C["Controller<br/>NeMo RL, or rollout collection"]
  CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"]
  K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"]
  subgraph G["One participant per Gym server"]
    E["Environment server<br/>episode steps"]
    M["Policy model<br/>held responses, generation cuts"]
    A["Agent<br/>sessions parked at boundaries"]
    R["Resources server<br/>session state"]
  end
  W["Inference worker<br/>stages cut prefixes"]
  D[("Checkpoint directory<br/>records, then manifest")]
  L[("Capture ledger<br/>retire and delete from #3938, #3939")]
  C --> CO --> K
  K --> E & M & A & R
  M --> W
  M --> L
  G --> D
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class E,A,R this
Loading

One checkpoint, a crash, and the restore, end to end:

sequenceDiagram
  participant C as Controller
  participant G as Gym participants
  participant D as Checkpoint directory
  C->>G: prepare, in order environment, model, agent, resources
  Note over G: admission closes, in-flight work parks at a boundary,<br/>undelivered model responses are held
  G-->>C: prepared, or blockers at the deadline
  C->>G: commit with the episodes the controller continues
  G->>D: each participant writes its records, then its manifest
  Note over G: restored state the commit's scope leaves out is deleted
  C->>C: publish the checkpoint with the controller's own state
  C->>G: resume, in order resources, agent, model, environment
  Note over C,G: crash - every Gym process dies
  C->>G: restore the checkpoint in fresh processes, all or nothing
  D-->>G: records installed under attempt N+1, attempt N's capture ledger retired
  C->>G: resume
  C->>G: /run as attempt N+1 continues each episode from its boundary
  Note over C,G: retiring an episode, only while no checkpoint is open
  C->>G: retire - environment, then agent, then model and resources
  Note over G: each server marks the attempt, stops its work, waits, frees its state, then replies
  C->>G: forget, once the rollout is done for good - same order
  Note over G: the mark refusing the retired attempt goes away
Loading

This PR

One server with several uvicorn workers is still one participant. Its coordinator, in the main process, drives each worker's own participant; the router sends a session's requests to the worker that claims it.

flowchart LR
  CO["Coordination"] -->|"control call on the shared port"| W1["worker 1<br/>participant and router"]
  CO -->|"control call on the shared port"| W2["worker 2<br/>participant and router"]
  W1 <-->|"Unix socket"| C["Coordinator in the main process<br/>one participant: phases, records,<br/>restored episodes"]
  W2 <-->|"Unix socket"| C
  W1 <-->|"forward to the session's claimant"| W2
Loading

Restoring sessions into new workers. A cookie or MCP session token names only the session ID; the restore rewrites that ID's claim to name the worker that holds the session now.

sequenceDiagram
  participant C as Coordinator
  participant W1 as Worker 1
  participant W2 as Worker 2
  participant X as Caller
  C->>C: read the records, place each session on a live worker
  C->>W2: install session s
  W2->>W2: rewrite the claim of s to name worker 2
  C->>W1: install nothing here, and drop any claim on s
  X->>W1: request with a cookie naming session s
  W1->>W1: read the claim of s: worker 2
  W1->>W2: forward
  W2-->>X: reply from the restored session
Loading

Whole-episode state, such as an environment episode or an agent's own /run, waits in the coordinator. Exactly one worker claims it, whichever receives the replacement attempt's /run.

sequenceDiagram
  participant X as Controller
  participant W as Worker that receives the call
  participant C as Coordinator
  X->>W: /run for attempt N+1
  W->>C: claim the episode's restored record
  C-->>W: the record, or nothing if another worker claimed it
  W->>W: continue the episode from its boundary
  Note over C: unclaimed records stay while the controller's commit scope lists them,<br/>and are retired when it no longer does
Loading

Where this sits in the stack

This is one PR in a stack of draft PRs for partial-rollout checkpointing. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.

The stack builds on the token-capture cleanup PRs #3938 (capture ledger retire and delete) and #3939 (complete-record retire and delete), which have merged. #3882 to #3889, #3903 and #4690 have merged too. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.

This PR builds on #4997, which has merged and gives each checkpoint concept one name before the feature is released. This PR uses those names, and its own new names follow them: its error codes carry the checkpoint_ prefix, a participant marks rather than fences, and a restored record a failed claim is still returning is refused with checkpoint_restored_claim_settling.

#4887, which has merged, fixes a Simple Agent prepare race in the turn loop's policy call. This PR changes Simple Agent's /run call to its own /v1/responses, and the two changes compose.

  1. feat(checkpoint): add the participant control plane, episode steps, and coordination #3882 (ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)
  2. feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883 (ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)
  3. feat(token-capture): add worker staging helpers for generation cuts #3884 (ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)
  4. feat(checkpoint): continue environment server episodes from their boundaries #3885 (ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)
  5. feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 (ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks (merged)
  6. feat(checkpoint): declare twenty training verifiers stateless with replayable verification #3887 (ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verification (merged)
  7. feat(checkpoint): add the agent session participant and Simple Agent continuation #3888 (ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuation (merged)
  8. test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 (ananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordination (merged)
  9. feat(checkpoint): spans and metrics for partial-rollout checkpoints #3903 (ananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpoints (merged)
  10. feat(checkpoint): name the episodes an agent's own /run drives in the agent commit reply #4690: feat(checkpoint): name the episodes an agent's own /run drives in the agent commit reply (merged)
  11. fix(server): route a session's requests to the worker that holds it #3905 (ananthsub/fix-session-worker-routing): fix(server): route a session's requests to the worker that holds it, (merged)
  12. fix(simple_agent): stay at the boundary until a held policy reply's body arrives #4887 (ananthsub/simple-agent-held-reply-boundary): fix(simple_agent): stay at the boundary until a held policy reply's body arrives (merged)
  13. refactor(checkpoint): one name per checkpoint concept before release #4997 (ananthsub/checkpoint-naming): refactor(checkpoint): one name per checkpoint concept before release (merged)
  14. feat(checkpoint): resources, agent, and environment servers with several workers #3909 (ananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workers (this PR)
  15. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 (ananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operations

#3893 (ananthsub/partial-ckpt-rollout-collection, checkpoint evaluation runs from rollout collection) is independent of this chain and targets main. Its two spans, gym.checkpoint.collection.checkpoint and gym.checkpoint.collection.restore, are added to it once #3903 merges, or in a small follow-up if #3893 merges first.

Server ports that build on the stack: #3894 (Workplace Assistant), #3895 (Gymnasium), #3896 (Blackjack), #3897 (indirect prompt injection), #3898 (proof refinement). Session routing for several workers is #3905, which has merged. This PR's own additions to nemo_gym/session_routing.py are the claim rewrite for restores and its batched form, the lock that keeps a release from removing a claim a rewrite has just written, claims that last until a retired session's forget, the x-ng-session-owner header, and the hook that lets a worker join its coordinator once its private socket listens.

Relationship to the old stack

The old stack (#2939 to #2946) required one worker for every checkpointing server except the policy model. The policy model already had multi-worker support in #3883, as a coordinator in the main process with worker gates. This PR generalizes that coordinator rather than adding a second mechanism. It replaces the three num_workers=1 guards the re-cut stack kept for resources, agent, and environment servers.

Issue

No tracking issue exists. Multi-worker support for these servers was a stack requirement. Session routing, which it depends on, is #3905, which has merged.

Validation

Run on this branch, on the current main, which includes #4997:

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider -n 16 tests/unit_tests/test_checkpoint_*.py tests/unit_tests/test_session_routing.py: passed. test_checkpoint_participant_workers.py runs each scenario with a real coordinator socket and two or three worker links:
    • prepare closes every worker and waits for the episodes of all of them, and commit collects every worker's records;
    • readiness that changes without a notification still reaches the coordinator, and a readiness report that did not arrive is sent again;
    • every worker leaves out the restarts an earlier stage reported until resume;
    • a lost worker blocks prepare and commit until resume, fewer workers than configured block prepare, and a worker lost after the commit checked readiness and before the export fails the commit;
    • a worker lost in the middle of a message is reported as checkpoint_worker_lost;
    • a restored environment episode and a restored agent /run episode are each claimed by exactly one worker, and an unclaimed one is exported by the next checkpoint while the scope continues it, and retired, on every worker, when it does not;
    • a restored record returns to the coordinator when its worker's install fails, the session cannot open, the claim reply is lost, or the worker is lost, and prepare waits while it is unsettled;
    • when the teardown of a rolled-back install raises, raises CancelledError by itself, or hangs past its attempt timeout, the worker forgets the session at once and the record goes back only after a later attempt succeeds; a later /run of the same key on that worker waits for the teardown;
    • a teardown that keeps hanging keeps the record unclaimable and its attempt named among prepare's blockers, and a /run of its key on that worker gets checkpoint_restored_claim_settling; once a retire ends the lease, a teardown that keeps failing, or one that ignores its cancellation, stops within its cap and leaves nothing behind, and one that succeeds within the cap still settles its claim;
    • a teardown given up while its call still runs keeps its key unclaimable on that worker until the call ends, and the key can be claimed again after;
    • a settlement that is retried after another claim, on another worker or the same one, has leased the record neither ends that lease nor returns the record;
    • a claim that arrives while a failed claim's record is on its way back waits and continues from the record;
    • restored agent and resources sessions are spread over the live workers and each one's claim names its holder, through repeated restores, from a single-process server too, and for a worker that joins while the restore installs;
    • a worker that claimed a session before its restore drops the claim, and a restored resources session keeps its claim until the participant frees it;
    • a worker that holds restored sessions and is lost after the restore fails the resume and every later one;
    • a worker lost after it acknowledged its install while another worker still installed fails the restore and leaves nothing installed, and so does a loss after an install that outlived its restore's deadline, after which admission reopens;
    • no worker reopens until every worker holding restored sessions confirmed it is ready, a holder lost before that leaves every worker closed, including one that joins afterwards, and a holder lost after it leaves the others serving;
    • a resources session freed by a retire keeps its claim, so a cookie-only request through another worker is refused, until the forget, after which the claim ends;
    • a clean disconnect stops the worker's report, readiness poll and claim-settlement tasks, after waiting a bounded grace for a teardown still running, which settles its claim before the worker leaves if it finishes in time and is cancelled if it does not;
    • an export sends its first message before it has encoded every record;
    • two real counter resources servers with session routing serve a restored session from its old cookie on either worker, and every worker refuses the replaced attempt's MCP token;
    • two real agent servers route an agent session cookie from before a restore to the worker that holds the session now, and route a cookieless seed and close of a restored session there;
    • a worker joins its coordinator only once its private socket accepts;
    • two of an agent's own /run episodes sent with one cookie jar, on two workers, and an unclaimed restored one are all exported and named in the commit reply's run_episode_ids;
    • a stale restored copy is reported and freed, on several workers at once;
    • a worker message that times out or is cancelled stops the worker's handler before a retry runs;
    • a restore whose install fails on one worker, or whose worker is lost while it installs, leaves nothing installed and no claim behind;
    • a rollback that fails on a worker keeps every worker closed, and the resume fails with checkpoint_rollback_incomplete, until the rollback is repeated successfully or that worker is lost, and nothing is left afterwards;
    • a prepare during a failed restore's recovery is refused with checkpoint_restore_recovery_pending, and once the recovery has run, a worker that held the failed restore's sessions and was lost leaves no worker closed; a resume of a checkpoint that began during the recovery anyway still rolls the failed restore back, and a holder lost after a retried restore succeeded still fails its resume;
    • restoring many sessions rewrites their claims without stalling the event loop;
    • a retire whose teardown outlasts the old fixed message timeout, but stays within the request's deadline, succeeds;
    • a forget cut short leaves every worker refusing the attempts, a forget one worker cannot do yet changes nothing, a worker that joins while a forget runs forgets too, and a forget retried while the repair of one cut short runs waits for that repair;
    • /status adds up every worker's report and answers without a worker that does not;
    • a large export travels in chunks, and a worker lost between chunks fails the commit and leaves nothing buffered;
    • every participant takes the hooks a worker link sets.
  • test_session_routing.py adds: an old cookie follows its session to the worker a restore moved it to, a release after a restore moved the claim leaves the new claim, a rewrite that lands between a release's check and its removal survives, a batched rewrite whose caller is cancelled finishes before the worker counts its claims, and a request without a session goes to the worker its x-ng-session-owner header names, or gets a 410 if that worker is gone.
  • test_server_utils.py and test_session_routing.py also check the listen backlog: it defaults to 16384 with one worker and with several, follows uvicorn_backlog, refuses a value below 1 before uvicorn starts, reaches the head server, and reaches each worker's private routing socket. These checks fail without the setting.
  • test_checkpoint_agent.py: the coordinator the multi-worker agent test starts is stopped, with its thread and socket gone.
  • test_checkpoint_control.py adds: a prepare while a failed restore's install still runs, or after its recovery could not reopen, is refused with checkpoint_restore_recovery_pending until admission reopens, and then succeeds.
  • The full core unit suite (tests/unit_tests, sixteen processes, in a development environment with CI's extras): passed, except one benchmark-listing check that fails the same way on main on this workstation, whose host name resolves to 127.0.1.1.
  • The simple_agent, example_session_state_mgmt and single_agent_turn tests, each run on its own: passed. The simple_agent tests include fix(simple_agent): stay at the boundary until a held policy reply's body arrives #4887's check that a held reply's body keeps the session at its boundary.
  • NEMO_GYM_MULTIWORKER_E2E=1 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/session_routing: passed.
  • The checkpoint end-to-end suite with several workers, and a stress test that checkpoints a thousand counter rollouts across 8 resources and 4 agent workers and restores onto 4 and 8, and the reverse, run on test(checkpoint): check checkpoint invariants over random sequences of operations #4197, which carries them; every rollout finished or continued with reward 1.0, and nothing was left behind.
  • pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.
  • Every commit carries a Signed-off-by line.

Rollout evidence

The real-model and scale runs below ran on commit 16713dafec, #4197's tip on top of this PR's ffa3646fa6. The many-worker stress rows ran again on 3fea619731, #4197's tip on top of this PR's e495d7b05b, which adds the changes to failed-restore recovery, rollback, forget, teardown key blocks and batched claim rewrites. They ran before this PR was rebuilt on #4997; the rebuild renames names and error codes and changes no behavior. The workstation was shared with other jobs during every run, with a load average of 26 to 55 on 24 cores, so the times below are slower and noisier than on an idle machine.

  • Real model, on one workstation: Qwen/Qwen3-4B-Instruct-2507 served by vLLM 0.29.0 on one RTX 6000 Ada (--enable-auto-tool-choice --tool-call-parser hermes --max-model-len 32768), driving real Gym server processes that the opt-in tests in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 and this PR checkpoint mid-run, kill and restore. Each scenario ran with one and two policy workers and with one and two workers per server, and the whole set passed eight runs in a row.

    • Saved rollouts. Weather rollouts on the typed single_agent_turn request path, sent 20 ms apart until the first one finished. Each checkpoint caught 20 to 41 rollouts in flight, most inside their agent turn and the rest parked before closing the agent session, before verification, or before returning, while 2 to 13 had finished before it. After every Gym process group was killed and the checkpoint restored into fresh processes, every continued rollout finished with reward 1.0, as did every rollout that finished before the checkpoint.
    • Restarts. Half the rollouts used a restart_only resources server. Prepare listed every restart still in flight and no saved rollout, and the commit left the restarts out. After the kill, the 10 to 21 restarts in each run were retired and started over from their input while the 10 to 16 saved rollouts continued, and every rollout finished with reward 1.0.
    • With two workers per server, the restore places each restored session on one worker and rewrites its routing claim to name that worker, and the replacement /run claims its restored episode on whichever worker receives it. The runs with two workers per server go through that path.
    • No Gym process was left running after any run.
    • NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_CHECKPOINT_VLLM_URL=http://127.0.0.1:8129/v1 NEMO_GYM_CHECKPOINT_VLLM_MODEL=Qwen/Qwen3-4B-Instruct-2507 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint -k real_model
  • Running the real-model tests with two workers per server found a gap in the e2e harness: uvicorn's workers import a server by the path its config names, so a test server that subclasses a repository server ran its parent's code with several workers, and the restart-only test server was not restart-only. Test servers now always run one worker, and the fake-backend restart scenarios also run with two workers per server; the two-worker restart scenario fails without the harness fix.

  • Scale, on one workstation with every Gym server and the fake inference backend. The run used weather rollouts on the typed single_agent_turn request path, with token capture and generation cuts on. Each rollout's first model call returns and its second is held, so every rollout is in flight at the checkpoint. At most 500 rollouts open connections at once, the scale test's default. The test checkpoints, kills every Gym process group, restarts, restores, and runs the replacement attempts. Every server ran with two workers, and each row is the faster and slower of two runs:

    Rollouts Policy workers Environment, agent, and resources workers In flight at checkpoint (exported) Prepare (s) Commit (s) Restore (s) Replacements (s) Replacement failures
    8,000 2 2 8,000 0.82 to 1.12 1.03 to 1.37 3.39 to 5.86 29.1 to 58.5 0
    12,000 2 2 12,000 1.22 to 6.48 3.56 to 5.08 4.51 to 14.66 31.1 to 72.7 0
    • Most of each prepare is the policy model server cutting the in-flight generations: 0.80 to 1.09 s at 8,000 rollouts and 1.18 to 6.38 s at 12,000.
    • Records at 12,000 rollouts: 9.8 MB for the environment, 39.4 MB for the agent, 23.5 MB for the model, and 1.7 MB for the resources server. At 8,000: 6.5 MB, 26.3 MB, 15.7 MB, and 1.1 MB.
    • Every in-flight generation was cut, 12,000 and 8,000, and every cut call was continued.
    • Every exported rollout scored 1.0 after the restore.
    • The kernel recorded no listen overflow during any run.
    • In a repeat of the 8,000-rollout run with a 5-second session close window on the agent and resources servers, no claim, session, or request in flight was left on either server once the window had passed.
    • An earlier version of the scale test sent every /run at once. Repeated at 12,000 rollouts, it failed in about 3 of 10 runs, with 1 to 509 replacement rollouts failing their session cleanup, on this PR and on its earlier version alike. Two causes add up. With uvicorn's listen backlog of 2048, the agent server's accept queue overflowed, and a close on a new connection waited on TCP retransmission backoff; this PR's uvicorn_backlog raises that limit. Each worker also runs uvloop, which accepts about one new connection per event-loop iteration, so a busy worker can leave hundreds of connections unaccepted for tens of seconds. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 therefore bounds how many rollouts the scale test has opening connections at once.
    • At 16,000 rollouts, the test still does not reach its checkpoint on this workstation; test(checkpoint): check checkpoint invariants over random sequences of operations #4197 describes why.
    • NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_CHECKPOINT_SCALE=12000 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint/test_checkpoint_e2e.py -k "test_checkpoint_at_scale and servers2-2"
  • Many workers, on this version, with test(checkpoint): check checkpoint invariants over random sequences of operations #4197's opt-in stress test: the /run relay over Simple Agent and the counter resources server, 100 rollouts finished first and 1,000 held mid-episode at the checkpoint, every Gym process killed, and the checkpoint restored onto other worker counts:

    Resources and agent workers, before and after Continued with reward 1.0 Resources workers holding sessions, before and after Commit (s) Restore (s) Replacements (s) Claims and sessions left after the close window
    8 and 4, then 4 and 8 1,000 of 1,000 8, then 4 0.13 0.09 4.31 none
    4 and 8, then 8 and 4 1,000 of 1,000 4, then 8 0.09 0.09 4.79 none

    Every rollout that finished first scored 1.0 too, and each continued rollout made only the model calls it still needed.

Compatibility

  • Resources, agent, and environment servers now accept num_workers > 1 with checkpointing on. Single-worker servers behave as before, and their records do not change.
  • Checkpoints written by an earlier version of this PR do not load, because their records carry owner and session_id fields the record models no longer accept. Checkpoints from main load as before. The earlier version never shipped.
  • The worker count may change between a checkpoint and a restore, in either direction: from one worker to several, from several to one, or between different counts.
  • With several workers, a restored environment episode or agent /run episode whose replacement /run arrives while a checkpoint is open is refused with checkpoint_admission_closed, as a new episode is, and the caller retries it. With one worker, it is admitted and parks at its first boundary.
  • With several workers, commit and restore carry each worker's records over the coordinator's socket, in chunks of about 2 MiB, each encoded off the event loop while the one before it is sent and checked. A worker therefore holds at most two encoded chunks instead of its whole encoded export. In a synthetic 177 MB export with the coordinator and two workers in one process, peak memory above the level before the commit was 635 to 649 MB, against 783 to 788 MB when each worker encoded its whole export first, and the longest event-loop stall had a median of 60 ms. Servers with very large sessions still put all their records through the main process. Per-worker record files are the alternative if that becomes a bottleneck.
  • New top-level setting uvicorn_backlog, default 16384. Every Gym server, including the head server and each worker's private routing socket, now listens with this backlog instead of uvicorn's 2048. The kernel caps it at net.core.somaxconn, so a host with a lower value gets that value.
  • New header x-ng-session-owner: on a server with several workers, a request without a session ID that names another worker in it is forwarded to that worker, or gets a 410 if that worker is gone.
  • New error code checkpoint_worker_lost (409), for a worker lost during a coordinator's message, for a restore that lost a worker it installed sessions on, and for a resume after a worker holding restored sessions was lost.
  • New error code checkpoint_restored_claim_settling (503), retryable, for a replacement /run whose restored record a failed claim is still returning, after the worker's own retries ran out.
  • New error code checkpoint_rollback_incomplete (503), retryable, for a resume while a worker could not yet delete what a failed restore installed on it. Admission stays closed until it can.
  • New error code checkpoint_restore_recovery_pending (503), retryable, for a prepare while a failed restore's recovery has not reopened admission yet, on every participant, with one worker or several. The controller resumes, which finishes the recovery, then prepares again.
  • A resume that retries a failed restore's recovery and fails again with a typed error now returns that error's own code, such as checkpoint_worker_lost or checkpoint_deadline_exceeded, instead of the generic checkpoint_error.
  • With several workers, a cookie from before a crash whose session a restore did not install, for example because the controller left the resources server out of the restore, is now served where it lands and claimed there, as on one worker. The previous version answered such a request with a 410.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:core Shared APIs, servers, telemetry, health, and registries labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch from d64723f to a92580a Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch from a92580a to 6647f1c Compare October 2, 2026 19:43
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch from 6647f1c to ec6b311 Compare October 2, 2026 21:17
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch 2 times, most recently from 51dc9c8 to bd1073e Compare October 6, 2026 19:28
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch from bd1073e to 8b7a26d Compare October 6, 2026 20:54
@copy-pr-bot

copy-pr-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-multi-worker branch from 8b7a26d to c059299 Compare October 6, 2026 22:42

@macandro96 macandro96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I reviewed the multi-worker checkpoint changes and verified the main findings against the code; several were reproduced with small tests.

Blockers

  1. Restored records are removed before the worker confirms the claim (participant_workers.py:183).
  2. Agent records are de-duplicated by the wrong identity, and agent stale copies are never cleaned up (participant_workers.py:225).
  3. Cookie-less agent seed/close are not serialized across workers (session_routing.py:209).
  4. Cancelling a worker RPC doesn't stop the worker's handler, so retries overlap (workers.py:143).

Fix before merge
5. Restore across workers is not atomic (needs confirmation of the intended contract) (participant_workers.py:318).
6. Retire uses a fixed 30s timeout instead of the request deadline (workers.py:450).

Minor / suggestions: frozen /status with several workers, serial stale-copy cleanup, agent owner vs cookie owner.

Perf follow-up: large synchronous JSON frames.

Relation to #4690: #3909 passes the merged worker records to the agent participant's commit_reply(), so #4690 can produce run_episode_ids with several workers. Finding 2 occurs before that hook: if two distinct agent-owned /run records share a cookie session ID, _one_copy_per_session() drops one, so run_episode_ids also omits that run. This is a #3909 de-duplication bug exposed by the combined stack, not a bug introduced by #4690.

return None
if not self.accepting:
raise AdmissionClosedError(f"{self.kind} admission is closed for a checkpoint")
record = self.restored.pop(body["key"], None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Restored record is removed before the worker confirms the claim.

handle_now pops the record from self.restored as soon as a worker asks for it. If the reply is lost (timeout/disconnect), the worker crashes, or the agent's restore_agent_sessions hook fails (agent.py _claim removes the session on error), the checkpoint state is gone for good: the coordinator no longer holds it and no worker exports it. A single process restarts from the checkpoint on a crash, so this loss only exists with several workers.

Suggestion: lease the record to the claiming worker and drop it only after an explicit acknowledgement that it was installed; return it to restored on failure or worker loss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A claim now leases the record to the claiming worker instead of removing it.
The worker installs the record inside the claim and then settles the lease: installed, or not, if the install or the agent's restore_agent_sessions hook raised.
A record that was not installed goes back to restored, where any worker can claim it again and the next commit exports it if nobody does.
If the claim's reply is lost or times out after the coordinator leased the record, the worker gives it back the same way.
A settlement that cannot be sent is retried in the background until it is sent or the connection closes.
A lost worker's leases go back to restored when its connection closes, and a retire of the attempt drops the lease, so a lease ends on settlement, on worker loss, or on retire.
While a lease is open, prepare reports the replacement attempt as a blocker, so a commit never starts with a record half installed.

Covered by:

  • test_a_claimed_record_whose_install_fails_goes_back_and_another_worker_claims_it;
  • test_a_claimed_record_whose_reply_is_lost_goes_back;
  • test_a_claimed_record_of_a_lost_worker_goes_back_and_the_next_checkpoint_exports_it;
  • test_an_unsettled_claim_holds_up_prepare_until_the_install_finishes.
    On the previous head the record never came back in the first three, and prepare reported prepared while the install was still running in the last.

for raw in reply["records"]:
# Built and checked on the worker, then parsed from its message: its payloads are JSON.
record = self.record_model.model_validate(raw, context=PARSED_JSON_CONTEXT)
session_id = getattr(record, "session_id", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Agent records are de-duplicated by the wrong identity.

Records are keyed by the cookie session_id, but agent sessions are identified by session_key. This fails both ways:

  • Different session_keys, one cookie: two agent-owned /run episodes sent with the same agent cookie collapse into one record, and the other episode is silently dropped (only an error is logged). Gym's current first-party callers do not reuse cookies this way, but an external controller with a cookie jar can.
  • Same session_key, different cookies: after a restore, a cookie-less re-seed can land on another worker and get a new session ID. Both the stale restored copy and the live copy are exported, giving two records for one episode.
  • AgentSessionParticipant has no unclaimed_sessions() / delete_unclaimed_session(), and the getattr fallback at L450 silently reports []. So agent stale copies are never classified or freed.

Suggestion: let the participant define the logical identity (session_key for agents, session_id for resources), and make stale-copy reporting and cleanup an explicit participant contract implemented for agents.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Each participant kind now names a record's identity with record_identity(): the session key for agent records, the session ID for resources records, and none for environment records.
De-duplication uses it, so two agent /run episodes sent with one cookie are both kept, and a stale restored copy and a live copy of one agent session are exported once.
Stale-copy reporting and cleanup are now an explicit part of CheckpointParticipant: unclaimed_sessions() and delete_unclaimed_session() raise NotImplementedError unless the kind implements them, and the worker calls them directly, with no getattr fallback.
The agent participant implements them over its restored sessions that no activation or request has used; the environment and restart-only agent participants implement them as empty, since nothing of theirs is restored on a worker.

Covered by test_two_agent_run_episodes_sent_with_one_cookie_are_both_exported, which runs two /run episodes under one cookie session on two workers and checks both the written records and the records the agent kind's commit_reply() receives.
That is the list #4690's run_episode_ids would be built from, so both runs would be in it.
On the previous head only run:a was kept.
And by test_a_stale_restored_agent_session_and_its_live_copy_export_once_and_the_stale_copy_is_freed, which re-seeds a restored session on the other worker under a new cookie session. On the previous head both copies were exported, and the stale one was never freed.

Comment thread nemo_gym/session_routing.py Outdated
return
owner = self._owner(HTTPConnection(scope))
owner = self.aliases.get(owner, owner)
if owner is None or owner == self.routing_id:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Cookie-less agent operations are not serialized across workers.

A request with no cookie/owner is served by whichever worker receives it. Seed idempotency (_locked_agent_session by agent_session_id) and the unknown-ID close receipt in base_responses_api_agent.py (seed_agent_session / close_agent_session) only exist inside one worker. With several workers:

  • a retried identical seed can reach a second worker and create a duplicate session/harness state;
  • a close for an unknown ID on worker A doesn't stop a delayed seed on worker B, so the close incorrectly "succeeds".

Suggestion: reserve agent_session_id ownership (and close tombstones) at the coordinator before the operation is served, then route it to that owner.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #3905, which this PR carries unchanged: a seed or close without a cookie naming its owner is routed to the first worker that saw its agent_session_id, and a close of an unknown ID claims the ID, so a delayed seed reaches the same worker and is refused.
The covering tests are in tests/unit_tests/test_session_routing.py; the reply on #3905 lists them.

Restored agent sessions now agree with those claims: a restore claims each agent session it installs on its worker before running the agent's hook, and releases the claims if the hook fails.
test_a_restored_agent_session_is_seeded_and_closed_on_its_worker_without_a_cookie runs two real agent servers with session routing and checkpointing, restores a session onto one worker, and sends the replacement attempt's seed and close without a cookie through the other worker. Both reach the worker that holds the session.
On the previous head the other worker claimed the ID itself and closed a second copy, and the restored session was never closed.

# Typed, so callers that fall back on a lost coordinator, such as a cut claim, do so here too.
raise CoordinatorUnavailableError(f"checkpoint message {kind!r} failed: {error!r}") from error
finally:
self._pending.pop(message_id, None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] A timeout/cancel here doesn't stop the worker's handler.

On timeout or cancellation, only the local reply future is dropped. The worker's _serve task keeps running, so a retry can overlap the original operation. Verified with a repro: a retire whose deadline passed, retried immediately, ran retire_session_state twice concurrently for the same session. In a single process, _concurrently cancels each stop and waits for it to finish before releasing the lock; that invariant is lost here. The same applies to other control messages (e.g. a slow export finishing after admission reopens).

Suggestion: track inbound tasks by message ID and send a cancel/abort that the worker acknowledges, or coalesce retries onto the in-flight operation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by cancelling the other side's handler.
When a caller of Channel.call stops waiting, at its timeout or because it was cancelled, it sends a cancel naming the message, and the other side cancels that handler and answers once the handler has stopped.
The caller returns only then, so each stop is cancelled and awaited before the control lock is released, as _concurrently does in one process.
It waits as long as the handler takes while the other side keeps answering a ping, and gives up, with an error in the log, only if the other side stops answering or the connection closes.
A cancel that arrives after the handler finished is answered at once.
This applies to every message kind, export and install included.
Cancellation was chosen over coalescing a retry onto the operation in flight because it covers every kind in one place and keeps the single-process behavior, where a cut-short operation has stopped before its retry starts; coalescing needs an identity per operation and would still leave an export running after the checkpoint resumed.

Covered by:

  • test_a_retire_cut_short_by_its_deadline_stops_the_worker_s_teardown_before_a_retry_starts_it, your repro: on the previous head the teardown was still running when the cut-short retire returned, and the immediate retry ran it a second time alongside;
  • test_a_slow_export_is_stopped_on_every_worker_before_resume_returns: on the previous head the worker's export was still running after the resume;
  • test_a_cancel_that_arrives_after_the_handler_finished_is_answered_at_once.

)
for worker_index, placed in installs.items()
}
await self.call_each(messages, timeout=None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Needs confirmation] Restore across workers is not atomic.

If one worker's install fails after others succeed, the coordinator never records its aliases/placements/restored, but the successful workers keep their installed state. The failed-restore path in control.py (_open_after_failed_restore) then reopens admission without rolling those workers back. The comment there says the controller is expected to retire what the install brought. If that's the contract, could you confirm and document it? Otherwise this needs prepare/install/commit-or-abort or a rollback of the workers that succeeded before reopening.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and closed the gap rather than relying on the controller.
The contract on one process is that a failed restore leaves nothing usable, and rollout collection's coordination.restore() then retires the restored attempts everywhere.
With several workers, a failed install on worker B used to leave worker A's installed state, and admission reopened before that retire.
Now the coordinator's install() collects every worker's outcome; if any worker failed or was lost, the workers that installed theirs delete it again and reset their routing tables to the coordinator's, and only then does the restore fail.
Admission reopens after that, so nothing of the failed restore can be used, and the controller's retire finds nothing left to free.
The contract is in the docstrings of the coordinator's install() and of participant_workers.

Covered by test_a_restore_that_fails_on_one_worker_leaves_nothing_on_the_others, where B's hook raises after A installed: A holds nothing, both routing tables are empty, admission is open, and the controller's retire succeeds. On the previous head A still held its session.
And by test_a_worker_lost_while_it_installs_fails_the_restore_and_leaves_nothing_on_the_others.

Comment thread nemo_gym/_checkpoint/workers.py Outdated

async def retire(self, episode_id: EpisodeId) -> None:
self.retire_restored(episode_id)
await self.broadcast(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retire uses a fixed 30s timeout instead of the request deadline.

The single-process retire is bounded only by the RetireRequest deadline (_within(request, ...) in control.py). Here it's MESSAGE_TIMEOUT_SECONDS, so a teardown hook slower than 30s returns a 503 checkpoint_coordinator_unavailable even with minutes of deadline left, while the worker keeps retiring. Verified with a scaled-down repro. delete_restored has the same issue (it can await retire_session_state too). Suggest passing the request deadline through, together with the cancellation fix on Channel.call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed together with the cancellation above.
Retire, delete_restored, forget and its check no longer have a fixed timeout: the control call's deadline bounds them, and when it passes the workers' handlers are cancelled and awaited.
A teardown slower than 30 seconds now finishes within a longer deadline instead of returning a 503 while it keeps running.

Covered by test_a_teardown_slower_than_the_message_timeout_finishes_within_the_retire_deadline, scaled down with the fixed timeout patched below the teardown's time; it also deletes a restored session through a commit's cleanup. On the previous head the retire failed with checkpoint_coordinator_unavailable.

Comment thread nemo_gym/base_resources_server.py Outdated
app,
participant,
num_workers=self.config.num_workers or 1,
status=participant.status_extra(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: /status is a frozen snapshot with several workers.

participant.status_extra() is evaluated once at setup in the main process, whose participant never serves sessions. So retired_sessions always reports 0 with num_workers > 1 (verified). The e2e assertion that it's 0 passes regardless. Suggest aggregating status from the workers' reports.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A status request now asks every worker for its participant's status_extra() and readiness, waiting up to two seconds for each, and merges the answers: numbers add up, lists join, and flags hold only if every worker's does.
The status's report merges the workers' readiness the same way outside a checkpoint; while one is open it is the readiness prepare reads, as before.
status_workers says how many workers answered, so a status taken while a worker is unreachable still answers and says so.

Covered by test_status_adds_up_what_every_worker_reports and test_status_answers_without_a_worker_that_does_not; on the previous head retired_sessions was missing from the coordinator's status.
The end-to-end retire test now runs with one and with two workers per server.
It first sends the resources server a retire whose deadline has passed, which only fences the episode's session, and checks that the status counts that session as retired; after the full retire and after the forget it checks for none.
The old assertion, retired_sessions == 0 after the forget, was true whatever the workers held, so it could not catch this; on the previous head the new one fails with two workers per server.

)
if len(live) > 1:
LOGGER.error("%s session %s is live on several workers; exporting one copy", self.kind, key)
for worker_index, session_id in stale:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: stale-copy deletes are serialized on the export critical path.

Each delete_unclaimed is awaited in sequence (each up to 30s, and each may call a user teardown hook), so export latency is the sum of all deletions. The export task is shielded: after a restore with many re-seeded sessions, the commit caller can hit its deadline while cleanup continues, and a retry then waits for the same export. Suggest batching per worker and running workers concurrently, or otherwise using bounded concurrency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The coordinator sends each worker one message with all of its stale copies, to every worker at once, and each worker deletes up to 16 at a time.
The export is bounded by the commit's deadline as before.

Covered by test_stale_resources_copies_on_several_workers_are_deleted_at_once and its agent twin, with six teardowns of 0.3 seconds on two workers: the commit takes about one teardown's time. On the previous head it took 1.8 seconds.

Comment thread nemo_gym/_checkpoint/agent.py Outdated
if not self.accepting and not (seed and session_key in self._seeding):
raise AdmissionClosedError("agent admission is closed for a checkpoint")
self._add(_Session(key=session_key, episode_id=episode_id))
self._add(_Session(key=session_key, episode_id=episode_id, owner=self.owner, session_id=current_session_id()))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: agent records the serving worker, not the cookie's owner.

Resources sessions keep the owner named in the session cookie; agent sessions record self.owner. If a client keeps an aliased pre-restore cookie and seeds here, the record says Y while the cookie still says X, so after the next restore X may not be aliased and the cookie gets a 410. Current first-party flows do not appear to reuse aliased cookies this way, so this is low reach, not a blocker.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. An agent session now records the owner its cookie names, as a resources session does, and falls back to the serving worker for a cookie without one.
Covered by test_an_agent_session_seeded_with_a_cookie_from_before_a_restore_keeps_the_cookie_s_owner, with real agent servers: a client keeps a cookie from before a restore, seeds a new agent session with it, and the next checkpoint records the pre-restore owner, which its restore aliases again. On the previous head the record named the serving worker.

Comment thread nemo_gym/_checkpoint/workers.py Outdated
# The same encoder as the checkpoint writer:
# orjson would refuse integers beyond 64 bits and silently turn NaN and infinities into null,
# so a record could fail to reach a worker, or arrive changed.
data = json.dumps(message, separators=(",", ":")).encode()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perf follow-up: large JSON frames on the event loop.

Frames up to 1 GB are encoded/decoded with json.dumps/json.loads synchronously on the loop, and the commit collects every worker's export reply before processing any. (The choice of stdlib json over orjson is reasonable per the comment above.) Consider chunking/batching records, moving (de)serialization off the loop, and processing worker replies incrementally.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, keeping the stdlib json encoder.
A worker now encodes its export once, in a thread, into messages of about 2 MiB, and the coordinator checks each one in a thread as it arrives, while the other workers are still exporting.
Frames over 256 KiB are decoded in a thread, and a restore's install messages are encoded in one.
A worker lost between chunks fails the commit, and the coordinator keeps nothing of that export.

Measured with two workers exporting 200 sessions of about 1 MB each (185 MB of records), all in one process so that a stall anywhere shows: the longest event-loop stall during the commit fell from about 760 ms to between 75 and 130 ms, and the commit took about the same time, 2.5 to 2.8 seconds.
With 4,000 small sessions (17 MB), the longest stall fell from about 200 ms to about 160 ms; what remains is per-record work in the participants, which runs on the loop on one process too.
Covered by test_a_large_export_arrives_in_chunks_with_every_record and test_a_worker_lost_between_export_chunks_fails_the_commit_and_keeps_nothing.

@ananthsub

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Every item is fixed in this round, in the rebuilt commit of this PR (295ad31e34), except item 3, which is fixed in #3905 (78dfc99c7d) and carried here unchanged. Each inline reply names the tests and what they showed on the previous head.

Blockers

  1. A restored record is now leased to the claiming worker, settled once installed, and returned on a failed install, a lost reply, or a lost worker. Prepare waits while a lease is open. See the reply on participant_workers.py:183.
  2. Each participant kind names a record's identity: the session key for agents and the session ID for resources. De-duplication uses it, and stale-copy reporting and cleanup are an explicit participant contract, implemented for agents. Two /run episodes sent with one cookie reach commit_reply() as two records, so feat(checkpoint): name the episodes an agent's own /run drives in the agent commit reply #4690's run_episode_ids keeps both. See the reply on participant_workers.py:225.
  3. Seeds and closes without a cookie naming their owner are routed to the first worker that saw the agent_session_id (fix(server): route a session's requests to the worker that holds it #3905), and an environment server's typed resources seed and close are routed the same way by resources_session_id. A restore claims the agent sessions it installs. See the reply on session_routing.py:209.
  4. A caller that stops waiting on a worker message now cancels the handler on the other side and waits until it has stopped, for every message kind. See the reply on workers.py:143.

Fix before merge

  1. Confirmed: a restore is now all or nothing across workers. If one worker's install fails or the worker is lost, the others delete what they installed and reset their routing tables before the restore fails and admission reopens. The contract is in the coordinator's install() docstring. See the reply on participant_workers.py:318.
  2. Retire, delete_restored and forget are bounded by the request's deadline instead of a fixed 30 seconds. See the reply on workers.py:450.

Minor and suggestions

  1. /status merges every worker's participant status and readiness, and the end-to-end retire test checks retired_sessions with two workers per server. See the reply on base_resources_server.py:346.
  2. Stale copies are deleted in one batch per worker, on all workers at once, a bounded number at a time. See the reply on participant_workers.py:241.
  3. Agent sessions record the owner their cookie names. See the reply on agent.py:277.

Performance

  1. Exports travel in chunks of about 2 MiB, encoded and checked off the event loop as they arrive, and large frames are decoded in a thread. In a synthetic 185 MB export, the longest event-loop stall fell from about 760 ms to between 75 and 130 ms. See the reply on workers.py:89.

@asynccontextmanager
async def legacy_run(self, session_key: str, episode_id: EpisodeId) -> AsyncIterator[LegacyRun]:
"""Hold one legacy ``/run`` episode and its turn-loop session."""
await self._claim(session_key, episode_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Roll back the claim when open_session() fails

_claim() settles the restored record before open_session() runs. If open_session() rejects the request, execution never enters the try/finally, so the claimed session remains registered, is exported at every later checkpoint, and every retry fails the same way. Please include open_session() in rollback protection and remove/return the claimed session on every failed-open path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. _claim now opens the session inside the claim block, after the agent's restore hook ran.
If open_session() refuses, for example because the /run is of another attempt than the one the record continues, the session is removed, the state the hook installed is released through retire_agent_session, and the record goes back to the coordinator, so the attempt it continues can claim it.
A failed hook already gave the record back; that path is unchanged.

test_a_restored_run_whose_session_cannot_open_goes_back_and_leaves_nothing_on_the_worker runs a /run of attempt 2 against a record that continues attempt 1, then the attempt-1 /run on the other worker.
On the previous head the refused session stayed registered on the first worker and the record was settled as installed, so the record never came back (the wait for it timed out).
Now the first worker keeps nothing, its hook state is released, and the second worker continues from the saved boundary.

Restored agent sessions installed at restore time take the same care with their routing claims: a worker rewrites a restored session's claim only after its install succeeded, and a worker whose install or claim rewrite fails deletes what it installed and releases the claims it rewrote.

async def open_admission(self) -> None:
self.accepting = True
self.request = None
self.lost_worker = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Do not clear restored-worker loss before validating reopen

A worker lost while admission is closed sets lost_worker, but this clears it before open is broadcast and suppresses broadcast failures. If that worker held restored sessions and dies after install but before resume, resume succeeds and silently loses their state. Keep the loss/generation until every expected restored worker has acknowledged reopen, or fail recovery so the affected episodes can restart.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The coordinator now remembers which workers the last restore installed sessions on, until the resume that reopens them.
If one of them is lost before that resume, or does not acknowledge reopening, the resume fails with the new checkpoint_worker_lost error and admission stays closed, and every later resume fails the same way.
Those sessions' state is gone, so the restore has to be repeated on a restarted server; the error says so.
A worker lost meanwhile that held no restored session is handled as before, since it held nothing a checkpoint needs.

test_a_worker_lost_with_restored_sessions_before_the_resume_fails_the_resume restores two agent sessions onto two workers, closes the connection of one holder, and resumes twice.
On the previous head the first resume succeeded (DID NOT RAISE); now both fail with checkpoint_worker_lost.

With the routing redesign in #3905, the requests of a session whose worker died get a 410 from the router rather than running against empty state, so after a successful resume a later loss is visible to the caller too.


@asynccontextmanager
async def lifespan_with_coordinator(application: FastAPI) -> AsyncIterator[Any]:
await link.connect()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[RISK] Register only after the private routing socket is ready

This registers the worker before entering the wrapped lifespan, while the outer session-routing lifespan starts its Unix socket only after this lifespan yields. A restore can place sessions on this worker during that window, and callers then receive a misleading 410 because its socket is not listening. Start the socket first or add a routing-ready handshake before registration.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by starting the socket first.
A worker of a server that routes sessions now joins its coordinator from inside session routing's lifespan, after its private socket listens and before uvicorn hands it any request.
Session routing enters the lifespans listed in the app state under ROUTED_LIFESPANS_STATE at that point, and the checkpoint link adds itself there. A worker without a private socket, such as an environment server's, still joins before its own startup.

test_a_worker_joins_its_coordinator_only_once_its_private_socket_accepts checks, at the moment each of two real counter server workers registers, that its socket accepts a connection.
On the previous head neither did ([False, False]).

Comment thread nemo_gym/mcp_auto_exposure.py Outdated
# With several workers, the token names the worker that holds the session,
# so a tool call that lands on another worker is routed back here (see nemo_gym.session_routing).
# Single-worker tokens are unchanged.
owner = getattr(request.app.state, "nemo_gym_routing_id", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[RISK] Encode the session's stable owner, not merely the serving worker

After a restore, a request may be served by worker Y while its cookie/session record intentionally preserves pre-restore owner X so the next restore aliases X to the new holder. This token records Y instead; after another restore no alias for Y is guaranteed, so MCP tool calls can route to a dead or wrong worker. Use the cookie/restored owner when present, falling back to this routing ID only for a genuinely new session.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by the routing redesign in #3905 rather than here: cookies and MCP session tokens no longer carry a worker ID at all.
They name only the session ID, and the router forwards by that ID's claim, so there is no stable owner to encode and no alias table to keep in step with it.
A restore installs each session on one live worker, which rewrites the session's claim to name itself; every other worker drops any claim it held on that session.
A cookie or token from before any number of restores therefore reaches whichever worker holds the session now.

test_an_agent_session_cookie_from_before_a_restore_reaches_the_worker_that_holds_the_session_now seeds an agent session, checkpoints, restores onto two fresh workers, and sends the old cookie through each worker: both requests are served by the holder.
test_a_restored_resources_session_serves_a_request_with_its_old_cookie_on_any_worker does the same for counter resources sessions, test_after_a_restore_onto_several_workers_every_worker_refuses_the_replaced_attempt_s_mcp_token for MCP tokens, and test_an_old_cookie_follows_its_session_to_the_worker_a_restore_moved_it_to in test_session_routing.py for the claim rewrite itself.


async def _send_reports(self) -> None:
while self._report_pending and not self.accepting:
self._report_pending = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[RISK] Preserve a failed readiness report for retry

_report_pending is cleared before sending, and the participant's _numbered_report() records _last_report before the send succeeds. On failure this returns with neither a pending report nor a changed value for the polling fallback, so prepare can wait until its deadline even though the worker is ready. Mark the report acknowledged only after a successful call, or restore both pending/last-report state and retry with backoff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed both halves.
A report that does not reach the coordinator stays pending and is sent again, waiting 0.1 s and doubling up to 2 s, until it arrives or admission reopens.
The worker records a report as the last one sent only after the coordinator received it, so the poll that catches readiness changing without a notification also sends it again.

test_a_readiness_report_that_did_not_arrive_is_sent_again loses the report a worker sends when its last request ends.
On the previous head the following prepare stayed preparing until its deadline; now it is prepared.

Comment thread nemo_gym/_checkpoint/workers.py Outdated
# Bounded by the forget request's deadline.
await self.broadcast("check_forget", {"rollout_ids": rollout_ids}, timeout=None)
self.retired.forget(rollout_ids)
await self.broadcast("forget", {"rollout_ids": rollout_ids}, timeout=None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[RISK] Recover from a partially applied forget

The coordinator clears its mark before broadcasting the mutation. If cancellation arrives after one worker has completed forget but before another has, the deployment is split and the unfenced worker can admit a late request for that rollout. A clean retry normally repairs the state, but the unsafe window remains. Re-mark/reconcile on failure, or make the mutation transactional across workers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The coordinator now keeps its own marks until every worker has forgotten.
If the broadcast fails or is cut short, it sends every worker the marks again, as a retire does, so a worker that had already forgotten refuses the attempts again.
That repair runs to the end even if the forget is cancelled again, and a retried forget starts from where every worker refuses.

test_a_forget_cut_short_leaves_every_worker_refusing_the_attempts blocks one worker's forget past the request's deadline.
On the previous head the other worker and the coordinator had forgotten ([0, 1, 0]); now all three still refuse ([1, 1, 1]) until a later forget reaches every worker.

for worker_index, (kind, result) in outcomes.items():
if isinstance(result, ControlError):
# Keep the worker's status and code, such as invalid_phase, so the controller can act on it.
raise _RemoteControlError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NOTE] Report worker loss as worker loss, not coordinator unavailability

When a coordinator-to-worker channel closes, Channel.call() produces checkpoint_coordinator_unavailable; this wrapper preserves that 503 code even though the coordinator itself is healthy and a worker was lost. The previous model-worker path returned a lost-worker conflict. Map this direction of channel failure to a dedicated lost-worker error so callers get an accurate, actionable response.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A connection to a worker that closes or times out on the coordinator's side is now WorkerLostError, with code checkpoint_worker_lost, status 409, and the worker's index in the detail.
An error a worker itself reports, including its own loss of the coordinator, keeps its status and code as before.

test_a_worker_lost_in_the_middle_of_a_message_is_reported_as_a_lost_worker closes a worker's connection while it handles delete_restored.
On the previous head the error was checkpoint_coordinator_unavailable.

# The participant and this worker share one set of retired attempts, which the coordinator keeps current.
participant.retired = self.retired
participant.on_change = self.changed
if hasattr(participant, "claim_restored"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NOTE] Make participant capabilities explicit

claim_restored (and owner below) is wired only when an attribute happens to exist. A typo, rename, or incomplete participant then silently disables restore/routing behavior. Please make these optional hooks explicit on the participant contract or use typed capability protocols so a missing implementation fails visibly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. claim_restored, on_session_freed (new, below) and fence_retired are now part of CheckpointParticipant itself, with defaults of None and a no-op, and the worker link sets the first two on every participant without checking for them.
The owner hook is gone with the worker IDs in cookies.
The participants that need claim_restored (environment, and agent for its own /run episodes) use the base attribute, so a rename now fails type checks and tests rather than silently disabling the claim.
on_session_freed tells session routing when the resources participant frees a session outside a request, such as a retire or a commit that drops a restored episode, so that session's claim ends too.

test_every_participant_takes_the_hooks_a_worker_link_sets builds a link for a resources, an environment and an agent participant.
On the previous head the resources participant had no claim_restored (AttributeError).


with pytest.raises(ValueError, match="num_workers=1"):
agent.setup_webserver()
app = agent.setup_webserver()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NOTE] Stop the coordinator created by this test

setup_webserver() starts WorkerCoordinator on a daemon thread with its own event loop. Unlinking the socket path does not stop that thread, so the test leaves a live coordinator behind. Add a coordinator shutdown API/fixture cleanup and await or join it here; monkeypatch already handles restoring the environment variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. WorkerCoordinator.stop() closes the coordinator's server, stops its event loop, joins its thread and removes its socket.
The test now calls it in a finally, and checks that the thread has ended and the socket is gone; the environment variable is left to monkeypatch as you say.
No other test starts a coordinator in the background: the others run their workers with the worker flag set, so they link to a coordinator the test serves and closes itself.

def _client(self, owner: str) -> ClientSession:
client = self.clients.get(owner)
if client is None:
client = self.clients[owner] = ClientSession(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NOTE] Document why this client is intentionally not shared

This per-owner ClientSession is the correct exception to the usual shared-client rule because it requires a distinct UnixConnector for each worker socket. A short comment stating that constraint would prevent a future refactor from incorrectly replacing it with the application's shared HTTP client.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: each forwarding client holds a UnixConnector bound to one worker's private socket, so the application's shared HTTP client cannot replace it.
That client lives in #3905's session_routing.py, which this PR carries unchanged, so the comment went into #3905, next to the clients pools, and this PR carries it from there:
# Not Gym's shared aiohttp client: each pool needs a UnixConnector bound to its claimant's socket.

@ananthsub

Copy link
Copy Markdown
Contributor Author

Thanks for the second pass. Every item is addressed in the rebuilt commit of this PR, and each inline reply names its tests and what they showed on the previous head.

#3905 now routes by one key, and this PR follows it. Cookies and MCP session tokens no longer carry a worker ID: the router finds a request's session ID and forwards it to the worker that claims that ID. That removed three things from this PR:

  • The worker stamps. Resources and agent session records no longer carry an owner, agent records no longer carry the cookie's session_id, and nothing reads the worker ID out of a cookie.
  • The alias table. A restore installs each session on one live worker, which rewrites the session's claim to name itself, with one atomic rename. Every other worker drops any claim it held on that session. A cookie or token from before any number of restores names only the session ID, so it reaches the session's current holder.
  • The placement table. Sessions checkpointed by a single-process server are restored the same way, claimed by their session ID. A typed resources session's session ID is its resources_session_id, so restored typed sessions are now claimed by it too.

The manifest's session_owners and placed_sessions are gone with them. Checkpoints written by the previous version of this PR do not load, because their records carry fields the record models no longer accept; checkpoints from main load as before.

A restored resources session is open on its holder, as after a seed, so its claim lasts until its close or verify. When the participant frees a session outside a request, such as by a retire or a commit that drops a restored episode, the session's claim ends a window later as well.

The inline items

  • Blocker, open_session() after a claim: the session is opened inside the claim, and a refusal releases what was installed and gives the record back.
  • Blocker, a restored worker lost before reopening: the resume fails with checkpoint_worker_lost, and keeps failing, instead of reopening without that worker's restored sessions.
  • Registration before the private socket: a worker of a server that routes sessions joins its coordinator only once its private socket listens.
  • The stable owner: resolved by the one-key routing above.
  • A failed readiness report: kept pending and sent again with backoff, and recorded only once the coordinator received it.
  • A forget cut short: the coordinator keeps its marks until every worker forgot, and marks every worker again if the forget fails.
  • Worker loss: reported as checkpoint_worker_lost, not as an unavailable coordinator.
  • Participant capabilities: claim_restored, fence_retired and the new on_session_freed are part of the participant contract.
  • The test's coordinator: WorkerCoordinator.stop(), called by the test.
  • The forwarding client: the comment is in fix(server): route a session's requests to the worker that holds it #3905, carried here.

#4690's run_episode_ids is covered with two workers: two of an agent's own /run episodes sent with one cookie jar, on two workers, plus an unclaimed restored one the coordinator holds, are all named in the commit reply.

A new opt-in stress test checkpoints a thousand counter rollouts in flight across 8 resources and 4 agent workers, kills every Gym process, and restores onto 4 and 8 workers, and the reverse. Every rollout finished before the checkpoint or continued with reward 1.0, and no claim or session was left once the close window passed.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

f"{self.kind} worker {worker_index} did not complete install: {error.detail}"
) from error
raise ControlError(f"{self.kind} worker {worker_index} did not complete install: {error}") from error
self._restored_workers = set(placement.values())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Record restored holders before dispatching installs

_restored_workers is populated only after every install call has settled. If a worker acknowledges its install and then disconnects while another worker is still installing, leave() cannot recognize it as a restored holder; this assignment then clears the observed loss, and resume can succeed with that worker's state gone. Please clear stale loss state at the start, populate the intended holders before sending, preserve losses observed during installation, and reset tracking appropriately after rollback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. install() now records the workers it is about to install sessions on, and clears any loss left over from an earlier restore, before it sends the first install.
So leave() recognizes a holder lost at any point after that, including one that already acknowledged its install, and the resume that follows fails with checkpoint_worker_lost.
A loss seen during the installation is kept: a successful install no longer resets the tracking.
A failed restore rolls back on every worker and then clears both, since nothing it installed is left to lose.

test_a_holder_lost_after_its_install_while_another_worker_still_installs_fails_the_resume holds one worker's install, lets the other worker acknowledge its own, and then disconnects that worker.
On the previous head the restore succeeded and the resume did too (DID NOT RAISE), with the lost worker's session gone.

lost = failed & self._restored_workers
if lost:
self._restored_lost |= lost
self.accepting = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Roll back a partially reopened resume

If a restored holder fails during reopen_workers(), reachable workers have already reopened, while this branch closes admission only in the coordinator. Those workers continue serving even though resume failed. The checkpoint request was also cleared, so a replacement worker joining the closed coordinator validates None as its close request. Preserve the request until reopen succeeds and re-close or roll back workers that already acknowledged open.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. When a worker holding restored sessions does not acknowledge open, open_admission() now closes every worker again under the checkpoint request they were closed for, then raises checkpoint_worker_lost.
That puts the coordinator back in its closed state with the real request, so a replacement worker that joins afterwards closes under that request instead of validating None.
Later resumes keep failing, as before, until the server restarts and the restore is repeated.
A worker that cannot be closed again is logged; it is lost, or the next prepare finds it.

test_a_resume_whose_restored_holder_is_lost_while_reopening_closes_the_other_workers_again makes the holder die while it reopens.
On the previous head the other worker kept serving after the failed resume, and the replacement worker's join raised a ValidationError on None as its CheckpointRequest.

Comment thread nemo_gym/_checkpoint/agent.py Outdated
self.open_session(session_key, episode_id)
except BaseException:
if installed:
await self.hooks.retire_agent_session(session_key)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Do not return the claim when rollback teardown fails

If retire_agent_session() raises or is cancelled here, _remove() and the restored-episode pop are skipped. The claim context gives the record back while this worker still retains the installed session, so a later checkpoint can save both copies. Guarantee local cleanup, but make the record claimable elsewhere only after external teardown succeeds; otherwise retain or fence ownership, or fail recovery.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by keeping ownership of the record until the external teardown succeeds.
The agent's rollback in _claim() now does only the local cleanup, _remove() and popping the restored episode, in its except with no await, so a cancellation cannot skip it.
The teardown of the agent's copy moved into the claim: claim_restored(key, teardown=...).
When the block raises, the worker keeps the coordinator's lease and runs the teardown in the background, retrying with a wait that doubles up to 30 seconds.
Only once the teardown has succeeded does the worker settle the claim, and only then can another worker claim the record.
While the lease is held, prepare waits for it and other claims are told the record is still settling (the next comment), so no checkpoint can contain the record and also find the installed copy.

Why this option: it keeps state bounded and keeps the record.
There is at most one background task per failed claim, and it ends when the teardown succeeds or the worker disconnects.
If the worker is lost or stops, the coordinator returns the record, which is then correct because the copy went with the worker.
A retire of the attempt ends the lease as before, and the teardown still runs to free the copy.
Failing the recovery would strand an episode whose record is fine, and returning the record at once is the bug you found.
The cost is that a teardown that never succeeds blocks prepare, with the attempt named as a blocker, until the controller retires that attempt.

One more case came up while fixing it: with the teardown in the background, a later /run of the same session key on the same worker could open a new session that the late teardown would then free.
A claim of a key whose rolled-back install this worker is still freeing now waits for it, at most MESSAGE_TIMEOUT_SECONDS, and is refused as still settling after that.

test_a_restored_run_whose_teardown_fails_keeps_its_record_until_a_teardown_succeeds runs with a teardown that raises and with a request cancelled during the teardown.
On the previous head both left the session on the worker and returned the record before the teardown had finished.
test_a_run_that_reuses_a_key_whose_rolled_back_install_is_still_being_freed_waits_for_it fails on the previous head because its teardown ran inside the request and never returned.

if not self.accepting:
raise AdmissionClosedError(f"{self.kind} admission is closed for a checkpoint")
key = body["key"]
record = self.restored.pop(key, None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Correctness] Treat a leased record as still settling, not absent

A failed claimant returns its record through asynchronous settle_claim, while a new claim is handled immediately. A retry can reach this line while the key is still leased, receive record=None, start fresh, and then have the old record returned. If the key is leased, return a retryable “claim still settling” result instead of treating it as no restored record.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The coordinator now refuses a claim of a leased key with ClaimSettlingError (checkpoint_claim_settling, status 503), instead of answering that there is no restored record.
The worker sends the claim again after 50 ms, doubling the wait up to 1 second, for as long as its caller waits and at most MESSAGE_TIMEOUT_SECONDS; an environment server's claim runs inside the episode's deadline.
After that the retryable error reaches the caller.

Each claim attempt also names its lease with a fresh ID, and settle_claim ends a lease only when the ID matches.
A claim whose reply was lost gives back a lease it may not have got. Now that a lease is kept through a teardown, that settlement could otherwise end a lease the same worker holds for an earlier failed claim still being torn down.

test_a_claim_that_arrives_while_a_failed_claim_settles_waits_and_continues_from_the_record holds the failed claim's settlement while the other worker's /run claims.
On the previous head that /run returned at once and started over without the restored boundary; now it waits and continues from it.

# The participant freed the session outside a request, which the router would otherwise never see end.
lifetimes = self._lifetimes()
if lifetimes is not None:
lifetimes.ended(session_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Correctness] Keep retired-session refusal routable until forget

After retire frees a resources session, this ends its routing lifetime, so its claim is released after the close window. A later cookie-only request can land on another worker, which has no session-ID tombstone and handles it as unclaimed instead of retired. Retain the routing claim until forget, or distribute the retired session-ID tombstone to every worker.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by keeping the claim until forget, as you suggested.
When a retire frees a fenced resources session, the participant now calls on_session_freed(session_id, until_forget=True).
The worker then holds the session's claim with ResourcesSessionLifetimes.retain(), whatever later requests for the session do, including the harness's cleanup close.
The forget that removes the session's refusal calls on_session_freed(session_id) again, which ends the session, and the sweeper releases its claim one close window later.
So the claim has the same end of life as the refusal it routes to.

A forget cut short works as round 25 arranged: the coordinator marks every worker again, and a retried forget finishes on the rest.
A worker that had already forgotten has also ended the claim, so after its window a cookie-only request is served like any request after a forget, as on one worker.

test_a_retired_session_s_cookie_is_refused_on_every_worker_until_forget uses two counter server workers with a short close window.
After the retire and well past the window, a cookie-only request through the other worker is forwarded to the holder and refused with stale_attempt, and after the forget the claim is gone and no state is left on either worker.
On the previous head the claim was already gone after the retire, so the request was served on the other worker, which created a new counter for the retired session.

Comment thread nemo_gym/session_routing.py Outdated
return
try:
os.unlink(self._path(session_id))
os.unlink(path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Correctness] Serialize release with restore rewrite

This ownership check and unlink are a TOCTOU pair. Another worker can rewrite() the claim after the check but before unlink(), allowing the former owner to delete the restored holder's new claim. Another check remains racy; use cross-process serialization or ownership-specific claim files.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed with serialization across processes.
SessionClaims.release() now checks the claimant and unlinks while it holds an fcntl.flock on a lock file beside the claims directory, and rewrite() replaces the claim file under the same lock.
The lock is held only for those file operations, never across an await.
claim() takes no lock: its hard-link create fails whenever any claim exists, so it cannot replace a rewritten claim.

Cost, over 20,000 releases on one workstation: release went from 15.2 to 15.5 µs at p50 to 19.8 to 20.4 µs.
Only a release that actually removes this worker's claim takes the lock; the check after each request returns before it.

This changes nemo_gym/session_routing.py, which came with #3905.
It belongs in this PR because the race only exists with this PR's rewrite().

test_a_release_cannot_delete_a_claim_a_restore_rewrites_while_it_releases runs the other worker's rewrite() in a thread right after the release's check.
On the previous head the release then deleted the rewritten claim, leaving none.

Comment thread nemo_gym/_checkpoint/workers.py Outdated
self.worker_label,
exc_info=True,
)
self._channel.close()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Cleanup] Stop worker-link background tasks on clean disconnect

The channel remains non-null after this close, so report and claim-settlement loops continue retrying against it; a readiness poll can also remain alive while the worker is locally closed. Clear the channel and cancel and await the report, poll, and settlement tasks during disconnect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. disconnect() now clears the link's channel first, tells the coordinator it is stopping over its own reference, closes the connection and waits for the reader.
It then cancels and awaits background_tasks(): the report task, and on a participant link also the readiness poll and every claim settlement, including any teardown retry from the claim fix above.
With the channel cleared, a task that wakes before its cancellation finds no connection and stops.

test_a_clean_disconnect_stops_the_worker_link_s_background_tasks disconnects a worker while a checkpoint is open, its report keeps failing, and a claim settlement keeps retrying.
On the previous head all three tasks were still running after the disconnect.

# Converted and encoded once, off the event loop, into messages of bounded size,
# which the coordinator checks as they arrive.
# The exported state does not change until the checkpoint resumes.
chunks = await asyncio.to_thread(encode_records, records, chunk_bytes=EXPORT_CHUNK_BYTES)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Perf, non-blocking] Stream export chunks as they are encoded

Although encoding is off the event loop, encode_records() builds the complete serialized export before this loop sends the first chunk. A large export therefore exists as both record objects and a full encoded checkpoint. A bounded producer/consumer pipeline would keep only a few chunks in memory and overlap encoding, IPC, and validation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. encode_records() is now a generator, and the worker encodes one message at a time in a thread.
It starts encoding the next message before it sends the current one, and the coordinator checks each message before it answers, so encoding, sending and checking overlap.
The worker holds at most two encoded messages instead of the whole export.

Measured with round 24's synthetic export (1,000 agent sessions, 177 MB of records, two workers and the coordinator in one process), five alternating runs each:

  • peak resident memory above the level before the commit: 783 to 788 MB before, 635 to 649 MB now;
  • commit time: median 4.44 s before (2.64 to 6.47 s), median 3.23 s now (3.12 to 6.88 s), on a shared machine whose load varied a lot;
  • longest event-loop stall: median 114 ms before, 60 ms now.
    In separate processes the saving lands on the worker, which no longer builds a full encoded copy of its export.

test_an_export_sends_its_first_chunk_before_it_has_encoded_every_record counts the records encoded when the first message goes out.
On the previous head all 20 were; now at most a few are.

@ananthsub

Copy link
Copy Markdown
Contributor Author

@macandro96 thanks for the third pass. All eight items are addressed in the rebuilt commit, which is now on the current main. Each inline reply names its test and what that test showed on the previous head.

  • Restored holders recorded before installs (blocker). install() records the intended holders, and clears any earlier loss, before it sends the first install. A holder lost after it acknowledged its install now fails the resume with checkpoint_worker_lost, and a failed restore resets the tracking.
  • A partly reopened resume (blocker). If a restored holder does not reopen, every worker is closed again under the checkpoint's request before the resume fails. A worker that joins afterwards closes under that request too.
  • Rollback teardown (blocker). Local cleanup always happens, and the agent's installed copy is freed by a teardown the claim runs and retries in the background. The record goes back only after that teardown succeeds; until then the worker keeps the lease. A later /run of the same key on that worker waits for the teardown, so the teardown cannot free the new session.
  • A leased record is still settling. A claim of a leased key gets the retryable checkpoint_claim_settling (503), and the worker claims again with a growing wait, bounded by its caller's deadline. Leases now carry an ID, so only the claim that holds a lease settles it.
  • Retired sessions stay routable until forget. A resources session freed by a retire keeps its claim until the forget that ends its refusal, and its claim ends one close window after that forget.
  • Release and rewrite. The release's check and unlink, and every rewrite, hold one flock on a lock file beside the claims directory. A release that removes a claim costs about 4.5 µs more at p50. This touches nemo_gym/session_routing.py, because the race only exists with this PR's rewrite().
  • Clean disconnect. The channel is cleared first, and the report, readiness poll and claim-settlement tasks are cancelled and awaited.
  • Streaming export. Each message is encoded in a thread while the one before it is sent and checked, so a worker holds at most two encoded messages. On round 24's synthetic 177 MB export, peak memory fell from about 785 MB to about 640 MB above the starting level, and the median commit time went from 4.4 s to 3.2 s on a busy machine.

for worker_index, placed in installs.items()
}
outcomes = await self.call_each_settled(messages, timeout=None, bulk=True)
failed = {index: result for index, (_, result) in outcomes.items() if isinstance(result, BaseException)}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Roll back a restore whose holder was lost during installation

A holder can acknowledge its install and then disconnect while another worker is still installing. leave() records that loss, but if every install RPC eventually reports success, this path leaves _restored_lost set. When the controller has already timed out the restore, its failed-restore reopen and every later resume then raises WorkerLostError, leaving the server closed until restart. Treat a holder recorded lost during installation as an install failure, uninstall the surviving workers, and clear the restore-tracking sets before reopening; simply clearing the flag and exposing partial state would not be safe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. install() now treats a holder that leave() recorded as lost during the installation as an install failure, even when every install message reported success.
The workers still connected delete what they installed and release the claims they rewrote, the restore-tracking sets are cleared, and the restore fails with checkpoint_worker_lost. The controller's failed-restore reopen then finds nothing left to lose, so admission reopens and later resumes succeed.

One window remained: a restore that ran past its deadline, whose install then finished on every worker, and whose holder was lost after that but before the failed restore's reopen ran. install() has already returned there, so the control plane now calls a new participant hook, discard_failed_restore(), before it reopens after a failed restore. The coordinated participant rolls the restore back the same way when a holder was lost, and drops the records it holds. Without such a loss it keeps the installed state, which the controller's retire frees, as on one process. The default hook does nothing, so single-process participants are unchanged.

Tests, each failing on the previous head:

  • test_a_holder_lost_after_its_install_while_another_worker_still_installs_fails_the_restore: the previous head's restore succeeded (DID NOT RAISE).
  • test_a_restore_past_its_deadline_whose_holder_was_lost_while_installing_reopens_admission is your scenario. On the previous head the failed-restore reopen raised WorkerLostError, and the surviving worker never reopened.
  • test_a_holder_lost_after_an_install_that_outlived_its_restore_does_not_keep_admission_closed holds the control lock while the install finishes and the holder is lost. It failed the same way on the previous head.


async def reopen_workers(self) -> set[int]:
"""Reopen admission here and on every worker; return the workers that did not acknowledge it."""
self.accepting = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BLOCKER] Keep admission closed until every worker is ready to reopen

This opens the coordinator before all open calls have succeeded, and each worker's open_local() starts accepting before it sends its acknowledgment. If another restored holder then fails, _close_again() eventually closes the healthy workers, but requests can be admitted and complete during that window. Please use a two-phase reopen or a shared admission gate so workers become externally available only after every required acknowledgment succeeds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed with a two-phase reopen in CoordinatedParticipant.reopen_workers().
In the first phase every worker answers a prepare_open message and stays closed. The coordinator stays closed too, and keeps the checkpoint request, so a worker that joins meanwhile closes under it. A new check_reopen() hook then decides on the workers that did not answer. The server participant raises checkpoint_worker_lost there if a worker holding restored sessions did not answer, or was recorded lost, and no worker has reopened at that point. Only after that does the coordinator open and send open to every worker connected by then. So a worker's open_local() never runs before every required acknowledgment has arrived, and _close_again() is gone.

Each phase is bounded by MESSAGE_TIMEOUT_SECONDS. The first phase is also cancelled by the deadline of the resume it serves, which leaves every worker closed. Once the second phase has started, it runs to completion even if its caller is cancelled. Otherwise some workers would serve while others stay closed. A holder lost during the second phase, after every worker said it was ready, is lost as it would be just after the resume: the other workers serve, and that worker's sessions get a 410.

Tests:

  • test_no_worker_reopens_until_every_worker_holding_restored_sessions_can makes the holder wait for the other worker to start serving before it dies. On the previous head the other worker was already serving at that moment; now it stays closed, and so does a replacement that joins.
  • test_a_holder_lost_after_every_worker_is_ready_to_reopen_is_lost_as_after_the_resume covers a loss in the second phase. On the previous head the resume failed and closed the other worker again.

wait = _SETTLE_RETRY_SECONDS
while teardown is not None:
try:
await teardown()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resilience, non-blocking] Bound a stuck teardown without releasing its lease

If this hook never returns, the settlement task and its lease remain forever. That lease keeps every checkpoint unready, while later /run requests for the same key repeatedly time out in _wait_torn_down(). Consider a bounded, cancellation-safe retry or a policy that fences/restarts the worker after a teardown exceeds its limit. The record must remain unclaimable until teardown is confirmed, so timing out must not simply return the lease.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as a bounded retry that keeps the lease.
ParticipantWorkerLink._tear_down() cancels each teardown attempt after _TEARDOWN_ATTEMPT_SECONDS (60 s), and tries again after a wait that grows to 30 s. A failure or a timeout counts as a failed attempt, and so does a hook that raises CancelledError by itself. Only cancelling the settlement task stops it, as a disconnect does.
The worker never gives the lease back while the teardown has not succeeded. So the record stays unclaimable, and prepare keeps naming the attempt as a blocker, until a teardown succeeds, the controller retires the attempt, or the worker leaves. The worker is not restarted.
A later /run of the same key on that worker waits in _wait_torn_down() for up to MESSAGE_TIMEOUT_SECONDS. Then it fails with the retryable checkpoint_claim_settling (503) instead of hanging. Gym's /run carries no deadline of its own, so that timeout and the caller's cancellation are what bound it. The behavior is documented in the claim() and _tear_down() docstrings and in the module docstring.

This turned up a related gap. A reopen sent the workers only the unleased restored keys, so a record that was leased during a checkpoint, and that went back after its teardown succeeded, could no longer be claimed on any other worker. The replacement then started over without its restored state. Leased keys now stay in the claimable set that a reopen or a join hands out.

test_a_stuck_teardown_keeps_its_record_blocks_prepare_by_name_and_a_run_of_its_key_gives_up keeps the teardown hanging. It checks that the teardown is cancelled and retried, that prepare reports r-a1 as a blocker, that a /run of the key gets ClaimSettlingError, and that the record cannot be claimed until the teardown succeeds, after which the other worker continues from it. On the previous head the first attempt never ended.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A follow-up on the end of life of a stuck teardown. The teardown used to keep retrying even after the controller's retire had dropped the record's lease, which left one task per stuck teardown running for the life of the worker.

While the lease is held, nothing changes: the worker keeps retrying, the record cannot be claimed, and prepare names the attempt. After each failed attempt, the worker now asks the coordinator whether it still holds the lease. Once the lease is gone, for example after a retire, the teardown gets _TEARDOWN_ATTEMPTS_WITHOUT_LEASE (3) more attempts. If none of them succeeds, the worker logs an error that names the session and the attempt and says that the external resource may be orphaned, then drops the task and the session's entry. A teardown that succeeds within those attempts settles its claim as before.

A cancelled attempt now gets _TEARDOWN_CANCEL_GRACE_SECONDS (10 s) to stop. If the hook ignores the cancellation, the worker logs it and stops waiting. While that hook call is still running, the worker does not start another one: the next retry waits for the same call again. When the attempts run out, the worker drops its reference to the call.

Tests:

  • test_a_teardown_whose_lease_a_retire_ended_stops_within_its_cap[always_fails] makes every teardown attempt fail, then retires the attempt. It checks that the teardown stops within the cap, logs the error, sends no settlement, and leaves no task or entry behind. On the previous head it kept retrying.
  • test_a_teardown_that_ignores_cancellation_is_no_longer_awaited_once_its_lease_is_gone uses a hook that swallows its cancellation. On the previous head the worker waited for that hook forever. Now the worker stops waiting once its attempts run out, and it never starts a second copy of the call.
  • [succeeds_after_the_lease_ends] checks that a teardown that succeeds after the retire still settles its claim. It passes on the previous head too, which retried until success. It guards against the cap cutting a success short.

lease = self.leases.get(body["key"])
# Only the claim that holds the lease settles it.
# A claim whose reply was lost settles a lease it never got, which changes nothing.
if lease is not None and (lease.worker_index, lease.lease_id) == (worker_index, body["lease"]):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Test] Exercise a stale settlement from an older lease

This lease-ID comparison is important but has no targeted regression test. Please cover the ordering where settlement A is processed but its reply is lost, another worker obtains lease B, and the retry of settlement A arrives afterward. The test should verify that stale settlement A cannot settle or return lease B.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test_a_settlement_retried_after_another_claim_leased_the_record_changes_nothing, parametrized on whether another worker or the same worker takes the second lease.
The first claim fails, and its settlement reaches the coordinator but its reply is lost. The worker's retry is held until the second claim holds lease B. The test then checks that the retry leaves lease B in place and does not return the record, and that the second claim's own settlement ends lease B with the record installed.
It passes on the previous head too, since the lease-ID check was already there. To show that it guards the check: comparing only the worker index fails the same-worker case, and dropping the comparison fails both.


run = asyncio.create_task(run_of_a_later_attempt())
await tearing_down.wait()
if failure == "is_cancelled":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Test] This branch does not cancel the teardown

By the time tearing_down is set, teardown is running in an independent _settling task and the foreground run task is already unwinding or complete. Cancelling run therefore does not cancel the teardown, and this variant makes no assertion that distinguishes cancellation. Please explicitly cancel/disconnect the tracked teardown task, or make the hook raise CancelledError, and assert the intended lease and cleanup behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that variant only cancelled the /run, after the teardown had moved to its own task. It is replaced by two variants that reach the teardown itself:

  • the hook raises CancelledError on its own. On the previous head this ended the settlement task, so the record was never returned while the worker stayed connected (the test timed out). Now it counts as a failed attempt and is retried;
  • the hook hangs, and the attempt timeout cancels it. The test checks that the hook saw the cancellation and that the next attempt ran. On the previous head the first attempt never ended.

In both, the record cannot be claimed and the session is gone from the worker until a later attempt succeeds. The other worker then continues from the record. Explicitly cancelling the tracked task is covered by test_a_clean_disconnect_waits_a_bounded_grace_for_a_teardown_in_flight[outlasts_the_grace], which checks that the hook is cancelled and that the coordinator takes the record back as the worker leaves.

await asyncio.gather(self._reader_task, return_exceptions=True)
tasks = self.background_tasks()
for task in tasks:
task.cancel()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resilience, optional] Give external teardown a graceful-shutdown window

For ParticipantWorkerLink, these background tasks include a claim's external teardown. Cancelling them immediately on a clean disconnect can leave a sandbox or other external resource alive, while leave() returns the record for installation elsewhere. Checkpoint ownership remains consistent, but the external resource may be orphaned or duplicated. If graceful shutdown is expected to clean such resources, await teardown for a bounded grace period before cancelling, or document and provide recovery for this tradeoff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. ParticipantWorkerLink.disconnect() now waits up to _TEARDOWN_GRACE_SECONDS (30 s, a documented constant) for the teardowns still running before the base disconnect cancels the background tasks.
The wait comes before the worker announces it is stopping and before the connection closes. So a teardown that finishes within the grace sends its settlement over the open connection, and the record goes back with no lease left for leave(). A teardown still running when the grace ends is cancelled, and leave() then returns the record as for any departing worker. The docstring states the tradeoff: the checkpoint stays consistent, but what the install built outside the participant, such as a sandbox, may outlive the worker alongside the copy that the next claim installs elsewhere.

test_a_clean_disconnect_waits_a_bounded_grace_for_a_teardown_in_flight records the leases the coordinator holds when the worker leaves. In the variant that finishes within the grace, the teardown completes and no lease is left. In the variant that outlasts it, the hook is cancelled at the grace, and the lease is still there for leave() to return. On the previous head the teardown was cancelled at once in both.

@ananthsub

ananthsub commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

@macandro96 thanks for the thorough review.

  • A restore whose holder was lost during installation (blocker). install() treats a holder that leave() recorded as lost as an install failure, even when every install message succeeded. The surviving workers uninstall, the tracking sets are cleared, and the restore fails with checkpoint_worker_lost, so the failed-restore reopen and later resumes succeed. A holder lost after an install that outlived its restore is rolled back the same way through a new discard_failed_restore() hook, which the control plane calls before it reopens after a failed restore.
  • Admission stays closed until every worker is ready (blocker). Reopening has two phases. Every worker first answers prepare_open and stays closed, and so does the coordinator. A worker holding restored sessions that does not answer, or was lost, fails the resume before any worker opens. Only then do the coordinator and the workers open. Once started, the second phase runs to completion even if its caller is cancelled, and a holder lost during it is lost as after the resume.
  • A stuck teardown is bounded and keeps its lease. Each teardown attempt is cancelled after 60 seconds and retried with a growing wait, and a hook that raises CancelledError by itself counts as a failed attempt. The lease, the unclaimable record and prepare's named blocker stay until a teardown succeeds, the attempt is retired, or the worker leaves. A /run of the same key on that worker gets the retryable checkpoint_claim_settling after 30 seconds. Leased keys also stay claimable on every worker after a reopen, so a record that goes back later can still be claimed.
  • Stale settlement test. A settlement that is retried after another claim, on another worker or the same one, has taken lease B neither ends lease B nor returns the record.
  • Cancellation test. The variant now cancels the teardown itself: the hook raises CancelledError, or hangs until the attempt timeout cancels it.
  • Graceful shutdown. A clean disconnect waits up to 30 seconds for teardowns still running, so one that finishes settles its claim before the worker leaves. One that outlasts the grace is cancelled, and leave() returns its record, which the disconnect() docstring documents as a possible orphaned external resource.

@macandro96 macandro96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1, pre-existing prerequisite] Finish failed-restore recovery before admitting another checkpoint
The missing _reopen_pending guard already exists on main; this PR did not introduce it. However, the new multi-worker lost-holder handling adds a permanent-closure failure mode: a new prepare can overtake pending failed-restore recovery, after which a lost restored-state holder can make every subsequent resume fail while surviving workers remain closed.
Please either fix this here or land a prerequisite fix before merging this PR. While _reopen_pending is true, a new checkpoint must not begin; resume should finish the failed-restore recovery first. As defense in depth, _reopen() should discard pending failed-restore state before reopening admission.

{worker_index: ("uninstall", {"scope": scope}) for worker_index in workers}, timeout=None
)
for worker_index, (_, result) in outcomes.items():
if isinstance(result, BaseException):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep admission closed when rollback does not complete

_uninstall() only logs worker cleanup failures. install() then clears the restore tracking and failed-restore recovery can reopen admission, even though a surviving worker may still hold state installed by the failed restore. I reproduced this with one worker failing its install and another worker failing its rollback hook: the restore returned an error, but the latter worker reopened with the restored session still present.

Please propagate and retain rollback failures instead of calling _end_restore() unconditionally. Admission should remain closed, and cleanup should be retried, until every installed copy is removed or the affected worker/server is replaced.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A failed rollback is now kept rather than only logged, and admission stays closed until it is repeated successfully.
Before the coordinator asks the workers to delete what a failed restore installed, it records each of them as still holding that state, with the restore's scope. It clears a worker only when that worker answers that it deleted everything. So a worker whose rollback hook fails stays recorded, and so does every worker if the rollback itself is cancelled.

Every reopen now starts by repeating the rollback on the recorded workers: CoordinatedServerParticipant.open_admission() calls _finish_rollback() before the two-phase reopen. It retries with a wait that grows from 0.1 s to 2 s, for at most 30 seconds or the caller's deadline, whichever comes first. If a rollback is still incomplete then, it raises the new retryable checkpoint_rollback_incomplete (503), and no worker has reopened. That covers the failed restore's own reopen, a resume that retries it, and a lease expiry. discard_failed_restore() repeats it too, so a restore retried after the failed one does not install anything until the earlier rollback has finished.

The record ends when the rollback succeeds on that worker, or when the worker is lost, since what it held went with its process. A replacement worker joins with nothing installed. If neither happens, admission stays closed and each resume tries again.

Tests, each failing on the previous head:

  • test_a_rollback_that_fails_on_a_worker_keeps_every_worker_closed_until_it_is_repeated is your scenario: one worker fails its install and the other fails its rollback hook. The test checks that neither worker serves, the session and its claim are still on the second worker, and the resume fails with checkpoint_rollback_incomplete. Once the hook works, the next resume rolls back and reopens every worker, with no session or claim left. On the previous head the resume returned (DID NOT RAISE), and the second worker was already serving with the restored session.
  • test_a_worker_lost_while_its_rollback_is_retried_lets_admission_reopen: the worker that cannot roll back is lost, and the next resume reopens the other one. On the previous head the other worker was already serving before the loss.

An adversarial pass, outside the PR, also covered rollbacks failing on two of three workers and recovering one at a time, a worker lost in the middle of a repeated rollback, and a restore retried while a rollback is incomplete, which fails until the rollback succeeds.

key,
episode,
)
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the key fenced while an abandoned teardown is still running

When a teardown ignores cancellation, this returns False after the lease ends even though the child attempt is still running. _settle() then removes the key from _tearing_down, so another /run can open the same key. The abandoned teardown can subsequently finish and free the new session because hooks such as retire_agent_session() operate only on session_key.

Please retain a per-key fence until the child task actually terminates. If it never terminates, the key must remain unavailable or the worker must be replaced.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. When a teardown is given up while its call is still running, because the call ignored its cancellation, the worker keeps the key fenced until that call actually ends.
The fence is a per-key entry, _fenced, that holds the call's task, and the task's done callback removes it. claim() waits for both a teardown still running and a fence on the key, for at most MESSAGE_TIMEOUT_SECONDS in total, as before. After that it raises the retryable checkpoint_claim_settling. So a late teardown can no longer free a session that a later /run of the same key opened on that worker.

When a teardown is given up while its call still runs, the worker logs an error that names the key and says it cannot be claimed on this worker until that call ends. A call that never ends keeps the key unavailable on that worker for the worker's life. That is the chosen tradeoff over restarting the worker, and _fence() documents it. A disconnect that cancels a teardown fences the key the same way, but does not wait for the call. The cap from the last round is unchanged.

test_a_key_whose_given_up_teardown_still_runs_cannot_be_claimed_on_its_worker_until_that_call_ends gives up a teardown whose call swallows its cancellation. A /run of the key on that worker then gets checkpoint_claim_settling. Once the call ends, a /run of the key opens normally, and the fence is gone. On the previous head the /run opened a new session while the old call was still running (DID NOT RAISE).

await asyncio.shield(remark)
raise
# Kept here until every worker forgot, so a worker that joins meanwhile still refuses the attempts.
self.retired.forget(rollout_ids)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Make forget linearizable with worker joins and older repairs

The forget broadcast uses a snapshot of workers. A worker that joins after that snapshot can inherit the current coordinator mark but miss the forget; this line then clears the coordinator mark, leaving only the new worker refusing the rollout. Separately, if a failed forget is cancelled again while its shielded repair is pending, a retry can complete successfully before the older repair marks the workers again. I reproduced both coordinator/worker divergences.

Please version mark/forget state, or otherwise serialize membership changes and overlapping repair tasks, so a successful forget cannot be missed by a joining worker or overwritten by an older repair.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by ordering forget against both, rather than by versioning the marks.

Joins: the coordinator sends the forget to the workers it knows, then again to any worker that joined since, until a round finds no new worker. It then drops its own marks with no await in between. A worker that joins before that point adopts the marks and is sent the forget, and one that joins after it finds no mark. A new worker holds nothing of those rollouts, so it can always forget them, and the loop ends once workers stop joining.

Repairs: a forget first waits for every repair still running from an earlier forget that was cut short. Each repair is bounded by MESSAGE_TIMEOUT_SECONDS per worker, and the wait is bounded by the forget's own deadline. Forgets already run one at a time under the control lock, so with this no older repair can land after a later forget. Repairs and retire marks only ever raise a mark, so their order among themselves does not matter.

Versions would have to travel to every worker and be checked there, for the same guarantee.

Tests, each failing on the previous head:

  • test_a_worker_that_joins_while_a_forget_runs_forgets_too: a worker joins while another worker is still forgetting. On the previous head it kept refusing the rollout after the coordinator forgot it ([0, 0, 1, 0]).
  • test_a_forget_retried_while_the_repair_of_one_cut_short_runs_waits_for_that_repair reproduces your second case. The repair is held on the worker that had already forgotten, and the forget is cancelled again. On the previous head the retry finished first, and the repair then left that worker refusing ([1, 0, 0]).

The adversarial pass also covered a worker that joins while a repair runs, and a forget of another rollout while a repair runs. That forget waits for the repair, and the repaired rollout stays refused everywhere.

claims.disown(session_id)
lifetimes = self._lifetimes()
for session_id in sorted(installed):
claims.rewrite(session_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Batch restored-session routing rewrites

This performs a synchronous temporary-file write, lock acquisition, and atomic replacement for every restored session on the event loop. In a local 12,000-session microbenchmark, _route_restored() took about 3.1 seconds, during which this worker could not promptly serve other asynchronous work.

Please consider a batched SessionClaims rewrite operation that holds the shared lock once for the batch, or move the complete batch off the event loop with carefully managed ownership of _mine.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, both ways. A new SessionClaims.rewrite_all() writes the temporary claim files and replaces them in a thread. It holds the shared lock once per 256 renames, so a release on another worker waits for at most one batch, not the whole restore. _route_restored() now awaits it.

Ownership of _mine:

  • The thread touches only files. _mine changes only on the event loop, after the thread has ended, and then gains exactly the IDs the thread replaced. If the caller is cancelled meanwhile, it still waits for the thread, updates _mine, and then passes the cancellation on, so the rollback that follows releases every claim the thread wrote.
  • Before the batch starts, _route_restored() marks each installed resources session open. An agent session is already held by its installed record. So holds() is true for every batched ID, and no release on this worker removes its claim, during the batch or after it.
  • A release on this worker of an ID that is not in _mine yet does nothing. A release on another worker still checks, under the same lock, that the file names it.
  • A claim on this worker during the batch either finds the old claim and forwards there, as before the restore, or creates one naming this worker, which the rename keeps.

Measured with _route_restored() alone at 12,000 resources sessions, on this workstation's ext4 /tmp, five runs each:

_route_restored() (s) Longest event-loop stall (ms)
Previous head 0.44 to 0.47 441 to 469
This change 0.41 to 0.43 6.7 to 8.5

This filesystem is faster than the one you measured on, where the same work took about 3.1 s. The stall before the change is the whole call either way, because it ran on the event loop.
The total stays close, since it is file work. What changes is that the event loop keeps running during it.

Tests, each failing on the previous head:

  • test_restoring_many_sessions_rewrites_their_claims_without_holding_up_the_event_loop restores 200 sessions with each rename slowed to 5 ms. It checks that every claim names its holder and that the event loop never stalls for 200 ms. On the previous head the loop stalled for 1.05 s.
  • TestCheckpointRestores::test_a_batched_rewrite_whose_caller_is_cancelled_finishes_before_the_worker_counts_its_claims cancels the caller part way through. No claim counts as this worker's while the thread runs, and every claim does once it ends. The previous head had no batched rewrite.

The adversarial pass also raced a batch against a former claimant releasing the same IDs from a thread, and against claims and releases on the holder's own event loop. Every claim ended up naming the holder.

A failed restore's rollback still releases its claims one at a time on the event loop. That happens only on the failure path, while admission is closed, and is not batched here.

@ananthsub

Copy link
Copy Markdown
Contributor Author

@macandro96 thanks for the fifth pass. All five items, including the review summary's prerequisite on main's control plane, are fixed in the rebuilt commit, which is now on the current main. Each inline reply names its tests and what they showed on the previous head.

  • A rollback that does not complete keeps admission closed (P1). The coordinator records every worker it asks to roll back a failed restore as still holding that restore's state, and clears a worker only when it answers that it deleted it. Every reopen first repeats the rollback on the recorded workers, with a growing wait, for at most 30 seconds or the caller's deadline. While one is still incomplete, no worker reopens, and the resume fails with the new retryable checkpoint_rollback_incomplete (503). The record ends when the rollback succeeds there or the worker is lost.
  • A key stays fenced while an abandoned teardown runs (P2). A teardown given up while its call still runs fences its key on that worker until the call ends. A /run of the key there waits as before and then gets checkpoint_claim_settling. The give-up is logged, and a call that never ends keeps the key unavailable on that worker for the worker's life.
  • Forget is ordered against joins and repairs (P2). The forget is sent again to any worker that joined while it ran, until none is new, and the coordinator then drops its marks with no await in between. A forget also waits for any repair of an earlier forget that was cut short, so an older repair can no longer refuse a rollout again after a later forget.
  • Restored claims are rewritten in one batch off the event loop (P3). SessionClaims.rewrite_all() writes the claims in a thread, holding the shared lock once per 256 renames, and changes the worker's own set of claims only on the event loop once the thread has ended. At 12,000 sessions the longest event-loop stall went from 441 to 469 ms to 6.7 to 8.5 ms on this workstation, where the whole call takes about 0.4 s either way.
  • A failed restore's recovery finishes before another checkpoint begins (review summary, P1). This gap is on main, and the fix is in this PR. While a failed restore has left admission closed, the control plane refuses a prepare with the new retryable restore_recovery_pending (503) and changes nothing. The controller already resumes after a failed prepare, and that resume finishes the recovery, so no controller change is needed. Refusing was chosen over finishing the recovery inside the prepare, because that would reopen admission and close it again under the new checkpoint within one call, mixing two operations' failures. As defense in depth, _reopen() runs discard_failed_restore() before it reopens whenever a failed restore's recovery is pending. Two related cases are covered as well. A restore retried while the recovery is pending first rolls back what the failed one left. A restore that succeeds ends the pending recovery, so a holder it installed on that is then lost still fails the resume, as before.
    • The new tests reproduce the overtaking prepare on one process and with two workers, and each fails on the previous head. In the two-worker test, a restore runs past its deadline while its install is held, a prepare is attempted, and the install then finishes on a worker that is lost. The test checks that the other worker reopens with nothing installed and that a later checkpoint prepares and resumes. Another test lets a prepare overtake the recovery anyway and checks that the resume still rolls the failed restore back. On the previous head, that resume failed with checkpoint_worker_lost, and every later one would have failed the same way.

The stack is rebuilt on the current main with no conflicts. #4197's diff is unchanged.

@macandro96 macandro96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly lgtm! just one low priority race that is cheap to fix

Comment thread nemo_gym/session_routing.py Outdated
batch: list[tuple[str, str, str]] = []
for session_id in session_ids[start : start + _REWRITE_BATCH]:
path = self._path(session_id)
temporary = f"{path}.{self.routing_id}.tmp"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] rewrite_all and claim() share one scratch file name, so a request during restore can abort the restore.

This temporary name is f"{path}.{self.routing_id}.tmp", the same name claim() uses (L264) for the same worker and session. Now that _replace_claims runs in a thread, the two can overlap:

  1. The restore thread writes X.<worker>.tmp for a restored session X.
  2. A seed or close naming X in its body reaches this worker. The router calls claim(X) before any admission check, and X is not in _mine until the thread ends. claim() truncates the same file, and its finally unlinks it.
  3. The thread's os.replace raises FileNotFoundError, _route_restored fails, and the whole restore rolls back.

The trigger is narrow: it needs a stale or out-of-band seed or close for a restored session during the install. The impact is a failed restore, not corrupted state, and a retry recovers. As a side effect, claim() can also link a scratch file the thread just truncated, so a reader briefly sees an empty claim.

Suggested fix: give each operation its own scratch file, for example f"{path}.{self.routing_id}.{uuid4().hex}.tmp" in both places, and unlink any leftover in a finally in _replace_claims. The final steps stay atomic and agree on the owner (os.replace vs os.link from the same worker), so nothing else changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 989f513a33, as suggested.
claim() and _replace_claims() each write their own scratch file, <claim path>.<worker>.<random suffix>.tmp, so a claim of a restored session on this worker during a restore's batched rewrite can no longer truncate or remove the rewrite's file.
_replace_claims() also removes any of its scratch files that a batch cut short left behind.
The final steps are unchanged: os.link for a claim and os.replace for a rewrite, both naming the same worker.

The suffix comes from secrets.token_hex rather than uuid4, so the scratch names stay independent of how the router mints session IDs.

Covered by test_a_claim_on_this_worker_during_a_batched_rewrite_of_the_same_id_does_not_fail_the_rewrite.
It claims the session on the same worker at the moment the rewrite takes the claims lock, which is the interleaving you described.
It checks that the rewrite completes, that the claim names this worker, and that no scratch file is left.
On the previous head it fails with FileNotFoundError from the rewrite's os.replace.

macandro96
macandro96 previously approved these changes Oct 23, 2026
…ral workers

Resources, agent, and environment servers may run several uvicorn
workers with checkpointing on. Each session's requests reach the worker
that holds it, and each server is one participant across its workers
through a coordinator in the main process, shared with the policy
model. The worker count may change between checkpoint and restore.

Gym servers listen with a configurable uvicorn_backlog, default 16384.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 04578d21 Deployed Oct 26, 2026 by copy-pr-bot[bot] via release / finalize / notify #7238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants