Skip to content

feat(checkpoint): add the agent session participant and Simple Agent continuation - #3888

Merged
ananthsub merged 1 commit into
mainfrom
ananthsub/partial-ckpt-agent
Oct 15, 2026
Merged

ananthsub merged 1 commit into
mainfrom
ananthsub/partial-ckpt-agent

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

This PR builds on the agent session lifecycle that #3784 added to main. The base server's seed and close also open and close the session in the checkpoint participant, and Simple Agent exports, restores and retires the sessions the base server holds.

Every agent takes part in checkpoints once checkpointing is on.

  • Agents that set checkpoint_sessions_supported = True and implement the session hooks get an AgentSessionParticipant. A session is idle, running, or at a boundary. A running activation parks at its next boundary when a checkpoint closes admission, and one waiting on a model call counts as parked at its last boundary. If that call's reply arrives while the checkpoint is open, the session stays parked at the boundary before the call and holds the reply until resume, so its readiness never regresses between prepare and commit. The model may already have recorded the call, because it delivered the reply before its own prepare. After a crash the restored agent calls again from that boundary, and the token-capture builder treats the two rows like a client retry: it keeps the call the agent continued from. Boundaries are lazy snapshots, so a long turn loop does not serialize its history at every step. Restored sessions are installed under the replacement attempt.
  • The hooks are coroutines, and export takes every session at once: export_agent_sessions(session_keys), restore_agent_sessions(sessions), and retire_agent_session(session_key). An agent whose state lives outside the process, such as a sandbox it runs tools or a harness in, can checkpoint all of it concurrently within the commit.
  • A legacy /run is an episode of its own: seed and the turn loop replay, and verification uses the mode the resources server reported on its seed reply.
  • Agents without the hooks get a restart-only participant. The server framework installs it on whatever app setup_webserver returns, so agents that build their own app, such as harbor_agent and the mini_swe_agent agents, take part too; it is installed once per app. Their rollouts are restarts: they never block prepare, keep running through the checkpoint, and are reported in prepare's restarts, so one opaque harness no longer stalls checkpointing for every environment. The seed reply carries x-ng-checkpoint-restart, and only a new seed waits while a checkpoint is open. After a crash, the controller retires the rollout and starts it over from its input.
  • With checkpointing on, agent work without a rollout ID is refused with rollout_id_required, and every agent refuses num_workers > 1.
  • Simple Agent implements the hooks for its environment server sessions and for its legacy /run loop. A session's export carries how many activations it has started, because each activation reports its observations under its own invocation ID (root, then activation-2, and so on). A restored session's next activation therefore gets the next ID, and an activation continued from its restored boundary keeps the ID it already had.
  • Simple Agent's session export carries its resources cookie jar as it is. Since fix(agent): keep the resources session cookie for verify when the agent holds no cookie jar #4449 on main, a session without a direct tool grant holds no resources cookie jar (resources_cookies is None), so the environment server keeps the session cookie from its own seed for /verify and /close_session. Export and restore keep that None instead of turning it into an empty jar, so a restored session's close still reports no jar and the environment server keeps the cookie from its resources seed.
  • A commit exports only the live sessions whose episode is inside its scope, as it already did for restored sessions. The controller leaves every participant's restarts out of the scope, so a session another server reports as a restart is not exported here either.
  • An unclaimed restored session, one whose replacement has not started yet, keeps its restored boundary in the next checkpoint while the controller's commit scope still continues it, so a second crash still continues it. Once the scope leaves it out, it is not exported, and it is retired.
  • Activations are idempotent. The environment server sends each activation's index, its 1-based count within the session, in the x-ng-agent-activation header. With checkpointing on, the agent exports the index and the reply of its last completed activation with the session. An invocation of an activation that already completed gets the stored reply back without running the loop: no model or tool call, and no change to observations or the activation count. This keeps the environment server and the agent consistent when the environment server's checkpoint still has the invocation ahead of it while the agent's checkpoint has the activation done. An index out of order is refused with 409 activation_out_of_order. Without the header, every invocation is a new activation, as before.
    • A request for an activation that is still running waits for it to end and then gets its reply, instead of being refused.
    • With checkpointing on, every activation of an agent session must carry the /ng-rollout/<id> prefix. One without it is refused with 400 rollout_id_required before any replay, claim or activation, since a restore rewrites the session's episode to the replacement attempt and the session can no longer stand in for a missing path. This holds for restored sessions and for sessions seeded in the running process; every caller that activates an agent session is a Gym environment server, and each sends the prefix.
    • Before it replays a reply or starts an activation, the agent checks the attempt the request's /ng-rollout/<id> path names against the session's, and refuses a mismatch with stale_attempt, so a crashed attempt's harness cannot use a restored session.
    • A replayed reply claims a restored session, like a new activation does, so a later commit that leaves the episode out does not free a session the live episode still uses.
    • The reply is kept only when the request carried an activation index.
  • A legacy /run sends activation index 1 on its self-dispatch. Its turn loop records its reply like an environment server session's activation, and the reply is exported with the legacy session. A checkpoint taken after the loop replied but before /run recorded its verify boundary therefore restores the reply instead of running the loop again; the reply leaves when the /run ends or a retire frees it.
  • Direct /v1/responses calls are restarts. With checkpointing on, a direct call with a rollout path is reported as a restart on every agent, and one without a rollout path is refused with rollout_id_required, as restart-only agents already did.
  • Restart-only seeds respect retires. A restart-only seed checks the retired mark after it waits for resume, a retire waits for seeds in progress, and the restart-only retire frees seeded sessions through retire_agent_session. The base retire_agent_session now releases a session through the agent's close hook, so a restart-only agent stops a sandbox it started.
  • Restarts from earlier stages do not hold up agent prepare. The agent participant leaves the restarts carried from earlier prepare stages (feat(checkpoint): add the resources server participant with asynchronous session hooks #3886) out of readiness, ready and its export. An activation of such an episode still parks at its next boundary until resume, but it no longer holds up prepare, so an episode single_agent_turn marked as a restart, from a restart-only resources seed or an MCP grant, does not stop the checkpoint while it is inside a slow tool call.
  • Retire cost is per rollout. Sessions, seeds in progress and direct restart calls are indexed by rollout, so a retire touches only its own rollout's, and each index entry leaves with the last item of its rollout.
  • Simple Agent boundaries leave out each turn's question, which repeats the request's input and the loop's earlier outputs. They store question_outputs, how many outputs each question held, and a restore rebuilds each question from the request's input and the restored outputs. The boundary snapshot is built directly, without a validate-and-dump round trip.
  • A completed activation's reply is kept once. A session-scoped reply to a request that carries an activation index leaves out _ng_trajectory, in the original and in the copy kept to replay it, since single_agent_turn takes the session's evidence from the AgentObservationBundle at close; callers that send no index, such as the UserSim environment server, still get it. A legacy /run drops its turn loop's kept reply in the same step that records its verify or return boundary, which holds the reply. Kept replies stay unserialized until a replay or an export needs them.
  • An episode that grants MCP access is a restart. single_agent_turn marks an episode as a restart when it grants an MCPToolAccess, so no checkpoint continues an agent that holds an MCP token past its seed; with feat(checkpoint): add the resources server participant with asynchronous session hooks #3886's token fence, a replaced attempt's token stops working after a restore.
  • Direct HTTP tool calls name their attempt. With checkpointing on, single_agent_turn's direct HTTP grant carries the seeding attempt's /ng-rollout/<capture key> prefix in its base URL, since feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 refuses a call on a restored session that names no attempt. Simple Agent sends each direct HTTP tool call under the rollout prefix of the activation it is serving, in place of the prefix in the grant, so after a restore the continued session's calls name the replacement attempt; its legacy /run tool calls carry the same prefix. No other agent with checkpoint support uses direct HTTP grants, and restart-only agents seed again after a crash and get a fresh grant.
  • A legacy /run closes its resources session after a replayable verification. It sends x-ng-checkpoint-close-follows: 1 on /verify, so feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 keeps the session tracked and exported until the close, and then closes the session, also when a replacement is restored at its return step. A failed close is logged and keeps the run's verdict.
  • A skip_verification legacy /run records its return boundary. It records the same boundary as a verified run, with the configured reward as its result, so a checkpoint that finishes while the turn loop's reply is on its way parks the run at return until resume instead of letting it finish uncaptured. A run restored at return returns its recorded result.
  • Closing a session stops its activation before cleanup. A close first stops an activation still running for it, so an activation the environment server gave up on stops calling the model and tools, and only then runs the agent's close hook. A close cancelled while it waits for the activation has not run the hook, so a retried close runs it exactly once. The session stays tracked, and holds up prepare, until the activation has stopped. A close cancels only an activation nothing has cancelled yet, so it never interrupts a retire's wait for the activation's cleanup.
  • A close waits for resume while agent admission is closed, as a seed does, and checks admission before it marks the session as closing. So a commit that has checked readiness always exports an open session, never a closing one. This cannot stall prepare through the environment server's final cleanup: coordination prepares environment servers before agents, and an episode in final cleanup keeps its environment server from being ready, so its agent close finishes before agent admission closes. A close that a retire took over is refused with stale_attempt, and checks after every await that its session record is still current, so it never writes a receipt for a session a retire freed.
  • A retire cancels the attempt's activations, or a restart-only agent's /run calls, and waits until they have stopped before the retire hook runs. The hook is where a sandboxed agent stops its sandbox. A retired session stays tracked until its activation has stopped and the legacy episode's retire and the retire hook have finished, so a retire cut short by its deadline at any of those points leaves it for a retry, which waits again without cancelling a second time. Meanwhile the session blocks the next checkpoint and is not exported, and a legacy /run blocks whether or not its session is still tracked.
  • A retired attempt's late activation is refused with stale_attempt until the controller forgets the rollout.
  • A session seed that arrives while a checkpoint is open waits for resume, and one already in progress holds up prepare and is exported. The environment server seeds inside a replay step that a checkpoint does not wait for, so the agent's session must match whichever boundary the environment server records. Refusing the seed failed the episode, and admitting it but leaving it out of the checkpoint let a replacement invoke a session nothing restored; a random checkpoint sequence against a real single_agent_turn deployment found both. A new session that is not a seed, such as a legacy /run, is still refused. Seeds are counted per session key, so a retried seed overlapping the first is tracked until both finish. A retire also waits for seeds of its attempt that are in progress, and a seed refused because its attempt was retired frees the state it built before the retire replies. A restore is refused while a seed is in progress.
  • SimpleResponsesAPIAgent._free_session frees a retired session that has no close receipt; it was _discard_session.
  • The participant answers the readiness wait's ready() without building its blocker list, and notifies only while a checkpoint is open.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR adds.

flowchart LR
  C["Controller<br/>NeMo RL, or rollout collection"]
  CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"]
  K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"]
  subgraph G["One participant per Gym server"]
    E["Environment server<br/>episode steps"]
    M["Policy model<br/>held responses, generation cuts"]
    A["Agent<br/>sessions parked at boundaries"]
    R["Resources server<br/>session state"]
  end
  W["Inference worker<br/>stages cut prefixes"]
  D[("Checkpoint directory<br/>records, then manifest")]
  L[("Capture ledger<br/>retire and delete from #3938, #3939")]
  C --> CO --> K
  K --> E & M & A & R
  M --> W
  M --> L
  G --> D
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class A this
Loading

One checkpoint, a crash, and the restore, end to end:

sequenceDiagram
  participant C as Controller
  participant G as Gym participants
  participant D as Checkpoint directory
  C->>G: prepare, in order environment, model, agent, resources
  Note over G: admission closes, in-flight work parks at a boundary,<br/>undelivered model responses are held
  G-->>C: prepared, or blockers at the deadline
  C->>G: commit with the episodes the controller continues
  G->>D: each participant writes its records, then its manifest
  Note over G: restored state the commit's scope leaves out is deleted
  C->>C: publish the checkpoint with the controller's own state
  C->>G: resume, in order resources, agent, model, environment
  Note over C,G: crash - every Gym process dies
  C->>G: restore the checkpoint in fresh processes, all or nothing
  D-->>G: records installed under attempt N+1, attempt N's capture ledger retired
  C->>G: resume
  C->>G: /run as attempt N+1 continues each episode from its boundary
  Note over C,G: retiring an episode, only while no checkpoint is open
  C->>G: retire - environment, then agent, then model and resources
  Note over G: each server marks the attempt, stops its work, waits, frees its state, then replies
  C->>G: forget, once the rollout is done for good - same order
  Note over G: the mark refusing the retired attempt goes away
Loading

This PR

An agent session's states. Prepare is ready once no activation is between boundaries.

flowchart LR
  I(["idle"]) -->|an activation starts| R(["running"])
  R -->|before a model call, or after a tool round| B(["at boundary"])
  B -->|next step, when no checkpoint is open| R
  B -->|parked while a checkpoint is open| B
  R -->|the activation ends| I
Loading

Which participant an agent gets.

flowchart TD
  Q{"checkpoint_sessions_supported"} -->|yes| P["AgentSessionParticipant<br/>export_agent_sessions, restore_agent_sessions,<br/>retire_agent_session"]
  Q -->|no| O["Restart-only: reported in prepare, never blocks,<br/>retired and started over after a crash"]
Loading

Closing or retiring a session stops its activation first: close cancels it, and retire also waits for it before the retire hook, which is where a sandboxed agent stops its sandbox.

flowchart LR
  E["close from the environment server,<br/>or retire"] --> X["cancel the running activation"] --> W["retire waits for it"] --> H["retire_agent_session hook<br/>stops a sandbox the agent owns"]
Loading

Where this sits in the stack

This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.

The stack builds on the token-capture cleanup PRs #3938 (capture ledger retire and delete) and #3939 (complete-record retire and delete), which have merged. #3882 to #3887 have merged too, so #3888's base is main. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.

  1. feat(checkpoint): add the participant control plane, episode steps, and coordination #3882 (ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)
  2. feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883 (ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)
  3. feat(token-capture): add worker staging helpers for generation cuts #3884 (ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts (merged)
  4. feat(checkpoint): continue environment server episodes from their boundaries #3885 (ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (merged)
  5. feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 (ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks (merged)
  6. feat(checkpoint): declare twenty training verifiers stateless with replayable verification #3887 (ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare twenty training verifiers stateless with replayable verification (merged)
  7. feat(checkpoint): add the agent session participant and Simple Agent continuation #3888 (ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuation (this PR)
  8. test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 (ananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordination
  9. feat(checkpoint): checkpoint evaluation runs from rollout collection #3893 (ananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collection
  10. feat(checkpoint): spans and metrics for partial-rollout checkpoints #3903 (ananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpoints
  11. feat(checkpoint): resources, agent, and environment servers with several workers #3909 (ananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workers
  12. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 (ananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operations

Partial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the checkpoint: block unset, no server installs any checkpoint routes or behavior.

Relationship to the old stack (#2939 to #2946)

Issue

No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.

Validation

Run on this branch, on the current main:

  • Each of these behaviors has a test that fails without its change:
    • an agent session activation without the rollout prefix is refused, for a restored session and for one seeded in the running process, with no model call, replay or claim, while checkpointing off still accepts it;
    • a legacy /run that skips verification parks at its return boundary until resume;
    • an episode the environment server reports as a restart, from a restart-only resources seed or an MCP grant, holds up neither the agent stage nor the resources stage of prepare, checked through coordination with a single_agent_turn server and real environment, agent and resources participants, and the agent neither waits for nor exports such an episode's activation until resume;
    • an indexed session reply and its replay carry no trajectory, nor does the export, and a kept reply stays unserialized until a replay or an export needs it;
    • a checkpoint of a legacy /run parked at verify holds its reply once, and a restore still replies;
    • a restore after the legacy turn loop replied returns that reply without running the loop again;
    • an episode that grants MCP access is a restart;
    • a close that arrives after prepare waits for resume and the commit exports its session, a cleanup close of a parked episode after the agent stage waits without holding up prepare, and a retire that frees a session while its close waits leaves no receipt behind;
    • a close stops the activation before its cleanup starts, and a close cancelled while it stops the activation runs the cleanup once when retried;
    • a request for an activation still running waits for it and gets its reply;
    • a request on the replaced attempt's path is refused for the restored session;
    • a restart-only seed that arrives after its rollout was retired is refused, and a restart-only retire frees its rollout's sessions and waits for a seed in progress;
    • a direct /v1/responses call is a restart, and one without a rollout path is refused;
    • a boundary leaves out the turn questions, and a resumed loop rebuilds them;
    • an activation without an index keeps no reply;
    • a restored session answered with its stored reply survives a commit that leaves its episode out;
    • with checkpointing on, a direct HTTP grant names the seeding attempt, and a restored session's direct HTTP tool call reaches the restored resources session under the replacement's prefix, through Simple Agent and a real resources server;
    • a legacy /run closes its resources session after a verification a checkpoint does not wait for.
  • A separate ordering check confirms that an environment episode's cleanup close finishes before the agent stage of prepare; it passes with or without the change.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed, including:
    • closing a session stops its running activation;
    • a retire stops the activation before it replies, and a straggler is retired after resume;
    • a restart-only agent's /run has stopped when its retire replies;
    • a commit whose scope leaves out a restored session retires it;
    • a session whose retire hook was cut short stays blocking, is not exported, and is freed by a retry;
    • a seed arriving during an open checkpoint waits for resume, and one in progress when admission closes holds up prepare and is exported;
    • overlapping seeds of one session each hold up prepare, a retire waits for a seed in progress and frees what it built, and a restore is refused while a seed is in progress;
    • a closed session holds up prepare until its activation has stopped, and neither a close nor a retire cancels an activation twice;
    • a restored legacy /run claimed at verify is no longer pending and survives a commit that leaves it out;
    • an agent that builds its own app still reports its runs as restarts, and the participant is installed once per app;
    • ready() does not build the readiness report, and nothing notifies outside a checkpoint;
    • a commit does not export live sessions outside its scope.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider responses_api_agents/simple_agent/tests: passed, including that a restored session's next activation does not reuse an invocation ID and that an activation continued from its restored boundary keeps its own; that a restored activation keeps its boundary through a second checkpoint; that a model reply arriving after prepare is held at the boundary before the call; that an activation already completed is answered with its stored reply without running again; that a legacy /run restored at verify or return still reports its trajectory; and that a session without a resources cookie jar restores without one.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider environment_servers/single_agent_turn/tests: passed, including the timeline where the agent's reply to an invocation is lost to a restore: the environment server invokes again and the restored agent answers with the stored reply while the model sees no new call.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_terminal_attribution.py: passed, including that a call delivered before a checkpoint and issued again after a restore keeps only the re-issued call.
  • The full core unit suite (tests/unit_tests, eight processes, without test_opensandbox_compose.py, which needs the optional opensandbox package): passed except a sandbox retry test, which fails the same way on main in this development environment.
  • pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.
  • Every commit carries a Signed-off-by line.

Rollout evidence

  • Process-level evidence is in the top PR (episodes on the typed single_agent_turn request path and legacy /run episodes continue from their last boundary).
  • Real model: see the rollout evidence in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889, whose opt-in tests drove this PR's code with Qwen/Qwen3-4B-Instruct-2507 on vLLM: saved rollouts continued after a kill and restore, restarts started over, and every rollout finished with reward 1.0. Every rollout ran Simple Agent.

Compatibility

  • New agent hooks and class attribute: checkpoint_sessions_supported, export_agent_sessions, restore_agent_sessions, retire_agent_session. Existing agents need no change and are restart-only under checkpointing.
  • With checkpointing on, an agent session activation without the /ng-rollout/<id> prefix gets 400 rollout_id_required. A caller outside Gym that activates Simple Agent sessions with only the session cookie must add the prefix; no caller in the repository is affected.
  • With checkpointing on, agent calls without a rollout ID get 400 rollout_id_required, and agents with num_workers > 1 fail at startup.
  • With checkpointing on, a direct HTTP grant's base URL ends in the seeding attempt's rollout prefix. An agent that continues a session after a restore must send its tool calls under its own activation's prefix; Simple Agent does.
  • With skip_verification and checkpointing on, a legacy /run caught by a checkpoint waits at its return boundary until resume. Nothing changes with checkpointing off.
  • New request header x-ng-agent-activation on agent session invocations, and a 409 activation_out_of_order refusal. An environment server that can invoke an activation again after a restore must send the header.
  • AgentSessionParticipant.close_session is now a coroutine, and the agent implements SimpleServer.setup_checkpoint_participant(app), the hook feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 adds that the framework calls after setup_webserver.
  • A commit exports only the live sessions inside its episode_ids scope. A commit request without episode_ids still exports every live session.
  • Nothing changes unless the global checkpoint: block sets enabled: true.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:agent Agent harnesses and Responses API agent behavior labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub requested a review from zyzhou5 October 1, 2026 19:04
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-agent branch from 600fcc2 to dddd8bf Compare October 1, 2026 19:38
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-agent branch from dddd8bf to cbabe4a Compare October 1, 2026 21:53
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-agent branch from cbabe4a to d25d7e6 Compare October 1, 2026 22:05
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-agent branch from d25d7e6 to a508546 Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-agent branch from a508546 to 5799bbe Compare October 2, 2026 19:43
Comment thread nemo_gym/_checkpoint/agent.py Resolved
Comment thread nemo_gym/base_responses_api_agent.py Resolved
Comment thread nemo_gym/base_responses_api_agent.py Resolved
Comment thread nemo_gym/_checkpoint/agent.py Resolved
Comment thread nemo_gym/_checkpoint/agent.py Resolved
response.set_cookie(k, v)
if trajectory is not None:
model_response = model_response.model_copy(
update={_INTERNAL_TRAJECTORY_KEY: trajectory.model_dump(mode="json")}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2/perf] Avoid duplicating native-session trajectories at the close boundary

When single_agent_turn parks at close_agent, the environment boundary already stores this response including _ng_trajectory, while the agent session exports the same response again through completed_reply. Because trajectory turns retain their cumulative question, this bypasses the compact boundary representation and can reintroduce quadratic checkpoint size across completed activations. Since session evidence is returned authoritatively as AgentObservationBundle on close, could session-scoped calls avoid attaching _ng_trajectory here? The original and replayed response should both omit it so replay equality is preserved.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied to the replies a checkpoint keeps, not to every session-scoped reply.
The UserSim environment server (environment_servers/nemo_user_sim/app.py) reads _ng_trajectory from each session-scoped reply to build that invocation's observations, so dropping it from every session reply would break it.
That server sends no activation index, and only a reply to a request with an index is kept and replayed.
So a session-scoped reply to a request that carries an activation index now has no _ng_trajectory, and neither does the copy kept to replay it.
The original reply and its replay are byte for byte equal.
single_agent_turn sends the index, and it still gets the session's evidence from the AgentObservationBundle at close.
With checkpointing on, the response that single_agent_turn sends to /verify and returns no longer carries the _ng_trajectory key, which nothing reads on that path.
Measured with sessions parked after one completed activation, observability on, each turn with a 2 KB input, a reasoning item, a tool call and a 1 KB tool output, the agent's records for those sessions go:

  • 256 sessions with 32 turns: 392 MB before, 38 MB after;
  • 1,024 sessions with 32 turns: 1,568 MB before, 151 MB after;
  • 256 sessions with 64 turns: 1,353 MB before, 75 MB after.
    Covered by test_an_indexed_session_reply_and_its_replay_carry_no_trajectory_and_neither_does_the_export, which also checks that a request without an index still gets the trajectory.

loop = self._legacy_replies.get(session_key)
if loop is None or loop.completed_reply is None:
return {}
return {"completed_activation": loop.completed_activation, "completed_reply": loop.completed_reply}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2/perf] Omit the legacy loop reply once the episode boundary owns it

At the legacy /run verify and return stages, AgentSessionRecord.episode already contains the response and _ng_trajectory, but this hook also exports the same full response in completed_reply. That leaves two trajectory copies in one checkpoint record. Could export omit completed_reply whenever the legacy episode boundary has reached verify or return? Clearing it only after await boundary(...) is too late because that call can park while the checkpoint exports; the handoff should be atomic with recording the new boundary or conditional during export.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, at the moment the boundary is recorded.
Right before a legacy /run records its verify or return boundary, which holds the turn loop's reply, it drops the session's own copy of that reply.
Nothing awaits between the two, so no export sees both copies or neither.
The completed activation number stays, so a late duplicate of the loop request is refused rather than run again.
With the same setup as the measurement in the other thread, legacy /run episodes held in a replayable verification go:

  • 256 episodes with 32 turns: 745 MB before, 373 MB after;
  • 1,024 episodes with 32 turns: 2,980 MB before, 1,490 MB after;
  • 256 episodes with 64 turns: 2,630 MB before, 1,315 MB after.
    The copy that remains is the boundary's, whose trajectory turns still hold their full questions.
    Covered by test_a_checkpoint_of_a_legacy_run_parked_at_verify_holds_its_reply_once_and_a_restore_still_replies.
    It commits while the /run is in its verification, checks that the record holds the trajectory once, and restores a replacement that replies with the same output and reward.

# Only a caller that sends the index can ask for the reply again.
loop.completed_reply = (
{
"response": model_response.model_dump(mode="json"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3/perf] Defer serialization of indexed activation replies

Every completed indexed session or legacy activation with checkpointing enabled dumps the full response here, even when no duplicate request arrives and no checkpoint occurs; FastAPI then serializes the returned response separately. Could completed_reply retain the response object and serialize it only in _completed_activation_reply or export_agent_sessions? That removes an eager full-response traversal from the normal path; the separate trajectory-deduplication changes are still needed to reduce retained memory.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied: completed_reply keeps the response object the activation returned.
_completed_activation_reply and export_agent_sessions serialize it only when they need it, and a restored reply keeps the JSON it was exported as.
The checkpoint format is unchanged.
Covered by test_an_indexed_reply_is_kept_unserialized_until_a_replay_or_an_export_needs_it, which checks that the kept reply is still the object after the activation, and that the replay and the export give the same bytes as the original reply.

"reward": float(self.config.skip_verification_reward),
"verification_skipped": True,
}
elif stage == "return":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1/race] Park skip-verification runs at a return boundary

The skip_verification branch returns without recording the final return boundary. If checkpoint preparation finishes while the self-dispatched activation reply is still in flight, /run can consume that reply, remove its legacy session, and finish while the participant remains prepared. A commit can then contain no agent continuation for an episode the controller still considers in flight, so a restore repeats the model and tool loop. I reproduced this on the current head: prepare returned prepared, then /run completed and _sessions was empty before resume. Please record the same replayable return boundary before returning the configured reward, and cover that the run stays parked until resume.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: with skip_verification, Simple Agent's legacy /run now records the same replayable return boundary a verified run records before it returns the configured reward.
The boundary holds the response, the result with the configured reward, the cookies and the verify mode.
So when prepare finishes while the self-dispatched turn loop's reply is still on its way back, the /run parks at that boundary until resume instead of finishing, its legacy session stays registered, and the commit exports the episode at its return step.
A replacement restored at return now returns the recorded result whether or not the run verified.
Before, the skip_verification branch was checked first, so a restored run would have rebuilt its result instead of using the recorded one.
The /run still closes no resources session after a run that skipped verification.
Covered by test_a_skip_verification_run_parks_at_its_return_boundary_until_resume, which follows your reproduction:

  • prepare returns prepared while the turn loop's reply is in flight;
  • before resume the /run has not finished and its legacy session is still held, and the commit exports a return boundary that holds the configured reward;
  • after resume the /run finishes with the configured reward;
  • a restore from that commit returns the same reward without a seed, model call, tool call or verification.
    On the previous head the test fails at the first check after prepare, because the /run has already finished.

| {episode_id.capture_key for seeds in self._seeding.values() for episode_id in seeds}
)
at_boundary = sum(session.state == "at_boundary" for session in self._sessions.values())
return PrepareReport(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2/liveness] Do not let already-restarted episodes block agent prepare

Restart status is currently local to the participant that reports it. When single_agent_turn marks an episode as a restart because the Resources seed says so (or because it grants MCP access), this native agent session is still treated as checkpointable here. If its activation is between boundaries—for example inside a slow Resources tool call—the environment reports ready with restarts=["r"], but the agent reports blockers=["r"], and staged preparation stops on a rollout the commit will exclude anyway. Please propagate the restart bit into the agent session, or carry accumulated restarts into later prepare stages, so these sessions keep running and do not block or export.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by carrying restarts forward in coordination.
coordination.prepare now collects the restarts each stage reports and sends them to every later stage in a new optional restarts field of the prepare request, as capture keys.
The model and agent stages get the environment servers' restarts, and the resources stage gets the environment servers' and the agents'.
A participant keeps the restarts it was given until the checkpoint ends: resume, an abort, an expired lease or a failed prepare clears them, and a prepare asked again adds to them.
It leaves those episodes out of its blockers and out of its export:

  • the agent participant (AgentSessionParticipant.readiness and ready) no longer waits for their running activations, closes or seeds; such an activation still parks at its next boundary until resume, but it no longer holds up prepare;
  • the resources participant no longer waits for requests in flight on their sessions;
  • with several workers, the coordinator passes them to every worker, including one that joins while the checkpoint is open.
    The model participant needs no change: it never waits for a generation it holds, only for a response that was already streaming when its admission closed, and that response finishes on its own.
    The commit scope already left out every reported restart (PrepareResult.continued), and with this change none of these participants exports one even for a commit without a scope.
    A retire still drains and refuses a restart's sessions as before.
    A prepare request without the field behaves as before, and coordination sends the field only when an earlier stage reported a restart.
    The field and the resources participant's use of it are in feat(checkpoint): add the resources server participant with asynchronous session hooks #3886, the agent participant's use is here, and the forwarding to workers is in feat(checkpoint): resources, agent, and environment servers with several workers #3909.
    Covered by:
  • test_an_episode_the_environment_reports_as_a_restart_does_not_hold_up_the_agent_or_resources_stage, your case end to end through coordination with a single_agent_turn environment server and real environment, agent and resources participants: the environment reports r as a restart, once from a restart-only resources seed and once from an MCP grant, while r's activation is inside a resources tool call the server holds; prepare succeeds without waiting for the call, the scope and every commit leave r out, and r finishes after resume. On the previous head prepare stops at the agent stage with blockers={'agent': ['r']}, and without the resources change the MCP case stops at the resources stage with the call in flight;
  • test_each_prepare_stage_carries_the_restarts_every_earlier_stage_reported, which also checks that a stage after no restart gets the request without the field;
  • test_an_activation_of_a_restart_an_earlier_stage_reported_neither_blocks_nor_is_exported_until_resume, where a prepare asked again names the restart, and after resume the next checkpoint waits for r again;
  • test_a_request_in_flight_for_a_restart_an_earlier_stage_reported_does_not_hold_up_prepare for the resources server;
  • test_every_worker_leaves_out_the_restarts_an_earlier_stage_reported_until_resume for several workers.

@macandro96 macandro96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly lgtm - just one small hardening

path_episode = EpisodeId.from_capture_key(capture_key) if capture_key is not None else None
if state is not None:
# The rollout path names the caller's attempt, which the session's must match.
return state.request.agent_session_id, path_episode or state.request.episode_id, state

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NOTE/hardening] Enforce attempt identity on checkpointed session activations

When capture_key is absent, this fallback attributes the request to state.request.episode_id, which restore has already rewritten to the replacement attempt. A late cookie-only call from the replaced harness would therefore be labeled as the replacement before check_session: it could replay and claim the cached activation, or start the next one. Built-in environment callers already use rollout-prefixed paths, and a full fresh-process restore makes this unlikely, so I do not consider it a blocker. Still, could we require the rollout prefix for checkpoint-enabled session activations (or at least persistently for restored sessions), matching the Resources participant's treatment of cookie-only restored calls? A regression for an unprefixed request after restore would make the fencing assumption explicit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems similar to this but the above situation itself doesn't seem super likely

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, and for every checkpointed session activation rather than only restored ones.
With checkpointing on, Simple Agent now refuses an activation of an agent session that has no /ng-rollout/<id> prefix with 400 rollout_id_required.
_activation_target no longer falls back to state.request.episode_id, so the refusal happens before the agent replays a completed reply, claims a restored session, or starts an activation.
It applies the same way to a restored session and to a session seeded in the running process, which matches the resources participant's refusal of cookie-only calls.
With checkpointing off nothing changes: the session cookie alone still activates the session.

The rule covers every activation rather than only restored sessions because every caller that activates an agent session is a Gym environment server, and each already sends the prefix:

  • the typed single_agent_turn request path and the legacy flat-row wrapper, which share _agent_responses_path;
  • the UserSim environment server, through responses_path.
    The legacy /run self-dispatch carries no agent session, so it is not an activation of one; it already sends /ng-rollout/<capture key> through url_path_for_run, which the checkpointed /run requires.
    Resources tool calls are different, because third-party harnesses send them, so that rule had to tolerate callers Gym does not control.
    Seeds and closes are unchanged: both name their episode in the request body and never used this fallback.

The refusal lives in Simple Agent, not in the agent base.
The base does not route activations: each checkpoint-capable agent's /v1/responses decides which attempt a request runs as and passes it to AgentSessionParticipant.check_session, and Simple Agent is the only in-tree agent that activates checkpointed sessions.
The check_session docstring now says the attempt must come from the request's rollout path and never from the session, so another agent that adds session activations follows the same rule.

Covered by:

  • test_an_activation_without_a_rollout_prefix_is_refused_for_the_restored_session, your case: after a restore as rollout-a1, an unprefixed call with the session's cookie is refused with rollout_id_required without a model call, without replaying the completed reply, and with the session still pending its claim; /ng-rollout/rollout/... is refused with stale_attempt; /ng-rollout/rollout-a1/... continues. It runs without an activation index, with the completed activation's index, and with the next one. On the previous head the unprefixed call is not refused, so it replays and claims the session or starts the next activation;
  • test_an_activation_without_a_rollout_prefix_is_refused_only_with_checkpointing, for a session seeded in the running process: with checkpointing on the unprefixed call gets 400 rollout_id_required and starts nothing, so the prefixed call that follows is still activation 1; with checkpointing off the unprefixed call runs as before. On the previous head the checkpointing-on case returns 200;
  • the UserSim bridge test now also checks that its activation path carries the episode's prefix.

macandro96
macandro96 previously approved these changes Oct 15, 2026
…continuation

An agent server is a participant for its agent sessions and its legacy
/run episodes. A checkpoint parks each session's activation at its
latest turn-loop boundary, and a restored session continues from it.
A seed that arrives during a checkpoint waits for resume, and one in
progress holds up prepare. Simple Agent implements the hooks, including
its activation numbering.

An agent without the hooks is restart-only: nothing of it is in a
checkpoint. Its in-flight work is reported as restarts, never holds up
prepare, and keeps running through the checkpoint; its seed replies say
so, so an environment server's episode that uses it restarts too. A
legacy /run whose resources server cannot capture its session is a
restart as well.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38821199883

This branch was previously deployed

1 inactive deployment
public — c56cb675 Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5557
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants