Skip to content

feat(example_mcp_weather): serve the typed resources sessions API - #5480

Open
yaoyu-33 wants to merge 44 commits into
mainfrom
yuya/gymint-169-mcp-weather-sessions
Open

yaoyu-33 wants to merge 44 commits into
mainfrom
yuya/gymint-169-mcp-weather-sessions

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Serve typed weather sessions and use the shared lifecycle for seed, verification cleanup, close and checkpoint retirement. Weather now supplies three state hooks and its scoring logic; the base owns locking, identity, release and bounded retry evidence.

The shared lifecycle in #5709 and MCP support in #5708 are merged into main. The child diff contains only this environment and its related documentation/configuration; shared implementation is reviewed once in the dependency. No separate issue is needed for this focused migration of an existing resources server.

  • The MCP validator/request-injection fix is isolated in fix(mcp): preserve seed validators and annotated request injection #5708 and the common lifecycle in feat(sessions): centralize resources lifecycle and verification ownership #5709. This child contains only the weather implementation/tests and its README/tutorial.
  • Verification captures a compact called-city snapshot before scoring, releases live state even when scoring fails, and uses the shared release hook. Concurrent close waits for verification cleanup. Close invalidates retained retries; a successful cookie reseed keeps the cookie ID and replaces the preceding run.
  • The tutorial uses public APIs, removes obsolete Field/PrivateAttr explanations and aligns cookie terminology. Claude Code wiring remains a compatibility example until its adapter supports per-session MCP grants; this change does not claim all agent-owned /run paths are deprecated.

Retains the unique called cities before scoring, because the tool history it needs is released at verify.

Validation on 636972ec12a3e8e146fbdcbe8567dcd77df55779, Python 3.13.14, with this checkout first on PYTHONPATH and NEMO_GYM_EXTRA_ROOTS pointing to it:

Final-source real-model smoke passed on 534e79e3852c9fcc05addaf82196a364d4a28217 (shared core d273280a6841) with nvidia/nemotron-3.5-lightning-30b-a3b, Python 3.13.14 and actual MCP initialize/list/tools/call on macOS. A task-local model/tool loop made two real Gym model HTTP calls and one MCP weather call. Verification scored 1.0 unmasked; a same-request retry also scored 1.0; owner close returned 200. All five live/ownership maps were empty after verify and close. All 9,181 regular source files and 32 symlink targets matched before and after execution, and tracked processes ended. This validates the MCP path, not a particular production agent adapter or benchmark baseline. Earlier helper-schema and shared-storage failures remain preserved in the versioned evidence. app.py differs from the smoke revision by a comment only. The shipped claude_code_agent /run pairing has no real run at this head; it is covered by unit tests.

These targeted checks do not replace the repository CI matrix or approval of the current head.

Compatibility

Weather now releases its call history at verify and retains only the unique called cities (main kept the whole state dict for the life of the process). An unseeded or released /get_weather call is now 404 (main created an empty session). /verify for an unseeded or expired session is now a masked session_lost sample (main scored 0 unmasked). ExampleMCPWeatherSeedSessionResponse was removed (the seed reply is unchanged on the wire). A second verify is answered only within session_close_retry_window_seconds. The tests now cover only weather-owned behavior and one contract run per transport; the base lifecycle is tested in #5709.

Main refresh validation

Merged main at acb67734c into this branch, retaining main's MCP annotation resolution and seed-validation regressions. The diff against main contains only this environment and its related documentation/configuration. No new environment behavior was added by this refresh.

On workstation ws (Python 3.13.14), this environment's complete test directory passed: 16 passed. The nine environment suites passed 904 tests in total. Shared lifecycle, MCP seed/exposure, and process-supervisor tests passed 213 tests. Tests ran in a combined validation checkout with the merged core and the exact environment sources from these nine heads, with the checkout first on PYTHONPATH and NEMO_GYM_EXTRA_ROOTS. Prior real-model smoke evidence above remains historical; no fresh model rollout was run for this merge-only update. New merge commits are DCO signed off.

pre-commit run --all-files passed on the combined checkout, including Ruff, formatting, guardwords, config flags, environment table generation, and import layering. The generated README row matches the row already included in #5485. git diff --check passed for each refreshed branch.

An Environment Server such as single_agent_turn seeds a resources server
with a typed ResourcesSeedSessionRequest and releases it with
/close_session. The weather server took only its legacy seed body and had
no close, so its per-session weather calls were never released, and a
tool call after the session ended silently recreated state.

The server now:
- accepts the typed seed next to the legacy body, keys the session by
  resources_session_id and binds the cookie to it, so MCP auto-exposure
  signs the same ID into the resources_tools token it returns;
- keeps the recorded weather calls when a seed is repeated for the same
  episode, refuses another episode (409) and task_data that fails
  validation (422);
- closes idempotently by resources_session_id, with or without the
  cookie, and fences a late seed with ResourcesSessionFence (409);
- answers a get_weather call for a session it no longer holds with
  ResourcesSessionLostError (404), since the signed MCP token outlives the
  close, and a verify for it with a masked lost-session sample.

The legacy /run path is unchanged apart from dropping the seeded
expected_city, which verify never read. The README and the MCP tutorial
describe the session lifecycle, and the tests run the session
conformance check with MCP exposure on and off.

Refs GYMINT-169

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Nov 3, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

yaoyu-33 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b515275

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

@yaoyu-33
yaoyu-33 marked this pull request as ready for review November 3, 2026 18:33
@yaoyu-33 yaoyu-33 added the needs-review PR is ready for code review and waiting on a reviewer label Nov 3, 2026
yaoyu-33 and others added 2 commits November 3, 2026 11:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…conformance module path

The README's Environment Server snippet reused the requests.Session from the block above, whose cookie names the legacy seed's session, so session routing answered the typed seed with 409. It now starts a fresh session.

The tutorial named nemo_gym.testing.session_conformance, which moved to nemo_gym.server_runtime.testing.session_conformance, and its simplified seed now notes the 422 validation that its prose promises. The test import moves to the new path too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test fda84b8

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 3, 2026 21:46

@cwing-nvidia cwing-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is meant to be a minimal MCP example, but most of the new code in app.py is session lifecycle, not MCP. A reader who came to learn MCP has to get through it first

Most of that is new lifecycle for the /run seed (release at verify, retry record, cookie rotation). Why add it here, given:

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33 and others added 8 commits November 5, 2026 16:27
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The wrapper strips Annotated only to recognize an injected Request. A
validator written directly on the body parameter, the form an author is
most likely to write, was covered only through a union arm, so a wrapper
that stripped every parameter still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

# Conflicts:
#	nemo_gym/server_runtime/base_resources_server.py
…ment the hook contract

- A cancellation that arrives while a failed initialization is being cleaned up now propagates instead of
  being swallowed by the cleanup guard.
- A cookie reseed that released the preceding run moves its finished receipt into the retry window, so a
  failed replacement no longer leaves a pending-cleanup record with nothing to release.
- Document that a disconnect leaves a verification release pending, that the release hook runs under the
  session lock, that state is per worker with session routing keeping a session on its owner, and the
  override rules for servers that keep their own seed and close.
- Rename the example server's private episode table so it no longer shares a name with the base's.
- Add regressions for the receipt deadline guard, worker ownership wiring, cookie seed races and fences,
  typed seed takeover, restore refusals and a cancelled close waiting on verification.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…cument the 404 at verify

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 636972e

The docs now describe only the typed Environment Server path. The cookie
seed still works unchanged; the PR description records that compatibility.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4db600a

Base automatically changed from yuya/shared-resources-lifecycle to main November 8, 2026 06:36
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d53e4ac

tayunvidia pushed a commit to tayunvidia/Gym that referenced this pull request Nov 13, 2026
* feat(openenv): serve the typed resources sessions API

An Environment Server such as single_agent_turn seeds a resources server
with a typed ResourcesSeedSessionRequest and releases it with
/close_session. The OpenEnv adapter took only the legacy seed body,
answered with a response the Environment Server rejects, and had no
close, so an episode that never reached /verify leaked its environment
instance.

The server now:
- accepts the typed seed next to the legacy body, keys the environment
  instance by resources_session_id and binds the cookie to it; a repeated
  seed for the same episode keeps the instance, another episode gets 409;
- closes the instance by resources_session_id, with or without the
  cookie, unless /verify already did; closing an unknown or closed
  session succeeds, a late seed is fenced with 409, and an env.close()
  failure raises so the close can be retried;
- answers a tool call (/step or a per-tool route) for a session it no
  longer holds with ResourcesSessionLostError (404), and a verify for it
  with a masked lost-session sample instead of a reward of 0;
- closes an environment instance whose reset() fails during a seed.

/verify still closes the instance, since a legacy /run never closes its
session. The README describes the session lifecycle and no longer claims
a verifier_metadata.expected_output check that the code does not do.

Refs GYMINT-176

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(openenv): retain typed state and failed reset ownership

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(openenv): end a typed episode at verify so a late step cannot change a retried verify

A typed session keeps its environment until /close_session, so a step still in flight when the Environment Server verified could add to the accumulated reward and make a retried verify report a different score. Verify now marks the episode done; later steps get the existing "Episode has ended" error.

Also points the conformance import at nemo_gym.server_runtime.testing, fixes the close docstring, and makes two tests check what their comments claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(openenv): release verified and retired session state

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(server_runtime): share resources session policy checks

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(openenv): reuse shared session checks

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(server_runtime): own stateful resources lifecycle

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(openenv): inherit the session lifecycle

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(resources): fold session lifecycle into SimpleResourcesServer

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(server_runtime): drop cookie rotation from the shared resources session lifecycle

Mirror of the NVIDIA-NeMo#5480 change so the two PRs keep the same base: a cookie seed keeps the cookie's session ID
and replaces the previous run's state instead of minting a fresh ID, so ResourcesSessionFence.
legacy_seed_session_id becomes check_cookie_seed. The partial-typed-seed guard is the internal
reject_partial_typed_seed and no longer rejects a dataset row's own task_id column. The shared lifecycle
and policy tests are the NVIDIA-NeMo#5480 versions, which also import EpisodeId and TaskId from
nemo_gym.core.episode_types (nemo_gym.core.episode does not exist). The OpenEnv test and README now
describe a legacy /run that reuses a verified cookie replacing its environment under the same ID.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(server_runtime): drop a finished run's verify record on a cookie re-seed

A cookie seed keeps its session ID and replaces the previous run's state, but the previous run's verify retry record stayed, and remember() keeps the first record. In openenv the next run's verify then returned the finished run's reward without scoring or releasing the new environment; in example_mcp_weather a resent verify of the next run replayed the finished run's weather calls.

Drop that record once the new run's state is initialized, through a new ClosedResourcesSessionRecords.forget() (unit-tested; the same hunk is in NVIDIA-NeMo#5485-NVIDIA-NeMo#5488). A failed re-seed keeps it. Shared base and lifecycle tests are identical in NVIDIA-NeMo#5480 and NVIDIA-NeMo#5483.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(openenv): preserve seed validation through shared MCP lifecycle

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(server_runtime): close gaps in the shared resources session lifecycle

- The repo contract test counts a server that implements the state hooks as
  closing through the inherited close_resources_session. newton_bench,
  workplace_assistant, tavily_search and indirect_prompt_injection override
  seed_session but close through the hooks, and failed it.
- Export reject_partial_typed_seed from nemo_gym.server_runtime, so servers
  that override seed_session reuse the shared guard instead of a copy.
- The MCP seed wrapper still recognises a request param annotated as
  Annotated[Request, ...]; keeping Annotated metadata had made it inject a
  second Request and fail with KeyError.
- Name the 409 for a typed seed over a cookie session "cookie-seeded", not
  "legacy", and document that a served seed_session override declares
  request before body.
- Shared tests now cover the MCP-wrapped seed route, close after verify
  (checked against the verified episode), and a cookie seed of a verified
  typed session.
- The add-benchmark skill describes the three state hooks as the default
  for a stateful server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(openenv): cover both malformed typed seed shapes behind MCP

The malformed typed seed test covered only a body with a resources_session_id
and no envelope. Parametrize it over both shapes reject_partial_typed_seed
refuses (that one, and dict episode_id plus dict task_id with no
resources_session_id), with and without MCP exposure, and assert that the
complete typed seed of the same session then returns 200, carrying
resources_tools when MCP is on. Both MCP cases fail if the seed wrapper drops
include_extras=True.

README: say that OpenEnv's own /verify masks a verify after close, moves the
typed identity out of _session_episodes and writes _verified_sessions, rather
than claiming the base owns all of that bookkeeping; and say that the shipped
configs pair OpenEnv with legacy_agent, whose simple_agent seeds by cookie.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(server_runtime): retire a hook session without fencing it

The inherited retire_session_state recorded a typed session as closed. The
checkpoint participant also calls that hook to undo a restore
(delete_restored) and to drop an unclaimed session, and both promise to
refuse nothing later; with the fence, a restore that was rolled back and
installed again in the same process left its session fenced, so its verify
was masked as lost and its continuation's seed got 409. The participant
already refuses a retired attempt's requests until it forgets the rollout,
so retirement now releases the state and drops the identity without
recording a close. A close still fences.

Also write down that an exported server using the state hooks carries each
session's identity through its checkpoint itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(mcp): preserve annotated seed validators and request injection

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* feat(sessions): centralize resources lifecycle and verification ownership

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(sessions): preserve cleanup ownership and shared verification semantics

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(resources): preserve grades and retry cleanup ownership

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(openenv): reject tools after completed verification

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(mcp): keep a validator annotated directly on the seed body

The wrapper strips Annotated only to recognize an injected Request. A
validator written directly on the body parameter, the form an author is
most likely to write, was covered only through a union arm, so a wrapper
that stripped every parameter still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(resources): keep cancellation and reseed cleanup consistent; document the hook contract

- A cancellation that arrives while a failed initialization is being cleaned up now propagates instead of
  being swallowed by the cleanup guard.
- A cookie reseed that released the preceding run moves its finished receipt into the retry window, so a
  failed replacement no longer leaves a pending-cleanup record with nothing to release.
- Document that a disconnect leaves a verification release pending, that the release hook runs under the
  session lock, that state is per worker with session routing keeping a session on its owner, and the
  override rules for servers that keep their own seed and close.
- Rename the example server's private episode table so it no longer shares a name with the base's.
- Add regressions for the receipt deadline guard, worker ownership wiring, cookie seed races and fences,
  typed seed takeover, restore refusals and a cancelled close waiting on verification.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(openenv): cover a retained zero reward and a partly reset reseed; correct the close wording

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* docs(openenv): drop the /run seed explanation

The docs now describe only the typed Environment Server path. The cookie
seed still works unchanged; the PR description records that compatibility.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

---------

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tayunvidia pushed a commit to tayunvidia/Gym that referenced this pull request Nov 13, 2026
…Mo#5485)

* feat(newton_bench): serve the typed resources sessions API

An Environment Server such as single_agent_turn seeds a resources server
with a typed ResourcesSeedSessionRequest and releases it with
/close_session. NewtonBench's seed took only its legacy body, so a typed
seed failed validation with 422, and there was no close.

The server now:
- accepts the typed seed next to the legacy body, reads module_name,
  difficulty, system, noise_level and law_version from task_data, keys
  the session by resources_session_id and binds the cookie to it;
  task_data that fails validation or names an unknown module gets 422;
- keeps the session, including its Python worker, when a seed is
  repeated for the same episode, and refuses another episode with 409;
- closes by resources_session_id, with or without the cookie, through
  the existing _close_session (worker process and metadata); closing a
  session that verify, /end_session or the idle sweep already released
  succeeds, and ResourcesSessionFence refuses a late seed with 409;
- answers a tool call for a session it no longer holds with
  ResourcesSessionLostError (404) instead of a 400, and a verify for it
  with a masked sample instead of a 400; a session the idle sweep reaped
  after session_ttl reports session_expired rather than session_lost.

/verify, /end_session and the idle sweep still release sessions as
before, so the legacy /run path is unchanged. A new
newton_bench_single_agent_turn.yaml runs the rows through
single_agent_turn_legacy with Simple Agent's direct HTTP grant, and the
README describes it and the session lifecycle.

Refs GYMINT-178

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): preserve typed state until session close

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): remember a swept session as expired for as long as session_ttl

The idle sweep's expiry records lasted only session_close_retry_window_seconds (300 s), while the sweep runs every 600 s with a 1800 s TTL. A client back from a stall more than 5 minutes after the sweep, the case the record exists for, got session_lost instead of the session_expired the README promises. The records now last the longer of session_ttl and the close retry window.

The README also says that a retried typed verify scores the law again. Adds a test that moves the clock past the close retry window (fails on the previous code) and moves the conformance import to nemo_gym.server_runtime.testing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): release verified and retired session state

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(newton_bench): name verify retry test after metadata cleanup

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): accept a /run row that carries its own task_id

The /run seed guard refused any body sharing a key with ResourcesSeedSessionRequest, so a dataset row with a scalar task_id got 422 on every seed. Treat a body as a malformed typed seed only when it names resources_session_id or carries typed episode_id and task_id objects, the rule NVIDIA-NeMo#5480 settled on.

Also import check_resources_session_contract from nemo_gym.server_runtime, which main's public-API test requires; name the session agent in the README's single-agent-turn command (--agent takes an agent, not the Environment Server); and state verify's no-await invariant next to its retry bookkeeping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): keep a reused /run cookie's session ID

A /run seed that reused a verified cookie minted a new session ID so the previous verify's retry record stayed valid. Like NVIDIA-NeMo#5480 and NVIDIA-NeMo#5483, a cookie seed now keeps its ID and replaces the previous run's state. It also drops that run's verify retry record, so the new run's tool calls and verify are not answered from the finished run.

Add ClosedResourcesSessionRecords.forget() for that, with a unit test; the same change is in each of the four session PRs (newton_bench, workplace_assistant, tavily_search, indirect_prompt_injection), so whichever merges first carries it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(newton_bench): adopt shared resources lifecycle hooks

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): avoid inheriting server sockets in Python workers

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(server_runtime): close gaps in the shared resources session lifecycle

- The repo contract test counts a server that implements the state hooks as
  closing through the inherited close_resources_session. newton_bench,
  workplace_assistant, tavily_search and indirect_prompt_injection override
  seed_session but close through the hooks, and failed it.
- Export reject_partial_typed_seed from nemo_gym.server_runtime, so servers
  that override seed_session reuse the shared guard instead of a copy.
- The MCP seed wrapper still recognises a request param annotated as
  Annotated[Request, ...]; keeping Annotated metadata had made it inject a
  second Request and fail with KeyError.
- Name the 409 for a typed seed over a cookie session "cookie-seeded", not
  "legacy", and document that a served seed_session override declares
  request before body.
- Shared tests now cover the MCP-wrapped seed route, close after verify
  (checked against the verified episode), and a cookie seed of a verified
  typed session.
- The add-benchmark skill describes the three state hooks as the default
  for a stateful server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(server_runtime): retire a hook session without fencing it

The inherited retire_session_state recorded a typed session as closed. The
checkpoint participant also calls that hook to undo a restore
(delete_restored) and to drop an unclaimed session, and both promise to
refuse nothing later; with the fence, a restore that was rolled back and
installed again in the same process left its session fenced, so its verify
was masked as lost and its continuation's seed got 409. The participant
already refuses a retired attempt's requests until it forgets the rollout,
so retirement now releases the state and drops the identity without
recording a close. A close still fences.

Also write down that an exported server using the state hooks carries each
session's identity through its checkpoint itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): start workers from a forkserver and keep end_session from dropping the session

Spawned Python workers each re-imported the server: about 2 s and 230 MB per
worker on a laptop, every session's first call paid it inside the
max_execution_time + 5 second response wait, and a spawned worker's
interpreter teardown made close() block the event loop about 250-390 ms.
Workers now fork from a forkserver that preloads the server (__main__ when it
runs as app.py) and the sandbox module once. Workers still hold none of the
server's sockets; a later worker starts and answers in a few milliseconds
with 17 MB RSS, and close() takes about 1 ms. Only the first start in a
process waits for the forkserver's import, so execute_python now starts the
worker in a thread, one start per session at a time, and stops a worker
whose session verify or the idle sweep released while it started.

A model can call /end_session as a tool. It released the task metadata, so
the Environment Server's verify was masked session_lost and the model's own
action dropped its sample from the score. /end_session now stops only the
Python worker; the metadata stays until verify, close, checkpoint retirement
or the idle sweep. The route is also no longer offered as an MCP tool.

Also:
- Use the shared reject_partial_typed_seed guard instead of a private copy.
- A verify after close of a session the idle sweep reaped reports
  session_expired, as the README says, instead of session_lost.
- README: worker start cost and the /end_session behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): mask a judge outage instead of scoring it as not equivalent

NewtonBench's symbolic judge catches every API error and, after three failed
attempts, returns False: the same value as a "not equivalent" verdict. A
missing judge key or a sustained 429 therefore set R_symbolic to -1, so an
exact law with RMSLE 0 scored 0.4 instead of 1.0.

- Refuse to start when NewtonBench cannot resolve an API key for its judge
  model, using NewtonBench's own key lookup (environment and its .env file).
- Record each judge API attempt during verify. When the last attempt raised,
  the judge gave no verdict, so verify raises a JudgeError and judge_failsafe
  masks the sample (mask_sample, failure_kind judge_failed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(newton_bench): give the judge a placeholder key and keep it off the network

The server now refuses to start without a judge key, so tests that build the
app with setup_webserver need one. The placeholder comes with a requests.post
that fails at once, so a test that reaches the judge never calls a real API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(mcp): preserve annotated seed validators and request injection

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* feat(sessions): centralize resources lifecycle and verification ownership

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(sessions): preserve cleanup ownership and shared verification semantics

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(resources): preserve grades and retry cleanup ownership

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): reject tools after verification cleanup fails

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* test(mcp): keep a validator annotated directly on the seed body

The wrapper strips Annotated only to recognize an injected Request. A
validator written directly on the body parameter, the form an author is
most likely to write, was covered only through a union arm, so a wrapper
that stripped every parameter still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(resources): keep cancellation and reseed cleanup consistent; document the hook contract

- A cancellation that arrives while a failed initialization is being cleaned up now propagates instead of
  being swallowed by the cleanup guard.
- A cookie reseed that released the preceding run moves its finished receipt into the retry window, so a
  failed replacement no longer leaves a pending-cleanup record with nothing to release.
- Document that a disconnect leaves a verification release pending, that the release hook runs under the
  session lock, that state is per worker with session routing keeping a session on its owner, and the
  override rules for servers that keep their own seed and close.
- Rename the example server's private episode table so it no longer shares a name with the base's.
- Add regressions for the receipt deadline guard, worker ownership wiring, cookie seed races and fences,
  typed seed takeover, restore refusals and a cancelled close waiting on verification.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(newton_bench): serialize exec on a worker pipe and keep a lost session a 404

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* docs(newton_bench): drop the /run seed explanation

The docs now describe only the typed Environment Server path. The cookie
seed still works unchanged; the PR description records that compatibility.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* refactor(newton_bench): make the session config the default

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* fix(ci): scope release rehearsal changelogs to PR changes

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

---------

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — d53e4ac4 Deployed Nov 9, 2026 by copy-pr-bot[bot] via release / finalize / notify #11196
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants