Skip to content

feat(environment): sign episode tokens for Environment Server tool grants - #5346

Merged
ffrujeri merged 2 commits into
mainfrom
ffrujeri/episode-tokens
Nov 16, 2026
Merged

ffrujeri merged 2 commits into
mainfrom
ffrujeri/episode-tokens

Conversation

@ffrujeri

@ffrujeri ffrujeri commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What changed

nemo_gym/server_runtime/episode_tokens.py adds EpisodeTokens, which mints and reads the credential an Environment Server grants for one episode's tools:

  • mint(episode_id) returns a signed token and the EpisodeTokenClaims (episode ID and a random nonce) that the server keeps.
  • read(token) returns the claims of a token this server signed, or None for anything else, including an MCP session token signed with the same key.
  • EpisodeTokenClaims.matches compares the episode and the nonce in constant time.
  • EPISODE_TOKEN_HEADER names the header that carries the token on a direct HTTP tool call.

Why

This is the first PR of a stack that lets an Environment Server own an episode's state and tools without a Resources Server (Linear GYMINT-641, under the design in the user-simulation RFC's layer 1, BaseEnvironmentServerWithTools). The tool calls arrive as separate requests, so they need a credential for the episode:

  • A rollout ID is predictable: <task index>-<rollout index> unless the caller sets one, and the agent sees its own in its capture path. A bare ID would let a policy call another episode's tools, for example to read hidden knowledge without paying for it.
  • A Gym server's signing key is derived from its class and instance names (SimpleServer.get_session_middleware_key), so a signature alone is forgeable.

The nonce makes the token unguessable; the server accepts it only while it holds the same claims. The signature keeps the claims intact, and lets a worker that does not hold the episode read which episode a token names, so a later PR can route the call to the worker that does.

Nothing uses the module yet; the next PR in the stack, the Environment Server base class that serves episode tools, does.

Validation

Command Result
pytest tests/unit_tests/server_runtime/test_episode_tokens.py tests/unit_tests/test_import_linter_coverage.py (after the rebase onto main at dda609f86) 15 passed. The module is now nemo_gym/server_runtime/episode_tokens.py, next to the other server runtime modules
pytest tests/unit_tests/server_runtime/test_episode_tokens.py 11 passed: round trip with an attempt, a new nonce for each grant, another episode's token, a forged token with a guessed nonce, a tampered token, and six strings this server did not mint (another key, an MCP token with the same key, a missing nonce, an invalid episode, a payload that is not an object, garbage)
pre-commit run --files nemo_gym/server_runtime/episode_tokens.py tests/unit_tests/server_runtime/test_episode_tokens.py passed

Rollout evidence

N/A: a library module that no server uses yet. The BIRD-Interact migration onto the Environment Server base class will run rollouts.

Compatibility

New module only. No existing behavior changes.

@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test e050173

@ffrujeri

ffrujeri commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 3dc43be

@ffrujeri

ffrujeri commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test fa0b016

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test bfa9415

@ffrujeri
ffrujeri requested a review from adil-a November 12, 2026 18:04
ffrujeri added a commit that referenced this pull request Nov 12, 2026
…er, and survive a raising header reader

Review of #5350:

- A worker that dies leaves its owner file, so a /run resent after the
  crash found the episode owned and failed for good. serve_tools now
  raises a non-terminal handled episode failure, so the attempt budget
  retries the episode under a new attempt key.
- The router reads the session header outside the exception
  middleware, so a reader that raised became a bare 500. A reader that
  raises now names no session, and the worker the call reached handles
  it.
- The workers of a multi-worker server read each other's episode
  tokens with the secret they inherit from the main process
  (episode_token_secret, #5346).
- The e2e test bounds its wait for the episode's grant and fails at
  once, with the reply, if /run ends first.
- The routing e2e job also runs when episode_tokens.py,
  episode_tools.py, tool_access.py or core/episode_types.py change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
ffrujeri added a commit that referenced this pull request Nov 12, 2026
…er, and survive a raising header reader

Review of #5350:

- A worker that dies leaves its owner file, so a /run resent after the
  crash found the episode owned and failed for good. serve_tools now
  raises a non-terminal handled episode failure, so the attempt budget
  retries the episode under a new attempt key.
- The router reads the session header outside the exception
  middleware, so a reader that raised became a bare 500. A reader that
  raises now names no session, and the worker the call reached handles
  it.
- The workers of a multi-worker server read each other's episode
  tokens with the secret they inherit from the main process
  (episode_token_secret, #5346).
- The e2e test bounds its wait for the episode's grant and fails at
  once, with the reply, if /run ends first.
- The routing e2e job also runs when episode_tokens.py,
  episode_tools.py, tool_access.py or core/episode_types.py change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
ffrujeri added a commit that referenced this pull request Nov 12, 2026
…er, and survive a raising header reader

Review of #5350:

- A worker that dies leaves its owner file, so a /run resent after the
  crash found the episode owned and failed for good. serve_tools now
  raises a non-terminal handled episode failure, so the attempt budget
  retries the episode under a new attempt key.
- The router reads the session header outside the exception
  middleware, so a reader that raised became a bare 500. A reader that
  raises now names no session, and the worker the call reached handles
  it.
- The workers of a multi-worker server read each other's episode
  tokens with the secret they inherit from the main process
  (episode_token_secret, #5346).
- The e2e test bounds its wait for the episode's grant and fails at
  once, with the reply, if /run ends first.
- The routing e2e job also runs when episode_tokens.py,
  episode_tools.py, tool_access.py or core/episode_types.py change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
ffrujeri added a commit that referenced this pull request Nov 12, 2026
…m with a per-run secret

Review of #5346:

- read() caught only BadSignature, so a validly signed payload that
  does not decode raised BadPayload. It now catches BadData and never
  raises.
- A non-ASCII nonce made hmac.compare_digest raise TypeError in
  matches(). read() now refuses it.
- The signing key was the server's derivable session key. Tokens now
  use a random secret for each server run (episode_token_secret). The
  main process sets it in NEMO_GYM_EPISODE_TOKEN_SECRET before uvicorn
  starts the workers, so every worker reads the tokens another worker
  minted. The nonce still authenticates each call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri
ffrujeri force-pushed the ffrujeri/episode-tokens branch from bfa9415 to c1a9e4c Compare November 12, 2026 20:09
ffrujeri added a commit that referenced this pull request Nov 12, 2026
…er, and survive a raising header reader

Review of #5350:

- A worker that dies leaves its owner file, so a /run resent after the
  crash found the episode owned and failed for good. serve_tools now
  raises a non-terminal handled episode failure, so the attempt budget
  retries the episode under a new attempt key.
- The router reads the session header outside the exception
  middleware, so a reader that raised became a bare 500. A reader that
  raises now names no session, and the worker the call reached handles
  it.
- The workers of a multi-worker server read each other's episode
  tokens with the secret they inherit from the main process
  (episode_token_secret, #5346).
- The e2e test bounds its wait for the episode's grant and fails at
  once, with the reply, if /run ends first.
- The routing e2e job also runs when episode_tokens.py,
  episode_tools.py, tool_access.py or core/episode_types.py change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test c1a9e4c

ffrujeri added a commit that referenced this pull request Nov 13, 2026
…m with a per-run secret

Review of #5346:

- read() caught only BadSignature, so a validly signed payload that
  does not decode raised BadPayload. It now catches BadData and never
  raises.
- A non-ASCII nonce made hmac.compare_digest raise TypeError in
  matches(). read() now refuses it.
- The signing key was the server's derivable session key. Tokens now
  use a random secret for each server run (episode_token_secret). The
  main process sets it in NEMO_GYM_EPISODE_TOKEN_SECRET before uvicorn
  starts the workers, so every worker reads the tokens another worker
  minted. The nonce still authenticates each call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri
ffrujeri force-pushed the ffrujeri/episode-tokens branch from c1a9e4c to 19feb7c Compare November 13, 2026 18:11
ffrujeri added a commit that referenced this pull request Nov 13, 2026
…er, and survive a raising header reader

Review of #5350:

- A worker that dies leaves its owner file, so a /run resent after the
  crash found the episode owned and failed for good. serve_tools now
  raises a non-terminal handled episode failure, so the attempt budget
  retries the episode under a new attempt key.
- The router reads the session header outside the exception
  middleware, so a reader that raised became a bare 500. A reader that
  raises now names no session, and the worker the call reached handles
  it.
- The workers of a multi-worker server read each other's episode
  tokens with the secret they inherit from the main process
  (episode_token_secret, #5346).
- The e2e test bounds its wait for the episode's grant and fails at
  once, with the reply, if /run ends first.
- The routing e2e job also runs when episode_tokens.py,
  episode_tools.py, tool_access.py or core/episode_types.py change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 19feb7c

@ffrujeri
ffrujeri requested a review from a team November 16, 2026 16:41
ffrujeri added a commit that referenced this pull request Nov 16, 2026
…m with a per-run secret

Review of #5346:

- read() caught only BadSignature, so a validly signed payload that
  does not decode raised BadPayload. It now catches BadData and never
  raises.
- A non-ASCII nonce made hmac.compare_digest raise TypeError in
  matches(). read() now refuses it.
- The signing key was the server's derivable session key. Tokens now
  use a random secret for each server run (episode_token_secret). The
  main process sets it in NEMO_GYM_EPISODE_TOKEN_SECRET before uvicorn
  starts the workers, so every worker reads the tokens another worker
  minted. The nonce still authenticates each call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri
ffrujeri force-pushed the ffrujeri/episode-tokens branch from 19feb7c to 73bc632 Compare November 16, 2026 17:08
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 73bc632

ffrujeri and others added 2 commits November 16, 2026 17:56
…ants

An Environment Server that serves an episode's tools needs a credential
that names one episode and cannot be guessed. A rollout ID is
predictable, and a Gym server's signing key is derived from its class
and instance names, so neither is enough. EpisodeTokens signs the
episode's capture key with a random nonce; the server accepts a token
only while it holds the same claims for that episode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…m with a per-run secret

Review of #5346:

- read() caught only BadSignature, so a validly signed payload that
  does not decode raised BadPayload. It now catches BadData and never
  raises.
- A non-ASCII nonce made hmac.compare_digest raise TypeError in
  matches(). read() now refuses it.
- The signing key was the server's derivable session key. Tokens now
  use a random secret for each server run (episode_token_secret). The
  main process sets it in NEMO_GYM_EPISODE_TOKEN_SECRET before uvicorn
  starts the workers, so every worker reads the tokens another worker
  minted. The nonce still authenticates each call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 96d85a3

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43936963334

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43943698668

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants