Skip to content

feat(agent_arena): serve AgentArena's tools from a BaseEnvironmentServerWithTools - #6267

Open
weijiac0619 wants to merge 15 commits into
mainfrom
weijiac/agent-arena-env-tools
Open

weijiac0619 wants to merge 15 commits into
mainfrom
weijiac/agent-arena-env-tools

Conversation

@weijiac0619

@weijiac0619 weijiac0619 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Built on #5917 (merged), which makes sandbox_provider required for BaseEnvironmentServerWithTools.

AgentArena (benchmarks/agent_arena/config.yaml) runs agentic_turn_simulation_agent behind the deprecated
legacy_agent relay. #6103 moved it onto a dedicated Environment Server that served the turn's tools through its own
routes (/harness, /simulate_tool_call) behind a per-episode secret, with one worker only. That was hand-rolled
while #5346 to #5350 were open. This PR builds the same server on BaseEnvironmentServerWithTools instead, as
suggested in the review: signed episode tokens, serve_tools, the shutdown sweep and the routing of tool calls to the
worker that runs the episode come from the base class. #6103 stays open until this one is accepted.

No issue is linked: the work is tracked internally.

AgentArenaEnvironmentServer subclasses BaseEnvironmentServerWithTools. run() for one row:

  1. Builds the turn's session from the whole row: harness, workspace replay, simulator prompt, obstacle.
  2. Serves the row's tools with serve_tools(request, TurnToolset(...), cleanup). TurnToolset is an
    EpisodeToolset:
    • specs() lists the tools a call may name, the same set the scoring's nonexistent_tool gate accepts: the row's
      tools, or Cline's tool names for a Cline row (which declares none). The row declares its tools to the policy
      itself, so a spec only names a tool the server answers.
    • call(name, arguments) answers a call as the resources server's /simulate_tool_call does: in code from the
      replayed workspace, else by the tool simulator LLM, or with the row's obstacle.
      The tools are POST /<tool name> on this server, reached with the grant's episode token.
  3. Seeds an agent session with the grant, and calls the agent's /v1/responses with the row's
    responses_create_params. Their metadata name the row's stop tools and tool-call protocol
    (agentic_turn_simulation_stop_tools, agentic_turn_simulation_protocol), which /harness used to return: the
    base's token middleware lets a token reach only tool paths, so the server cannot serve /harness to the agent. The
    agent takes both keys out before it calls the model.
  4. Closes the agent session and scores the turn in process: masking, code gates, judges. The result has the same
    fields as the resources server's /verify response, and /aggregate_metrics computes the same metrics.
  5. Stops serving the tools in the episode's cleanup.

No Resources Server takes part in the episode.

What the simulator sees (open question)

EpisodeToolset.call receives a tool's name and arguments only. On the relay the agent sent the turn so far with each
call, so the simulator saw what the policy wrote between its calls and the call's place in its reply. Here the server
rebuilds that view from the episode's earlier calls and their results (ServedTools.calls), after the row's history:
TurnToolset.turn_so_far is the one place that decides it. It keeps only results the policy saw: records the server
wrote for a call it refused or that failed ({"detail": ...}) are left out, since the agent answers a refusal with its
own text and retries a failure under the same call id, and of a call id's records only the last is kept. Two
consequences:

  • The simulator no longer sees the policy's text between calls (its "Let me check the build" message, or the text
    around a Cline XML call).
  • For a Cline reply with several calls, the server cannot tell that more results follow, so it keeps the
    <environment_details> on each result. The agent strips them from all but the last result when it joins them into
    Cline's user message, as the server did before.

Should the base pass more per-call context to a toolset, such as the call id or the agent's turn so far? If it does,
turn_so_far is the piece to replace. Under Validation, neither a fixed-policy replay (51/54 rows judged the same,
against 53/54 for two runs of the relay's own context) nor a paired run of the whole validation set (reward
difference +0.001, standard error 0.011) finds a measurable effect.

Changes

  • environment_servers/agent_arena (new): the server (app.py), TurnToolset (toolset.py), config, README,
    tests.
    • Its simulator, judge and scoring settings are read from the resources server its config names
      (resources_server). Overrides of those settings keep working, and gym eval reverify through that server scores
      a saved row with the same settings. The episode makes no call to it.
    • sandbox_provider: {local: {}}. No operation runs in a sandbox: a call is answered from the model of the agent's
      files or by a model call. The server logs the base's warning about local at startup.
    • It answers a benchmark's flat row with a flat result and a typed episode request with the episode reply, as
      single_agent_turn_legacy does. /run takes both shapes and stays ahead of the base's tool route.
    • Checkpointing is restart-only (the base default). A failed agent reply goes through agent_outcome_policy; a judge
      that cannot be reached is a retryable verification failure.
    • A time-capped or simulator-failed turn saves the decision of the resources server's outcome_policy (fix(envs): declare retry for environment-side faults they mask #6072), as
      the resources server's /verify does.
    • In a run that sets model_call_sampling, the server's own model calls (the tool simulator and the judges) send
      the null sampling segment, as a resources server's calls do (CallerSamplingMiddleware). They keep their own
      sampling, and a trained agent's Model Server, which refuses a generation call without a segment, takes them when
      the simulator or a judge shares it. Only the agent's calls carry the training framework's values.
  • resources_servers/agentic_turn_simulation: session setup, the tool simulator, the judge, scoring and metrics
    move out of app.py into schemas.py and turns.py, which both servers use. The code is moved as it was. app.py
    keeps the resources server as a thin wrapper; it still serves the relay and gym eval reverify. schemas.py also
    defines the two metadata keys. The outcome_policy setting that main added to app.py (fix(envs): declare retry for environment-side faults they mask #6072) is in the shared
    settings in schemas.py, and the decision a masked row saves is in turns.py.
  • responses_api_agents/agentic_turn_simulation_agent: serves agent sessions.
    • The row's harness comes from the request's metadata. A session request without it fails the turn; it never falls
      back to the config's stop tools and native calls.
    • Each tool call goes to POST /<tool name> through the grant (under the activation's rollout prefix): the
      arguments object is the body, the call id is in X-NeMo-Gym-Tool-Call-Id, with the grant's headers and the
      session's cookies. Error replies are retried, then become the simulator-failure output that scoring masks, as on
      the relay.
    • A reply that refuses the grant itself fails the turn, unretried: no valid episode token (401), a token for another
      episode (403), or an episode no longer running (410). That is a setup or protocol fault; masked as a simulator
      failure it would show only as a high failed rate.
    • A call the server cannot take is answered by the agent, unretried: arguments that are not a JSON object, a name
      no grant can carry, or a tool the row lacks (the server's 404). Scoring gives such a turn 0 whatever the call
      returned (malformed_arguments, nonexistent_tool); on the relay the simulator answered these calls.
    • A seed without a direct HTTP grant is refused (422). /run still calls the configured resources server's
      /harness and /simulate_tool_call, unchanged.
    • The session's close returns the turn's trajectory as /run returns it, under the seeded task. The session's
      observations keep no tool output, and the rollout's trajectory takes each output from the conversation's
      function_call_output items, which a Cline turn does not have (its results are in a user message).
  • nemo_gym/config/global_config.py: agentic_turn_simulation_agent joins AGENT_SESSION_AGENT_TYPES.
  • Configs:
    • benchmarks/agent_arena/config.yaml and the environment's own configs/agentic_turn_simulation.yaml use the
      agent_arena Environment Server.
    • benchmarks/agent_arena/config_legacy_agent.yaml and configs/agentic_turn_simulation_legacy_agent.yaml keep the
      relay, with a deprecation note.
  • READMEs for the three components describe the new path.

Compatibility

  • --benchmark agent_arena and gym env start --resources-server agentic_turn_simulation now run the agent_arena
    Environment Server. --benchmark agent_arena/config_legacy_agent and
    configs/agentic_turn_simulation_legacy_agent.yaml run the relay as before.
  • Instance names, the dataset path, num_repeats: 4, prompts, settings and scoring are unchanged. Overrides of the
    resources server's settings apply to the new path too.
  • The environment's own config moved too. A training setup on configs/agentic_turn_simulation.yaml should switch to
    the _legacy_agent config or be checked on the new path first. Training was not run here.
  • No changes to the base classes from feat(environment): sign episode tokens for Environment Server tool grants #5346 to feat(environment): require a sandbox provider for BaseEnvironmentServerWithTools and add start_episode_sandbox #5917; they are used as they are. The one core change is
    agentic_turn_simulation_agent joining AGENT_SESSION_AGENT_TYPES in nemo_gym/config/global_config.py.
  • Failure accounting follows the Environment Server: a turn that ends on a model or agent error becomes a failure row
    (retried within the attempt budget). A time-capped turn is still masked (agentic_turn_simulation:time_cap).
  • Python imports: the route bodies are still importable from the resources server's app.py; JudgeSpec and the
    private helpers now live in schemas.py and turns.py.

Validation

Unit tests

Each directory is a separate pytest run. The first four were run on the current head; the others on this branch before its latest merge with main:

Tests Result
environment_servers/agent_arena/tests 25 passed
resources_servers/agentic_turn_simulation/tests 953 passed
responses_api_agents/agentic_turn_simulation_agent/tests 108 passed
benchmarks/agent_arena/tests 15 passed
environment_servers/single_agent_turn/tests 292 passed
environment_servers/single_agent_turn_legacy/tests 54 passed
environment_servers/legacy_agent/tests 13 passed
responses_api_agents/simple_agent/tests 170 passed
tests/unit_tests (config, environment, tooling, server_runtime, public API, session contract, import-linter coverage) 5062 passed, 1 skipped

Real runs

One checkpoint served by vLLM plays the policy, the tool simulator and the judge. 15 rows: 10 AgentArena
validation rows (sandbox harness) and one row each for sandbox, Claude Code, Hermes, OpenCode and Cline. Each row runs
twice (30 rollouts per run). "New" is --benchmark agent_arena; "relay" is --benchmark agent_arena/config_legacy_agent.
The relay output had one extra rejudged copy of a rollout, left out of the counts below.

End to end:

Run Failures Masked Mean reward (measured) Simulator answers (exact / LLM / failed) Failed rate Same reward, ladder, gate and mask as the relay
New, temperature 1 0 2 0.964 (28) 60 / 150 / 0 0.0% 28 / 30
Relay, temperature 1 0 1 1.000 (29) 39 / 103 / 2 1.4%
New, temperature 0, seed 1234 0 3 0.963 (27) 57 / 70 / 1 0.8% 27 / 30
Relay, temperature 0, seed 1234 0 2 0.966 (28) 30 / 53 / 3 3.5%

Every row ran on both paths; the Cline row ended on attempt_completion with reward 1. The rows that differ are
within run-to-run noise: vLLM at temperature 0 is not deterministic with long reasoning, so the two runs' policies
diverge after a few tokens.

  • Temperature 0: two rows whose calls were all answered in code (no simulator LLM call), one flipping each way, and
    one row masked by the turn's time cap (agentic_turn_simulation:time_cap) on the new path.
  • Temperature 1: the same code-answered row, and a time-capped row.

Fixed policy, call by call. To separate the simulator's context from the policy's variance, each recorded rollout
from the four runs above (54 sandbox rollouts with native calls; 285 calls) was replayed against the simulator with
its recorded calls and outputs held fixed:

  • A: each call with the turn the agent sent on the relay (the policy's text between calls included), run twice
    (A and A').
  • B: each call through TurnToolset.call with the earlier calls as ServedTools.calls would hold them (no text
    between calls).

Each variant's outputs then replace the recorded ones and the turn is judged again.

A vs A' (noise floor) A vs B
Same reward, ladder, gate and mask 53 / 54 51 / 54
Same reward 53 / 54 51 / 54
Median text similarity of LLM-answered outputs (184 calls) 0.541 0.528
Variant Exact LLM Failed Mean reward
A 96 184 5 (1.8%) 0.963
A' 0.944
B 96 186 3 (1.1%) 0.907

Of the three rows where A and B disagree:

  • Two are the same task, both masked under B because one simulator call failed there. A had more failed calls in
    total; they fell on other turns.
  • One is a judge flip (1 → 0) on a task whose other rollout flips the same way between A and A'.

The simulator's text at temperature 0 is about as different between two runs of A as between A and B. The failed rate
does not rise without the text between calls. On these rows the loss of that text is not measurable.

Whole validation set, paired. All 433 AgentArena validation rows (sandbox harness), two rollouts each, on both paths:
the same checkpoint as policy, simulator and judge, one vLLM instance per path, temperature 1.0, top_p 0.95, 32768
output tokens, a 1200 s turn time cap. The relay output had one extra rejudged copy of a rollout, left out.

Path Rollouts Failures Masked (time cap) Mean reward (measured) Simulator answers (exact / LLM / failed) Failed rate Median rollout time
New 866 0 121 0.952 ± 0.008 (745) 2033 / 1839 / 7 0.18% 154 s
Relay 866 0 115 0.949 ± 0.008 (751) 2022 / 1860 / 5 0.13% 138 s

Paired by task (each task's mean over its rollouts on each path, new minus relay):

Difference Standard error 95% CI Tasks
Reward (measured rollouts) +0.0013 0.0109 [−0.020, +0.023] 377
Time-capped (masked) +0.0069 0.0107 [−0.014, +0.028] 433

Six tasks flip fully (both rollouts), four toward the new path and two toward the relay; 62 tasks differ in how many
rollouts were time-capped, 32 more on the new path and 30 more on the relay. An audit of every new-path rollout found
each tool call with its output and no harness metadata in a model request. One call named a tool the row lacks
(ffprobe) and was refused by the agent, as described above. The validation set has no Cline rows; the Cline
trajectory is covered by the agent's tests.

Checks

pre-commit run --files <changed files> passes (ruff, ruff-format, the README environment list, import-linter).

Known limits

  • Simulator context. See "What the simulator sees". The paired validation run bounds the reward difference to
    about ±0.02.
  • Captured model calls. The simulator and judge calls the server makes in process run inside the rollout, so the
    rollout's model-call capture holds them next to the policy's, with the gap model_call_ownership_unavailable, as
    BIRD-Interact's in-process user simulator does. On the relay they ran in the resources server and were not
    captured. The trajectory's turns come from the agent's own trajectory, so they are the policy's alone.
  • Time caps. A time-capped turn is usually one simulator call whose reasoning runs to the output limit (32k tokens
    and more, minutes each); the median simulator call takes about 10 s. Both paths hit the cap at the same rate (see
    the paired run).
  • Data scan. Every row's tools must be valid ToolSpecs (name pattern, object parameters) and must not take a
    route's name (run, aggregate_metrics, docs, openapi.json, redoc). A row that breaks this fails at setup.
    All AgentArena and generic RL rows that were checked pass: 433 validation rows and 1500 training rows (sandbox), and
    2089 generic training rows (OpenClaw 298, Hermes 633, OpenCode 639, Claude Code 515, Kilo Code 4), plus the
    example rows. No row had an invalid tool name, non-object parameters or a reserved name.
  • Crash recovery. With rollout checkpointing on, a crash restarts the episode from its input.
  • Re-verification. gym eval reverify re-scores through the resources server's /verify, which shares
    turns.py and the settings.
  • Shared agent-session code. The seed, invoke, close and failure handling follow single_agent_turn's, written out
    here rather than shared.
  • MCP. tool_transports stays [direct_http], the transport agentic_turn_simulation_agent takes. Row tool names
    are not checked against MCP's stricter name rule.
  • Training. The new path was not run for training (token-ID capture through agent sessions). The environment's
    own config (configs/agentic_turn_simulation.yaml) now runs it; configs/agentic_turn_simulation_legacy_agent.yaml
    keeps the relay. Open for the environment's training owners: the simulator's and judges' calls run in the rollout's
    context, so a rollout's model-call capture holds them (see "Captured model calls"); whether a training setup reads
    tokens from that capture, and so whether a simulator or judge served by the policy's own Model Server would reach
    training data, was not checked.

🤖 Generated with Claude Code

ffrujeri and others added 8 commits November 16, 2026 12:08
…erWithTools and add start_episode_sandbox

An episode of a BaseEnvironmentServerWithTools has a control plane, the
server, which decides what a tool call runs, and a data plane, the
episode's sandbox, which runs every operation on the episode's world.
The control plane never runs such an operation in its own process.

- BaseEnvironmentServerWithToolsConfig.sandbox_provider is required, so
  every environment says where its data plane runs. A provider that the
  config does not define fails at startup. The local provider runs
  commands as host subprocesses, so the server logs a warning at startup.
- start_episode_sandbox(request, cleanup, spec, setup) starts a sandbox
  for the episode. It requires ttl_s, labels the sandbox with the
  server's name and the episode's rollout over the provider's and the
  spec's metadata, and registers the stop in the episode's cleanup
  before it starts the sandbox, so the cleanup or the shutdown sweep
  also stops a sandbox whose start or first stop failed. A failed setup
  stops the sandbox.
- The class docstring states the control plane and data plane rule.

A runtime check that refuses process creation and sqlite3 in the
control plane is left for a follow-up (GYMINT-733).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…t Server example

BaseEnvironmentServerWithToolsConfig now requires sandbox_provider, so
the example would fail validation without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…up handle

start_episode_sandbox stopped such a sandbox with a direct call, which
the cleanup timeout did not bound and the episode's cleanup repeated. It
now closes the registered handle: the stop gets the cleanup timeout and
the entry's lock, a stop that succeeded is not run again, and one that
failed or timed out stays for the episode's cleanup to retry.

The docstring now names the providers that ignore ttl_s (Docker with a
custom entrypoint, Apptainer, Enroot and local).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…oring into shared modules

Session setup, the tool simulator, the judge, scoring and metrics move out of the resources server's app.py into
turns.py (TurnSimulation, TurnSession) and schemas.py (settings and route bodies), so an Environment Server can run
the same turns. The code moves as it was; verify takes the session as an argument. The resources server becomes a
thin wrapper and keeps serving the legacy relay and gym eval reverify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
…n through its direct HTTP grant

In an agent session the agent sends /harness and /simulate_tool_call to the grant's URL, under the activation's
rollout prefix, with the grant's headers and the session's cookies; any /harness error through the grant fails the
turn. A seed without a direct HTTP grant is refused. /run still calls the configured resources server. The agent
joins AGENT_SESSION_AGENT_TYPES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
… an agent session

In an agent session the row's tools are served by an Environment Server: each call goes
to POST /<tool name> through the session's direct HTTP grant, with the arguments object as
the body and the call id in X-NeMo-Gym-Tool-Call-Id. The row's stop tools and tool-call
protocol come in the request's metadata (agentic_turn_simulation_stop_tools,
agentic_turn_simulation_protocol), which the agent takes out before it calls the model.

A call the server cannot take (arguments that are not a JSON object, a name no grant can
carry, a tool the row lacks) is answered by the agent without a retry; scoring gates such a
turn. For Cline replies with several calls, only the last result keeps the environment
details. The legacy /run path is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
…verWithTools

environment_servers/agent_arena runs each AgentArena row as one episode. It builds the turn
from the whole row, serves the row's tools with serve_tools (TurnToolset, an EpisodeToolset
whose calls are answered as /simulate_tool_call answers them), seeds and invokes
agentic_turn_simulation_agent with the row's prompt and harness metadata, and scores the
turn in process with the resources server's settings.

The simulator's view of the turn is rebuilt from the episode's earlier calls and their
results (TurnToolset.turn_so_far). sandbox_provider is local: no episode starts a sandbox.
A flat benchmark row gets a flat result. The benchmark and the environment's own config use
the new server; their _legacy_agent twins keep the relay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
#5917 merged into main. Keep both new entries in AGENT_SESSION_AGENT_TYPES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…n agent session's close

The session's observations keep no tool output, and a rollout's trajectory takes each call's output from the
conversation's function_call_output items. A Cline turn has none (its results are in a user message), so its tool
calls had no output. The close now returns the trajectory the agent recorded, as /run returns it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Nov 17, 2026
weijiac0619 and others added 3 commits November 16, 2026 21:07
… trajectory

simple_agent's session turn records its trajectory under task "unscoped"; `/run` stamps the row's task on it, a
session did not, so every rollout's trajectory carried producer_trajectory_identity_mismatch. Stamp the seed's
task id on the trajectory and its turns, as the OpenCode agent does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
The conflict was in resources_servers/agentic_turn_simulation/app.py: this branch moved the server's settings and
scoring into schemas.py and turns.py, and main (#6072) added an outcome policy to them. The policy (TIME_CAP_KIND,
SIMULATOR_FAILED_KIND, the outcome_policy setting and its validators) now lives in AgenticTurnSimulationSettings, and
the saved decision of a masked row in TurnSimulation._mask_reason, so the agent_arena environment server saves the
same decision as the resources server. agent_arena's tests check it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
I, weijia chen <weijiac@nvidia.com>, hereby add my Signed-off-by to this commit: 62631a2
I, weijia chen <weijiac@nvidia.com>, hereby add my Signed-off-by to this commit: c2b750c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
@weijiac0619

Copy link
Copy Markdown
Contributor Author

/ok to test 8e0aa63

@weijiac0619
weijiac0619 marked this pull request as ready for review November 17, 2026 23:41

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

I found six RISKs in the migration. Supported controls are silently ignored, and reconstructed tool history changes opt-in obstacle behavior.

I reviewed all 27 changed files and traced the affected runtime contracts. I did not execute tests or rollouts; the validation numbers in the PR description are author-reported.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread environment_servers/agent_arena/app.py Outdated
self._verify_body(request, response) | verification_failure_fields(response)
)
try:
verified = await self._turns.verify(body, session)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Preserve skip_verification when moving off the relay

With ++skip_verification=true ++skip_verification_reward=0.25, this path still runs scoring and returns its reward. Config parsing places those controls on the agent, but this server always calls self._turns.verify. The previous SimpleAgent /run path returned the fixed reward without verification. Generation-only runs now depend on judge availability and can receive a different reward.

Please resolve the bound agent's controls before scoring, after agent-outcome handling. Add a helper using the typed parsing in the sibling implementation. When skipping is enabled, construct the result with:

SingleAgentTurnResult.model_validate(
    self._verify_body(request, response)
    | {"reward": skipped_reward, "verification_skipped": True}
)

Keep the existing attachment of observations and trajectory. Apply the same decision in aggregate_metrics: skip benchmark scoring metrics while still adding performance metrics, as the sibling server does. The invariant is that an explicit verification skip neither scores the turn nor reports its synthetic reward as measured turn_score.

Proposed checks, not executed: use agent skip_verification=true, reward 0.25, and a judge stub that fails if called. Expect the fixed reward, verification_skipped=true, no scoring call and normal cleanup. Retain judged rewards and benchmark metrics when skipping is false. Cover per-agent false overriding global true, string "false" parsing as false, and fixed reward 0.0. Verify the suggested construction before adopting it.

return AgenticTurnSimulationSettings.model_validate({k: v for k, v in raw.items() if k in fields})

def setup_webserver(self) -> FastAPI:
app = super().setup_webserver()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Mark simulator and judge calls as untrained sampling

A training run can set model_call_sampling.required_model_servers: [policy_model] and use that model-server instance for the policy, simulator and judge. The policy activation carries its encoded sampling segment, but simulator/judge calls from this server carry none. They therefore hit the model server's missing-segment 422, causing simulator-failure masks or failed verification. ServerClient only propagates the current segment; neither episode requests nor independent tool requests establish the null context here. The previous resources-server path installs that context.

A small correction in setup_webserver, immediately after super(), is:

if model_call_sampling_config(self.server_client.global_config_dict) is not None:
    app.add_middleware(CallerSamplingMiddleware)

Import model_call_sampling_config from the public core API. Export CallerSamplingMiddleware through the server-runtime public API before importing it here. This covers both /run and separately arriving tool calls. The policy's explicit encoded segment must stay unchanged; simulator/judge calls must carry /ng-sampling/null. Evaluation without sampling configuration must keep its current URLs.

Proposed checks, not executed: exercise an episode and granted tool route with one required model-server instance serving all roles. Assert encoded episode values on policy calls, null on simulator/judge calls, and no missing-segment rejection. Retain evaluation URL assertions and token-capture skip behavior. Verify the suggested middleware change before adopting it.

for index, record in enumerate(self._calls):
call_id = record.call_id or f"ng_call_{index}"
output = record.output if isinstance(record.output, str) else json.dumps(record.output, ensure_ascii=False)
items += [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Preserve reply boundaries for obstacle injection

The reconstructed history changes deterministic obstacle semantics, not only the simulator's prose context. Enable obstacle_injection, force a sandbox row's obstacle to transient, and let one native reply contain fetch_page followed by ask_user. The agent executes the fetch but does not send the stop call through the grant. turn_so_far therefore shows only the fetch, so an error is injected. The contract says a reply containing a stop tool must not meet its first obstacle because the policy cannot react. Final scoring sees the complete reply and reports obstacle.fired=false, contradicting the injected result. See the obstacle decision and the existing stop-tool regression.

Two identical calls in one reply also change behavior: interleaving every recorded call with its result makes the second look like a later reply. A transient obstacle then answers it normally, although both sibling calls should receive the same error. The existing test explicitly protects that distinction.

Please carry reply-batch context from the agent's _simulate_calls into the episode's simulation context, including unexecuted stop calls and the current call's position. Rebuild grouped calls followed by their results before evaluating obstacles, consistently with final scoring. EpisodeToolset.call currently has no batch context, so I cannot give a safe local replacement for this loop. A bounded fallback is to reject enabled obstacle injection on this path with guidance to use the relay until that context is available. Do not change the shared obstacle rules to match the lossy history.

Proposed checks, not executed: a fetch beside a stop tool receives a normal result and reports fired=false; duplicate sibling fetches both receive the transient error; the same fetch in the next model reply succeeds. Preserve persistent blocked_route failures, including a later blocked fetch beside a stop tool after an earlier failure. Also retain normal tool execution with injection disabled.

"""An episode request gets the episode reply; a flat row (a benchmark's) gets a flat result, as
single_agent_turn_legacy answers it."""
is_episode_request = "episode_id" in row and "task" in row
request = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Return HTTP 422 for invalid flat and typed requests

This replacement /run endpoint accepts a dict and validates it inside the handler. A flat row with valid indices and prompt but _ng_skip_token_capture: "yes" raises ValueError; a malformed typed task raises ValidationError. Neither is translated, so the served endpoint returns 500 instead of a terminal 422. Wrong-agent and negative-attempt rows follow the same path. Callers can consequently retry inputs that cannot succeed.

Please apply the narrow translation already used by the sibling flat adapter:

try:
    request = (
        SingleAgentTurnRequest.model_validate(row)
        if is_episode_request
        else self._episode_request_from_row(row)
    )
except ValidationError as error:
    raise RequestValidationError(error.errors(include_url=False)) from None
except ValueError as error:
    raise HTTPException(422, str(error)) from None

Add the corresponding FastAPI imports. Keep run_request outside the catch so runtime failures are not mislabeled as invalid input. _episode_request_from_row can still raise ValueError for direct callers; update the existing run_legacy wrong-agent test to check the HTTP contract.

Proposed checks, not executed: POST malformed capture controls, a mismatched agent_ref, a negative attempt and malformed typed task input. Each should return 422 without downstream calls. Retain successful flat results and typed episode replies on the single /run route, ahead of the tool catch-all. Verify the suggested code before adopting it.

agent_session_id=agent.agent_session_id,
episode_id=request.episode_id,
task_id=request.task.task_id,
tool_accesses=tool_accesses,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Forward the typed task's agent context instead of dropping it

A request containing task.task_input.agent_context.agent.values = {"tag": "x"} validates, but this seed omits the context. SingleAgentTurnTaskInput keeps it outside task_data, so it reaches neither the seed nor the policy request. The episode silently runs without supplied task context. The agent contract requires validation or rejection instead. The configured simulation agent accepts no context, so this example should fail at seed rather than be ignored.

Add this argument to AgentSeedSessionRequest:

agent_context=request.task.task_input.agent_seed_context,

The serializer already omits None, so context-free rows keep their current seed shape. Keep context out of simulator/scoring task data and policy metadata.

Proposed checks, not executed: submit nonempty context against the simulation agent and expect a terminal agent-seed failure caused by its 422 unsupported-context refusal, with no policy call. Against a context-aware agent stub, assert that the exact context reaches the seed. Retain the successful context-free episode and verify that empty context is omitted. Verify the suggested argument before adopting it.

async def _seed_agent_session_state(self, body: AgentSeedSessionRequest) -> SimpleAgentSessionState:
"""simple_agent's session, refused (422) without a direct HTTP grant: the row's tools are reached only through
it, so every turn of such a session would fail."""
state = self._new_session_state(body)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: Reject sessions that disable tool execution

With execute_tools=false, this agent now accepts a session seed containing a direct HTTP grant. Given valid harness metadata and a model reply calling bash, it still dispatches the tool and continues its whole-turn loop because _create_episode never reads execute_tools. SimpleAgent.responses rejects this setting only outside sessions. Its session contract is to return unresolved calls without executing them, as the existing test asserts. The previously refused session path is now reachable, so a disabled execution control can still cause simulated tool effects.

For this whole-turn adapter, explicitly reject the unsupported setting before creating state in _seed_agent_session_state:

if not self.config.execute_tools:
    raise AgentSeedRejectedError(
        "agentic_turn_simulation_agent requires execute_tools=true"
    )

Update _create_episode's comment, which currently claims inherited responses() already refuses it. Keep default native sessions executing through their grant, Cline sessions running only calls before a stop tool, and legacy /run rejecting disabled execution.

Proposed checks, not executed: configure execute_tools=false, POST a valid seed with a direct grant, and expect HTTP 422 with no model or tool requests. With execution enabled, retain the native close assertion ("c1", "result of c1") and the Cline close's read output in the added endpoint tests. Retain test_run_without_executing_tools and the Cline before/after-stop assertions. Verify the suggested guard before adopting it.

weijiac0619 and others added 2 commits November 17, 2026 16:05
… is refused

A 401 (no valid episode token), 403 (a token for another episode) or 410 (the episode is not running) from an
Environment Server's tool route was retried, then answered as a simulator failure that scoring masks: a misconfigured
grant masked every turn and showed only as a high failed rate. These refuse the grant itself, so the turn now fails
unretried and the Environment Server records a failure row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
…of the server's own model calls

- TurnToolset.turn_so_far gave the simulator every record the base wrote, including the server's own refusals and
  failures ({"detail": ...}: no such tool, a failed or timed-out call). The policy never saw those: the agent answers a
  refusal with its own text and retries a failure under the same call id. They are left out now, and of a call id's
  records only the last is kept.
- In a run that sets model_call_sampling, the tool simulator's and judges' calls carried no sampling segment, where a
  resources server's carry null (CallerSamplingMiddleware). A trained agent's Model Server refuses a generation call
  without a segment, so a simulator or judge sharing it failed on every call. They now send null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
@weijiac0619

Copy link
Copy Markdown
Contributor Author

/ok to test 935e2e1

They are exported there; tests/unit_tests/test_public_api_facades.py refuses an import of an exported name from its
submodule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
@weijiac0619

Copy link
Copy Markdown
Contributor Author

/ok to test 2134eaf

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Nov 18, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants