Repository navigation
feat(agent_arena): serve AgentArena's tools from a BaseEnvironmentServerWithTools - #6267
weijiac0619 wants to merge 15 commits into
Conversation
…erWithTools and add start_episode_sandbox An episode of a BaseEnvironmentServerWithTools has a control plane, the server, which decides what a tool call runs, and a data plane, the episode's sandbox, which runs every operation on the episode's world. The control plane never runs such an operation in its own process. - BaseEnvironmentServerWithToolsConfig.sandbox_provider is required, so every environment says where its data plane runs. A provider that the config does not define fails at startup. The local provider runs commands as host subprocesses, so the server logs a warning at startup. - start_episode_sandbox(request, cleanup, spec, setup) starts a sandbox for the episode. It requires ttl_s, labels the sandbox with the server's name and the episode's rollout over the provider's and the spec's metadata, and registers the stop in the episode's cleanup before it starts the sandbox, so the cleanup or the shutdown sweep also stops a sandbox whose start or first stop failed. A failed setup stops the sandbox. - The class docstring states the control plane and data plane rule. A runtime check that refuses process creation and sqlite3 in the control plane is left for a follow-up (GYMINT-733). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…t Server example BaseEnvironmentServerWithToolsConfig now requires sandbox_provider, so the example would fail validation without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…up handle start_episode_sandbox stopped such a sandbox with a direct call, which the cleanup timeout did not bound and the episode's cleanup repeated. It now closes the registered handle: the stop gets the cleanup timeout and the entry's lock, a stop that succeeded is not run again, and one that failed or timed out stays for the episode's cleanup to retry. The docstring now names the providers that ignore ttl_s (Docker with a custom entrypoint, Apptainer, Enroot and local). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…oring into shared modules Session setup, the tool simulator, the judge, scoring and metrics move out of the resources server's app.py into turns.py (TurnSimulation, TurnSession) and schemas.py (settings and route bodies), so an Environment Server can run the same turns. The code moves as it was; verify takes the session as an argument. The resources server becomes a thin wrapper and keeps serving the legacy relay and gym eval reverify. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
…n through its direct HTTP grant In an agent session the agent sends /harness and /simulate_tool_call to the grant's URL, under the activation's rollout prefix, with the grant's headers and the session's cookies; any /harness error through the grant fails the turn. A seed without a direct HTTP grant is refused. /run still calls the configured resources server. The agent joins AGENT_SESSION_AGENT_TYPES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
… an agent session In an agent session the row's tools are served by an Environment Server: each call goes to POST /<tool name> through the session's direct HTTP grant, with the arguments object as the body and the call id in X-NeMo-Gym-Tool-Call-Id. The row's stop tools and tool-call protocol come in the request's metadata (agentic_turn_simulation_stop_tools, agentic_turn_simulation_protocol), which the agent takes out before it calls the model. A call the server cannot take (arguments that are not a JSON object, a name no grant can carry, a tool the row lacks) is answered by the agent without a retry; scoring gates such a turn. For Cline replies with several calls, only the last result keeps the environment details. The legacy /run path is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
…verWithTools environment_servers/agent_arena runs each AgentArena row as one episode. It builds the turn from the whole row, serves the row's tools with serve_tools (TurnToolset, an EpisodeToolset whose calls are answered as /simulate_tool_call answers them), seeds and invokes agentic_turn_simulation_agent with the row's prompt and harness metadata, and scores the turn in process with the resources server's settings. The simulator's view of the turn is rebuilt from the episode's earlier calls and their results (TurnToolset.turn_so_far). sandbox_provider is local: no episode starts a sandbox. A flat benchmark row gets a flat result. The benchmark and the environment's own config use the new server; their _legacy_agent twins keep the relay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
#5917 merged into main. Keep both new entries in AGENT_SESSION_AGENT_TYPES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n agent session's close The session's observations keep no tool output, and a rollout's trajectory takes each call's output from the conversation's function_call_output items. A Cline turn has none (its results are in a user message), so its tool calls had no output. The close now returns the trajectory the agent recorded, as /run returns it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… trajectory simple_agent's session turn records its trajectory under task "unscoped"; `/run` stamps the row's task on it, a session did not, so every rollout's trajectory carried producer_trajectory_identity_mismatch. Stamp the seed's task id on the trajectory and its turns, as the OpenCode agent does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
The conflict was in resources_servers/agentic_turn_simulation/app.py: this branch moved the server's settings and scoring into schemas.py and turns.py, and main (#6072) added an outcome policy to them. The policy (TIME_CAP_KIND, SIMULATOR_FAILED_KIND, the outcome_policy setting and its validators) now lives in AgenticTurnSimulationSettings, and the saved decision of a masked row in TurnSimulation._mask_reason, so the agent_arena environment server saves the same decision as the resources server. agent_arena's tests check it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
|
/ok to test 8e0aa63 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
I found six RISKs in the migration. Supported controls are silently ignored, and reconstructed tool history changes opt-in obstacle behavior.
I reviewed all 27 changed files and traced the affected runtime contracts. I did not execute tests or rollouts; the validation numbers in the PR description are author-reported.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| self._verify_body(request, response) | verification_failure_fields(response) | ||
| ) | ||
| try: | ||
| verified = await self._turns.verify(body, session) |
There was a problem hiding this comment.
RISK: Preserve skip_verification when moving off the relay
With ++skip_verification=true ++skip_verification_reward=0.25, this path still runs scoring and returns its reward. Config parsing places those controls on the agent, but this server always calls self._turns.verify. The previous SimpleAgent /run path returned the fixed reward without verification. Generation-only runs now depend on judge availability and can receive a different reward.
Please resolve the bound agent's controls before scoring, after agent-outcome handling. Add a helper using the typed parsing in the sibling implementation. When skipping is enabled, construct the result with:
SingleAgentTurnResult.model_validate(
self._verify_body(request, response)
| {"reward": skipped_reward, "verification_skipped": True}
)Keep the existing attachment of observations and trajectory. Apply the same decision in aggregate_metrics: skip benchmark scoring metrics while still adding performance metrics, as the sibling server does. The invariant is that an explicit verification skip neither scores the turn nor reports its synthetic reward as measured turn_score.
Proposed checks, not executed: use agent skip_verification=true, reward 0.25, and a judge stub that fails if called. Expect the fixed reward, verification_skipped=true, no scoring call and normal cleanup. Retain judged rewards and benchmark metrics when skipping is false. Cover per-agent false overriding global true, string "false" parsing as false, and fixed reward 0.0. Verify the suggested construction before adopting it.
| return AgenticTurnSimulationSettings.model_validate({k: v for k, v in raw.items() if k in fields}) | ||
|
|
||
| def setup_webserver(self) -> FastAPI: | ||
| app = super().setup_webserver() |
There was a problem hiding this comment.
RISK: Mark simulator and judge calls as untrained sampling
A training run can set model_call_sampling.required_model_servers: [policy_model] and use that model-server instance for the policy, simulator and judge. The policy activation carries its encoded sampling segment, but simulator/judge calls from this server carry none. They therefore hit the model server's missing-segment 422, causing simulator-failure masks or failed verification. ServerClient only propagates the current segment; neither episode requests nor independent tool requests establish the null context here. The previous resources-server path installs that context.
A small correction in setup_webserver, immediately after super(), is:
if model_call_sampling_config(self.server_client.global_config_dict) is not None:
app.add_middleware(CallerSamplingMiddleware)Import model_call_sampling_config from the public core API. Export CallerSamplingMiddleware through the server-runtime public API before importing it here. This covers both /run and separately arriving tool calls. The policy's explicit encoded segment must stay unchanged; simulator/judge calls must carry /ng-sampling/null. Evaluation without sampling configuration must keep its current URLs.
Proposed checks, not executed: exercise an episode and granted tool route with one required model-server instance serving all roles. Assert encoded episode values on policy calls, null on simulator/judge calls, and no missing-segment rejection. Retain evaluation URL assertions and token-capture skip behavior. Verify the suggested middleware change before adopting it.
| for index, record in enumerate(self._calls): | ||
| call_id = record.call_id or f"ng_call_{index}" | ||
| output = record.output if isinstance(record.output, str) else json.dumps(record.output, ensure_ascii=False) | ||
| items += [ |
There was a problem hiding this comment.
RISK: Preserve reply boundaries for obstacle injection
The reconstructed history changes deterministic obstacle semantics, not only the simulator's prose context. Enable obstacle_injection, force a sandbox row's obstacle to transient, and let one native reply contain fetch_page followed by ask_user. The agent executes the fetch but does not send the stop call through the grant. turn_so_far therefore shows only the fetch, so an error is injected. The contract says a reply containing a stop tool must not meet its first obstacle because the policy cannot react. Final scoring sees the complete reply and reports obstacle.fired=false, contradicting the injected result. See the obstacle decision and the existing stop-tool regression.
Two identical calls in one reply also change behavior: interleaving every recorded call with its result makes the second look like a later reply. A transient obstacle then answers it normally, although both sibling calls should receive the same error. The existing test explicitly protects that distinction.
Please carry reply-batch context from the agent's _simulate_calls into the episode's simulation context, including unexecuted stop calls and the current call's position. Rebuild grouped calls followed by their results before evaluating obstacles, consistently with final scoring. EpisodeToolset.call currently has no batch context, so I cannot give a safe local replacement for this loop. A bounded fallback is to reject enabled obstacle injection on this path with guidance to use the relay until that context is available. Do not change the shared obstacle rules to match the lossy history.
Proposed checks, not executed: a fetch beside a stop tool receives a normal result and reports fired=false; duplicate sibling fetches both receive the transient error; the same fetch in the next model reply succeeds. Preserve persistent blocked_route failures, including a later blocked fetch beside a stop tool after an earlier failure. Also retain normal tool execution with injection disabled.
| """An episode request gets the episode reply; a flat row (a benchmark's) gets a flat result, as | ||
| single_agent_turn_legacy answers it.""" | ||
| is_episode_request = "episode_id" in row and "task" in row | ||
| request = ( |
There was a problem hiding this comment.
RISK: Return HTTP 422 for invalid flat and typed requests
This replacement /run endpoint accepts a dict and validates it inside the handler. A flat row with valid indices and prompt but _ng_skip_token_capture: "yes" raises ValueError; a malformed typed task raises ValidationError. Neither is translated, so the served endpoint returns 500 instead of a terminal 422. Wrong-agent and negative-attempt rows follow the same path. Callers can consequently retry inputs that cannot succeed.
Please apply the narrow translation already used by the sibling flat adapter:
try:
request = (
SingleAgentTurnRequest.model_validate(row)
if is_episode_request
else self._episode_request_from_row(row)
)
except ValidationError as error:
raise RequestValidationError(error.errors(include_url=False)) from None
except ValueError as error:
raise HTTPException(422, str(error)) from NoneAdd the corresponding FastAPI imports. Keep run_request outside the catch so runtime failures are not mislabeled as invalid input. _episode_request_from_row can still raise ValueError for direct callers; update the existing run_legacy wrong-agent test to check the HTTP contract.
Proposed checks, not executed: POST malformed capture controls, a mismatched agent_ref, a negative attempt and malformed typed task input. Each should return 422 without downstream calls. Retain successful flat results and typed episode replies on the single /run route, ahead of the tool catch-all. Verify the suggested code before adopting it.
| agent_session_id=agent.agent_session_id, | ||
| episode_id=request.episode_id, | ||
| task_id=request.task.task_id, | ||
| tool_accesses=tool_accesses, |
There was a problem hiding this comment.
RISK: Forward the typed task's agent context instead of dropping it
A request containing task.task_input.agent_context.agent.values = {"tag": "x"} validates, but this seed omits the context. SingleAgentTurnTaskInput keeps it outside task_data, so it reaches neither the seed nor the policy request. The episode silently runs without supplied task context. The agent contract requires validation or rejection instead. The configured simulation agent accepts no context, so this example should fail at seed rather than be ignored.
Add this argument to AgentSeedSessionRequest:
agent_context=request.task.task_input.agent_seed_context,The serializer already omits None, so context-free rows keep their current seed shape. Keep context out of simulator/scoring task data and policy metadata.
Proposed checks, not executed: submit nonempty context against the simulation agent and expect a terminal agent-seed failure caused by its 422 unsupported-context refusal, with no policy call. Against a context-aware agent stub, assert that the exact context reaches the seed. Retain the successful context-free episode and verify that empty context is omitted. Verify the suggested argument before adopting it.
| async def _seed_agent_session_state(self, body: AgentSeedSessionRequest) -> SimpleAgentSessionState: | ||
| """simple_agent's session, refused (422) without a direct HTTP grant: the row's tools are reached only through | ||
| it, so every turn of such a session would fail.""" | ||
| state = self._new_session_state(body) |
There was a problem hiding this comment.
RISK: Reject sessions that disable tool execution
With execute_tools=false, this agent now accepts a session seed containing a direct HTTP grant. Given valid harness metadata and a model reply calling bash, it still dispatches the tool and continues its whole-turn loop because _create_episode never reads execute_tools. SimpleAgent.responses rejects this setting only outside sessions. Its session contract is to return unresolved calls without executing them, as the existing test asserts. The previously refused session path is now reachable, so a disabled execution control can still cause simulated tool effects.
For this whole-turn adapter, explicitly reject the unsupported setting before creating state in _seed_agent_session_state:
if not self.config.execute_tools:
raise AgentSeedRejectedError(
"agentic_turn_simulation_agent requires execute_tools=true"
)Update _create_episode's comment, which currently claims inherited responses() already refuses it. Keep default native sessions executing through their grant, Cline sessions running only calls before a stop tool, and legacy /run rejecting disabled execution.
Proposed checks, not executed: configure execute_tools=false, POST a valid seed with a direct grant, and expect HTTP 422 with no model or tool requests. With execution enabled, retain the native close assertion ("c1", "result of c1") and the Cline close's read output in the added endpoint tests. Retain test_run_without_executing_tools and the Cline before/after-stop assertions. Verify the suggested guard before adopting it.
… is refused A 401 (no valid episode token), 403 (a token for another episode) or 410 (the episode is not running) from an Environment Server's tool route was retried, then answered as a simulator failure that scoring masks: a misconfigured grant masked every turn and showed only as a high failed rate. These refuse the grant itself, so the turn now fails unretried and the Environment Server records a failure row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
…of the server's own model calls
- TurnToolset.turn_so_far gave the simulator every record the base wrote, including the server's own refusals and
failures ({"detail": ...}: no such tool, a failed or timed-out call). The policy never saw those: the agent answers a
refusal with its own text and retries a failure under the same call id. They are left out now, and of a call id's
records only the last is kept.
- In a run that sets model_call_sampling, the tool simulator's and judges' calls carried no sampling segment, where a
resources server's carry null (CallerSamplingMiddleware). A trained agent's Model Server refuses a generation call
without a segment, so a simulator or judge sharing it failed on every call. They now send null.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: weijia chen <weijiac@nvidia.com>
|
/ok to test 935e2e1 |
They are exported there; tests/unit_tests/test_public_api_facades.py refuses an import of an exported name from its submodule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
|
/ok to test 2134eaf |
What and why
Built on #5917 (merged), which makes
sandbox_providerrequired forBaseEnvironmentServerWithTools.AgentArena (
benchmarks/agent_arena/config.yaml) runsagentic_turn_simulation_agentbehind the deprecatedlegacy_agentrelay. #6103 moved it onto a dedicated Environment Server that served the turn's tools through its ownroutes (
/harness,/simulate_tool_call) behind a per-episode secret, with one worker only. That was hand-rolledwhile #5346 to #5350 were open. This PR builds the same server on
BaseEnvironmentServerWithToolsinstead, assuggested in the review: signed episode tokens,
serve_tools, the shutdown sweep and the routing of tool calls to theworker that runs the episode come from the base class. #6103 stays open until this one is accepted.
No issue is linked: the work is tracked internally.
AgentArenaEnvironmentServersubclassesBaseEnvironmentServerWithTools.run()for one row:serve_tools(request, TurnToolset(...), cleanup).TurnToolsetis anEpisodeToolset:specs()lists the tools a call may name, the same set the scoring'snonexistent_toolgate accepts: the row'stools, or Cline's tool names for a Cline row (which declares none). The row declares its tools to the policyitself, so a spec only names a tool the server answers.
call(name, arguments)answers a call as the resources server's/simulate_tool_calldoes: in code from thereplayed workspace, else by the tool simulator LLM, or with the row's obstacle.
The tools are
POST /<tool name>on this server, reached with the grant's episode token./v1/responseswith the row'sresponses_create_params. Their metadata name the row's stop tools and tool-call protocol(
agentic_turn_simulation_stop_tools,agentic_turn_simulation_protocol), which/harnessused to return: thebase's token middleware lets a token reach only tool paths, so the server cannot serve
/harnessto the agent. Theagent takes both keys out before it calls the model.
fields as the resources server's
/verifyresponse, and/aggregate_metricscomputes the same metrics.No Resources Server takes part in the episode.
What the simulator sees (open question)
EpisodeToolset.callreceives a tool's name and arguments only. On the relay the agent sent the turn so far with eachcall, so the simulator saw what the policy wrote between its calls and the call's place in its reply. Here the server
rebuilds that view from the episode's earlier calls and their results (
ServedTools.calls), after the row's history:TurnToolset.turn_so_faris the one place that decides it. It keeps only results the policy saw: records the serverwrote for a call it refused or that failed (
{"detail": ...}) are left out, since the agent answers a refusal with itsown text and retries a failure under the same call id, and of a call id's records only the last is kept. Two
consequences:
around a Cline XML call).
<environment_details>on each result. The agent strips them from all but the last result when it joins them intoCline's user message, as the server did before.
Should the base pass more per-call context to a toolset, such as the call id or the agent's turn so far? If it does,
turn_so_faris the piece to replace. Under Validation, neither a fixed-policy replay (51/54 rows judged the same,against 53/54 for two runs of the relay's own context) nor a paired run of the whole validation set (reward
difference +0.001, standard error 0.011) finds a measurable effect.
Changes
environment_servers/agent_arena(new): the server (app.py),TurnToolset(toolset.py), config, README,tests.
(
resources_server). Overrides of those settings keep working, andgym eval reverifythrough that server scoresa saved row with the same settings. The episode makes no call to it.
sandbox_provider: {local: {}}. No operation runs in a sandbox: a call is answered from the model of the agent'sfiles or by a model call. The server logs the base's warning about
localat startup.single_agent_turn_legacydoes./runtakes both shapes and stays ahead of the base's tool route.agent_outcome_policy; a judgethat cannot be reached is a retryable verification failure.
outcome_policy(fix(envs): declare retry for environment-side faults they mask #6072), asthe resources server's
/verifydoes.model_call_sampling, the server's own model calls (the tool simulator and the judges) sendthe
nullsampling segment, as a resources server's calls do (CallerSamplingMiddleware). They keep their ownsampling, and a trained agent's Model Server, which refuses a generation call without a segment, takes them when
the simulator or a judge shares it. Only the agent's calls carry the training framework's values.
resources_servers/agentic_turn_simulation: session setup, the tool simulator, the judge, scoring and metricsmove out of
app.pyintoschemas.pyandturns.py, which both servers use. The code is moved as it was.app.pykeeps the resources server as a thin wrapper; it still serves the relay and
gym eval reverify.schemas.pyalsodefines the two metadata keys. The
outcome_policysetting that main added toapp.py(fix(envs): declare retry for environment-side faults they mask #6072) is in the sharedsettings in
schemas.py, and the decision a masked row saves is inturns.py.responses_api_agents/agentic_turn_simulation_agent: serves agent sessions.back to the config's stop tools and native calls.
POST /<tool name>through the grant (under the activation's rollout prefix): thearguments object is the body, the call id is in
X-NeMo-Gym-Tool-Call-Id, with the grant's headers and thesession's cookies. Error replies are retried, then become the simulator-failure output that scoring masks, as on
the relay.
episode (403), or an episode no longer running (410). That is a setup or protocol fault; masked as a simulator
failure it would show only as a high failed rate.
no grant can carry, or a tool the row lacks (the server's 404). Scoring gives such a turn 0 whatever the call
returned (
malformed_arguments,nonexistent_tool); on the relay the simulator answered these calls./runstill calls the configured resources server's/harnessand/simulate_tool_call, unchanged./runreturns it, under the seeded task. The session'sobservations keep no tool output, and the rollout's trajectory takes each output from the conversation's
function_call_outputitems, which a Cline turn does not have (its results are in a user message).nemo_gym/config/global_config.py:agentic_turn_simulation_agentjoinsAGENT_SESSION_AGENT_TYPES.benchmarks/agent_arena/config.yamland the environment's ownconfigs/agentic_turn_simulation.yamluse theagent_arenaEnvironment Server.benchmarks/agent_arena/config_legacy_agent.yamlandconfigs/agentic_turn_simulation_legacy_agent.yamlkeep therelay, with a deprecation note.
Compatibility
--benchmark agent_arenaandgym env start --resources-server agentic_turn_simulationnow run theagent_arenaEnvironment Server.
--benchmark agent_arena/config_legacy_agentandconfigs/agentic_turn_simulation_legacy_agent.yamlrun the relay as before.num_repeats: 4, prompts, settings and scoring are unchanged. Overrides of theresources server's settings apply to the new path too.
configs/agentic_turn_simulation.yamlshould switch tothe
_legacy_agentconfig or be checked on the new path first. Training was not run here.agentic_turn_simulation_agentjoiningAGENT_SESSION_AGENT_TYPESinnemo_gym/config/global_config.py.(retried within the attempt budget). A time-capped turn is still masked (
agentic_turn_simulation:time_cap).app.py;JudgeSpecand theprivate helpers now live in
schemas.pyandturns.py.Validation
Unit tests
Each directory is a separate pytest run. The first four were run on the current head; the others on this branch before its latest merge with main:
environment_servers/agent_arena/testsresources_servers/agentic_turn_simulation/testsresponses_api_agents/agentic_turn_simulation_agent/testsbenchmarks/agent_arena/testsenvironment_servers/single_agent_turn/testsenvironment_servers/single_agent_turn_legacy/testsenvironment_servers/legacy_agent/testsresponses_api_agents/simple_agent/teststests/unit_tests(config, environment, tooling, server_runtime, public API, session contract, import-linter coverage)Real runs
One checkpoint served by vLLM plays the policy, the tool simulator and the judge. 15 rows: 10 AgentArena
validation rows (sandbox harness) and one row each for sandbox, Claude Code, Hermes, OpenCode and Cline. Each row runs
twice (30 rollouts per run). "New" is
--benchmark agent_arena; "relay" is--benchmark agent_arena/config_legacy_agent.The relay output had one extra rejudged copy of a rollout, left out of the counts below.
End to end:
Every row ran on both paths; the Cline row ended on
attempt_completionwith reward 1. The rows that differ arewithin run-to-run noise: vLLM at temperature 0 is not deterministic with long reasoning, so the two runs' policies
diverge after a few tokens.
one row masked by the turn's time cap (
agentic_turn_simulation:time_cap) on the new path.Fixed policy, call by call. To separate the simulator's context from the policy's variance, each recorded rollout
from the four runs above (54 sandbox rollouts with native calls; 285 calls) was replayed against the simulator with
its recorded calls and outputs held fixed:
(A and A').
TurnToolset.callwith the earlier calls asServedTools.callswould hold them (no textbetween calls).
Each variant's outputs then replace the recorded ones and the turn is judged again.
Of the three rows where A and B disagree:
total; they fell on other turns.
The simulator's text at temperature 0 is about as different between two runs of A as between A and B. The failed rate
does not rise without the text between calls. On these rows the loss of that text is not measurable.
Whole validation set, paired. All 433 AgentArena validation rows (sandbox harness), two rollouts each, on both paths:
the same checkpoint as policy, simulator and judge, one vLLM instance per path, temperature 1.0, top_p 0.95, 32768
output tokens, a 1200 s turn time cap. The relay output had one extra rejudged copy of a rollout, left out.
Paired by task (each task's mean over its rollouts on each path, new minus relay):
Six tasks flip fully (both rollouts), four toward the new path and two toward the relay; 62 tasks differ in how many
rollouts were time-capped, 32 more on the new path and 30 more on the relay. An audit of every new-path rollout found
each tool call with its output and no harness metadata in a model request. One call named a tool the row lacks
(
ffprobe) and was refused by the agent, as described above. The validation set has no Cline rows; the Clinetrajectory is covered by the agent's tests.
Checks
pre-commit run --files <changed files>passes (ruff, ruff-format, the README environment list, import-linter).Known limits
about ±0.02.
rollout's model-call capture holds them next to the policy's, with the gap
model_call_ownership_unavailable, asBIRD-Interact's in-process user simulator does. On the relay they ran in the resources server and were not
captured. The trajectory's turns come from the agent's own trajectory, so they are the policy's alone.
and more, minutes each); the median simulator call takes about 10 s. Both paths hit the cap at the same rate (see
the paired run).
ToolSpecs (name pattern, object parameters) and must not take aroute's name (
run,aggregate_metrics,docs,openapi.json,redoc). A row that breaks this fails at setup.All AgentArena and generic RL rows that were checked pass: 433 validation rows and 1500 training rows (sandbox), and
2089 generic training rows (OpenClaw 298, Hermes 633, OpenCode 639, Claude Code 515, Kilo Code 4), plus the
example rows. No row had an invalid tool name, non-object parameters or a reserved name.
gym eval reverifyre-scores through the resources server's/verify, which sharesturns.pyand the settings.single_agent_turn's, written outhere rather than shared.
tool_transportsstays[direct_http], the transportagentic_turn_simulation_agenttakes. Row tool namesare not checked against MCP's stricter name rule.
own config (
configs/agentic_turn_simulation.yaml) now runs it;configs/agentic_turn_simulation_legacy_agent.yamlkeeps the relay. Open for the environment's training owners: the simulator's and judges' calls run in the rollout's
context, so a rollout's model-call capture holds them (see "Captured model calls"); whether a training setup reads
tokens from that capture, and so whether a simulator or judge served by the policy's own Model Server would reach
training data, was not checked.
🤖 Generated with Claude Code