Repository navigation
fix(env): own resources bindings and resolve inherited renames - #5108
Conversation
2b26f8f to
7fe55fd
Compare
…t_turn names none A legacy_agent relay names no resources server: the agent's /run uses its own. A single_agent_turn_legacy server that replaces the relay had to pin one, and a pinned name breaks a benchmark that reaches the server through an _inherit_from rename of the agent. The rename also renames the resources server, and Gym follows renames only for agent references, so the pinned name dangles (for example indic/math_500 through math_with_judge's server). resources_server is now optional. Unset, the server binds the resources server the bound agent names, as the relay did, so it follows the agent rename. A server that sets none in front of an agent that names none fails at startup instead of on the first episode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
About 307 configs reach simple_agent through a legacy_agent relay, so a crash repeats the whole episode. Under single_agent_turn_legacy, it repeats only the current step, and simple_agent resumes inside its turn (GYMINT-549). add_legacy_agent_environment_servers.py gains --switch-simple-agent. It replaces each relay in front of a simple_agent with a single_agent_turn_legacy server of the same name, so rows keep their routing. The server names no resources server, so it binds the agent's, as the relay did, and follows an agent rename. It grants direct HTTP access to the resources server's tools, the routes the agent's /run called. It resolves each config with its own config_paths, as Gym does, since instance names are unique only within a run. A relay stays, listed with its reason, when its agent binds no resources server, declares a grant simple_agent sessions reject (a required MCP access or a second direct HTTP access), binds a resources server whose seed_session takes only the legacy body (19 relays today, read from the server's source so a server passes once it migrates), or its run cannot load an include. A relay that inherits another keeps inheriting it, and the script refuses to switch only one of the two. The default mode is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The pairing guard only compared `_inherit_from` children that carry their own `legacy_agent` key, so a run that merged a switched relay with a same-name `legacy_agent` override (the hle and apex_shortlist opencode/pi recipes) or a bare `_inherit_from` child exited 0 with two environment server types. - Resolve every run after the planned switches. A relay that a run merges with a legacy_agent server stays on the relay; when the relay it inherits switches, the script refuses (exit 2) and names the config. Top-level nulls now remove instances, as in Gym. - Keep relays whose resources server reads the row's rollout coordinates (genrm_compare), whose implementation is not under resources_servers/, or that set keys the switched stanza would drop. - Build every rewrite before writing any file; a stanza that cannot be found exits 2 instead of raising StopIteration. - Dedupe scanned paths after resolving them. - Give the hle and apex_shortlist opencode/pi recipes a legacy_agent server of their own, so their benchmarks can switch. - Fix the docstring and docs examples, document the new stay reasons, note how to inherit the single_agent_turn templates, and test the legacy adapter with resources_server unset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
When two files of a run declare the same relay (a base and a partial override of it) and a later config merges it with a legacy_agent server, split_servers found two switching entries for the instance itself and looked for a different inherited relay that was not there, raising StopIteration. Every declaration of the instance now stays on the relay; the script still refuses (exit 2) only when a different relay the instance inherits switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
7fe55fd to
43f6d53
Compare
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Runs scripts/add_legacy_agent_environment_servers.py --switch-simple-agent over the repository (GYMINT-549). 287 relays in front of simple_agent in 279 config files become single_agent_turn_legacy servers of the same name, so a crash repeats only the current step instead of the whole episode, and simple_agent resumes inside its turn. 20 relays stay, as the script reports: - 19 whose resources server seeds only the legacy body (workplace_assistant, openenv, newton_bench, indirect_prompt_injection, example_session_state_mgmt, and the SWE example configs); they move once their servers accept the typed seed. - benchmarks/nemotron_3_ultra/benchmarks_no_external.yaml, whose include benchmarks/ruler/config_nemotron_3_256k.yaml does not exist on main. Tests and docs that read these configs follow the new key. The super35 container build now covers single_agent_turn_legacy too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Regenerated with the fixed --switch-simple-agent: genrm_compare's /verify needs the row's _ng_rollout_index and cohort keys, which single_agent_turn_legacy does not forward, so both genrm servers stay on legacy_agent. hle and apex_shortlist still switch now that their opencode/pi recipes declare their own relay. The YAML is byte-identical to a fresh generator run: 278 files, 285 relays switched, 22 kept. The docs said every shipped simple_agent config uses single_agent_turn_legacy; 22 stay on the relay, so say "most" and name why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The switched configs add single_agent_turn_legacy to the core batch's resolved server set, so the prebuilt dependency check now reports 16 environment paths. CI on Linux showed "16 server environment paths checked" against the old expectation of 15; macOS Bash 3.2 stops these shell tests before the count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
With model-call capture on, a simple_agent episode run through an Environment Server (single_agent_turn, single_agent_turn_legacy) lost its executed tool records: the session kept only the trajectory's invocations and gaps, so the collected ng_trajectory had no tool_calls and ng_perf.num_tool_calls was 0, while /run reported them. The session now adds a ToolCallObservation for each executed tool call to the agent observations it returns at close. Rollout collection already merges these into ng_trajectory.tool_calls and rejoins each call's output from the invocation conversation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 2ef4896) Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…_legacy A `--agent-type` swap onto an environment fronted by single_agent_turn_legacy kept that server type even when the incoming agent does not implement agent sessions, so every episode failed with NotImplementedError from /v1/agent_sessions. Composition now rewrites such a server to a legacy_agent relay that calls the agent's own /run, keeping its entrypoint, agent_server and host/port. AGENT_SESSION_AGENT_TYPES in global_config is the single list of session-capable agent types; the legacy-relay migration script reads it instead of its own copy, and a unit test keeps it in step with the agent apps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit d156ef2) Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ple_agent into simple_agent Both agents make one model call and pass its tool calls to the verifier unexecuted. simple_agent with execute_tools: false does the same as an agent session, so its configs run under single_agent_turn_legacy and resume inside the turn after a crash (GYMINT-217, GYMINT-147). 14 configs move: image_tools_pivot, swe_pivot, structured_outputs_v4, seven single_step_tool_use_with_argument_comparison configs and four terminal_multi_harness configs. Each agent keeps its instance name and becomes simple_agent with execute_tools: false; its relay becomes a single_agent_turn_legacy server, which simple_agent needs since it refuses /run without tool execution; allowed_agents gains simple_agent. The pivot-dataset skill's template and reference config writer emit the same pairing. Both agents stay, marked deprecated, so configs outside the repository keep working until a later release removes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ent configs With execute_tools: false the agent never calls a tool, so the direct_http grant on the 14 converted configs was unused, and the pivot skill's template already omits it. It also made a silent mis-score possible: `--agent-type simple_agent` is now allowed on these servers, composition does not carry execute_tools over, and the swapped-in agent would execute tools through the grant against verifiers that score one unexecuted call. Without the grant it fails loudly instead. A repository test now asserts that no execute_tools: false simple_agent sits behind a legacy_agent relay (its /run refuses) or gets a tool grant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
Hi @ananthsub , the import issue is fixed and CI is green. It still needs a code-owner approval because it touches |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41683812971 |
Merge codex/env-resources-bindings at c4e6830 to resolve the GenRM MMRB configuration ambiguity that failed server-suite shard 0. Preserve the DeepAgents diff unchanged. Validation: GenRM suite 47 passed; configuration, resources binding and migration generator tests 359 passed (the local port test passed outside the sandbox); pre-commit run --all-files and git diff --check passed. No new model rollout: this merge imports existing config bindings and a test import correction, with no agent or verifier code changes. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41693344268 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41694085699 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41695039879 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41699905303 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41704097016 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41705943440 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41709778427 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41711148823 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41743893874 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41750498064 |
NVIDIA-NeMo#5108 made the Environment Server's resources_server required again and dropped the fallback to the agent's. The startup check for tool servers now compares against the configured resources_server; the agent fallback for tool_servers stays. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
single_agent_turn servers now require resources_server (NVIDIA-NeMo#5108). Bind critpt_resources_server on the example server and the renamed critpt_benchmark_resources_server on the benchmark server, the servers their agents already use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…erver block Gym #5108 made resources_server a required field of single_agent_turn_legacy: the Environment Server no longer takes it from its agent. configs/agentic_mme.yaml (the environment's own agent) and benchmarks/agentic_mme/server_tools.yaml (simple_agent) left it out, so gym env validate failed for both with 'resources_server: Field required'. The base block now names agentic_mme, and every pairing in benchmarks/agentic_mme/ inherits it. opencode.yaml no longer repeats it. gym env validate passes for all 12 Agentic-MME configurations (each benchmark config, the no-internet overlay, the search overlay and the six retrieval overlays). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
…s tests Two core unit tests that arrived or changed with the origin/main merge (69c748d) failed on this branch's files: - test_public_api_facades flagged tests/test_app.py importing maybe_auto_expose from nemo_gym.server_runtime.mcp_auto_exposure; it is exported from nemo_gym.server_runtime. - test_repository_step_environments_declare_their_resources (#5108) flagged hle_web_tools_environment_server, whose single_agent_turn_legacy block named no resources_server. It now names hle_web_tools, as the tavily_search configs do. Before: the two tests, 2 failed. After: tests/unit_tests/ test_public_api_facades.py and test_add_legacy_agent_environment_servers.py 917 passed; resources_servers/hle_web_tools/tests 95 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eset (NVIDIA-NeMo#5176) * feat(hle_web_tools): upstream HLE web_search and web_fetch behind the blocklist over MCP Add resources_servers/hle_web_tools, a tool server that implements section 3 of centerforaisafety/hle docs/evaluation-with-tools.md (commit 22ed307): exactly two MCP tools, web_search (at most 20 results) and web_fetch (Markdown capped at 60,000 characters with a truncation indicator, no media payloads), with Tavily as the search and extraction backend and the keys held server-side. The pinned upstream docs/blocklist.json is shipped byte-exact and digest-checked at start; all seven rule families are enforced with the protocol's semantics, plus query refusal for blocked site: domains and verbatim question spans (8 consecutive words, configurable; the question arrives through /seed_session), fetch-URL checks after percent-decoding archive and proxy wrappers, title-and-Markdown re-checks of fetched and cached pages, and a log line per blocked call naming the family and rule. Add benchmarks/hle_diamond/opencode_hle_tools.yaml, the OpenCode preset with this server as its only tool server, document it and the residual Tavily-backend difference in the Diamond README, and extend the Diamond preset tests to it. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(hle_web_tools): refuse web_search from a session with no seeded question The plain POST /web_search route stays served next to /mcp under Gym's MCP exposure, and a call without the rollout's cookie or token lands in a fresh session that carries no question, so the verbatim-span rule was silently skipped (probe: the same 8-word query refused with the rollout cookie, served with no cookie and no token). Fail closed instead: web_search refuses any session that was not seeded with a question under the new require_seeded_question flag (default on), logged as family=unseeded_session; a seed without a question says its searches will be refused. The other families already applied on the direct route, and web_fetch has no session-dependent rule. Add the model-free test (seed, then a cookie-less POST /web_search is refused and the backend is never called; the flag off restores the old behaviour), and reword the module docstring and both READMEs to describe the enforced behaviour rather than claiming the agent reaches the server only over MCP. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): refuse re-seeding a seeded session with another question A session now keeps the question it was seeded with: a retried seed with the same words is served again, a seed that would change or drop the question answers HTTP 409 (ResourcesSessionConflictError), so a caller holding a session cannot swap the verbatim-span check out from under it. QuestionSpans keeps a SHA-256 of the normalized words, never the text, to tell the two apart. This does not close the fresh-session path: /seed_session is unauthenticated and network_access model_and_tools admits the whole tool-server host, so a sandbox can seed a new session with a question of its choice and search from it without the span rule. The server README and the opencode_reference.yaml header now state that residual and what closing it needs (an authenticated seed channel the sandbox never sees). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> (cherry picked from commit 27187af4bc522d498ca64039b14570f41434aae2) * fix(hle_web_tools): import nemo_gym names from their new paths main moved the modules this server imported (nemo_gym.base_resources_server, nemo_gym.server_utils names, nemo_gym.mcp_auto_exposure) and serves the old paths only through deprecation shims removed in nemo_gym 0.10.0. Rewritten with scripts/migrate_imports.py to the public API (nemo_gym.client, nemo_gym.server_runtime) and, for the MCP test, the defining module, as the other servers on main do; the README names the new request path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * docs(hle_diamond): say hle_web_tools exists as its own preset opencode_reference.yaml and the README's Reference preset section, both from main (#5177), said the reference preset would switch to the hle_web_tools server once it existed. This branch adds that server, so the sentence would turn into an unfulfilled promise on merge. The reference preset keeps the Tavily tools here; the wording now names opencode_hle_tools as the preset that runs hle_web_tools (on the science image) and leaves running it on the reference image and egress policy to a follow-up change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): end the Tavily call when every key is rejected A key pool whose keys Tavily all rejects (401, 432, 433) spun forever in the key-selection loop without an await, freezing the worker's event loop: one configured key, a key listed twice, or two distinct rejected keys were enough. A rejected key also spent the transient retry budget, so [bad, bad, bad, good] with three attempts never reached the good key. The pool is now deduplicated when the backend is built, rejected keys move on without spending an attempt, the next key comes from a bounded helper over the keys still accepted, and the call raises a RuntimeError naming the status once every key is rejected. Transient failures keep their backoff and budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): turn malformed Tavily bodies into tool errors web_search read Tavily's payload unguarded: a body that is not an object, `results: null`, a non-object item or a non-string title or content raised AttributeError or TypeError and reached the model as an HTTP 500. web_fetch had the same `results: null` gap and rendered a non-string raw_content as page text. Both tools now read the results list through one guarded helper. web_search answers a malformed body with a "Search failed" tool error, skips non-object items and treats non-string title and content as empty; web_fetch reports any malformed body, item or raw_content as "Fetch failed", and guards failed_results the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): add the example data CI validates CI's changed-server step runs ng_test with should_validate_data=true, which requires data/example.jsonl (5 rows), data/example_metrics.json and data/example_rollouts.jsonl, plus an agent with an example dataset in the server config; hle_web_tools had none of them. Following resources_servers/tavily_search: the config gains a simple_agent with an example dataset behind a single_agent_turn_legacy environment server over direct HTTP, example.jsonl holds 5 synthetic general-knowledge rows (no HLE text, each marked by a placeholder field) whose tools are web_search and web_fetch with the server's own schemas, example_metrics.json comes from gym dataset collate in example_validation mode, and example_rollouts.jsonl holds placeholder rows since no live model or Tavily call was made. The server stays verified: false. The README says so, and the update-readme-table hook adds the server to the environment list now that its config has an agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * docs(hle_web_tools): say the server must run single-worker Seeded questions and the page cache are per process, so with num_workers > 1 a rollout's seed and its searches could land in different workers; the module also defines no worker app, so such a launch fails to start. Rather than add the multi-worker entrypoint footer, the README states the single-worker requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): refuse URL spellings the policy could misread The URL check certified URLs it had not fully checked and matched the page-level families on the raw string only. Now url_block: - refuses as malformed_url any URL longer than 4,096 characters, with a backslash, whitespace or control character, or with any @ in the authority (an empty https://@host included); - fails closed when candidate expansion stops with candidates unexplored, at the 64-candidate cap or after the original plus four rounds, so 63 decoy query values or a fourth decoded form no longer hide a target; - runs url_prefixes and serp_url_patterns on a canonical form as well (IDNA lower-cased host, trailing dot, userinfo and default port dropped, dot segments resolved), so :443, a trailing dot or /x/../ no longer evade a prefix; - treats every *.archive.org host as a wrapper and reads https:/, https:/// and https: remainders as https://; - reads wrapped and decoded URLs the way WHATWG fetchers do (backslash as /, tab and newlines deleted); - takes each embedded http(s):/ up to the next one instead of through a lookahead to the end of the string, so extraction is linear. site: operands are read as host characters only, so (site:a OR site:b), site:a|site:b and site:b, yield bare hosts, and an operand whose IDNA encoding fails is refused instead of skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): check the whole fetched page before it is cut Fetched content was sanitized and cut at max_cached_page_chars before the content families ran, so a blocked span past 500,000 characters was served and cached, and a cache cap below max_fetch_chars served a cut page as truncated=False. Now: - a fresh page is checked against url_patterns and patterns_case_sensitive on the provider's raw Markdown, before data URIs are removed or anything is cut, and on the sanitized Markdown; a refused page is never cached; - the scan is bounded: a page longer than max_page_chars (new, default 1,000,000) is refused unread as oversized_page, and the regex work runs in a worker thread instead of on the event loop; - max_cached_page_chars no longer cuts the page: a longer page is served whole but not cached, so cache-hit rechecks see the complete sanitized Markdown; - data URIs with media-type parameters or a percent-encoded payload are removed too, the payload stops at the Markdown destination's end so following prose is kept, and every repetition is bounded so the regex stays linear (the unbounded suggested form is quadratic); - the heading fallback for the title is linear on long blank runs (100,000 characters took 42 s) and drops a closing # sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): bind typed sessions to their episode and fence closes A typed seed (an Environment Server's resources_session_id, episode_id, task_id and task_data) kept only the question, so the same ID could be seeded or closed for another episode or task, and a seed replayed after a close recreated a searchable session. Typed seeds are now validated as ResourcesSeedSessionRequest, their episode and task are recorded next to the question, and seed and close go through ResourcesSessionFence, as in example_single_tool_call: a conflicting seed or close answers 409, and a seed after its session closed is refused for the retry window. The identity map shares the 10,000-session cap with the questions and is evicted with them. Legacy Agent seeds are unchanged. The test runs check_resources_session_contract with keeps_state=True. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): treat a question with no words as no question A seed whose question has no word characters ("?!", or an empty user message) was stored as seeded with span_words=0, which disabled the verbatim-span rule while require_seeded_question still saw a seeded session. extract_question now returns None for such text (the question field falls back to the last user message first), so the session stays unseeded and its searches fail closed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): log a warning when the session cap forgets a question Past MAX_SESSION_QUESTIONS (10,000) seeded sessions the least recently seeded one was dropped silently, and its later searches were refused as unseeded_session with nothing pointing at the cause. The eviction now logs a WARNING naming the session and the cap. The test lowers the cap to one and also checks that the question and the episode binding leave together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * docs(hle_web_tools): record the MIT license of the vendored blocklist blocklist.json is an unmodified copy of docs/blocklist.json from centerforaisafety/hle at 22ed3074b1e7b134bcbc09028d0ba320839b0655, which is MIT (Copyright (c) 2025 centerforaisafety). The JSON stays byte-exact for its digest check, so the upstream LICENSE at that revision is reproduced unchanged in the sibling LICENSE-HLE, the file gets a row and a note under Vendored Components in ATTRIBUTIONS.md, and the server README gains a Licensing section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * docs(hle_diamond): name the sandbox-seeded session residual Both READMEs called the Tavily backend the one residual difference from upstream's wrapper, and the Diamond section said a session not seeded with the question is refused "so the span rule cannot be skipped". Both overstate the policy: /seed_session is unauthenticated, so a sandbox process can seed a fresh session with a question of its choice and search from it; the 409 on re-seeding covers existing sessions only, and the session and MCP-token signing key derives from public config. The server README now names that residual in its own paragraph, the Diamond section lists it next to the backend as the second residual, the fail-closed sentences are scoped to unseeded sessions, and the server README's dangling pointer to the Diamond "Reference preset" section now points at "Upstream web tools", where the residual is listed. Closing it needs an authenticated seed channel; both READMEs carry the marker TRACKING_ISSUE_TBD for the tracking issue, to be replaced by its number before push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> * fix(hle_web_tools): declare the task-data schema CI validates The core unit tests require every resources server to ship a task_data.py, and the committed example rows to validate against it. hle_web_tools had none, so TestSchemaPresence and TestRepoDataMatchesSchemas failed. Declare both keys the example rows carry: the optional question that /seed_session reads, and the placeholder note that marks the rows as synthetic. Keeping the note in the data makes each row say what it is; the schema records that the server never reads it. tests/unit_tests/test_task_data.py: 2 failed, 421 passed before; 425 passed after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): map hosts the way UTS #46 fetchers do normalize_hostname used Python's IDNA 2003 codec, which leaves code points unassigned in Unicode 3.2 unmapped. Browsers (WHATWG URL) and yarl, which aiohttp uses, map a host with UTS #46 first, so https://sc\U0001f130le.com/leaderboard was allowed by the policy but fetched as scale.com. The same held for lastexam.ai and arxiv.org spelled with such letters. Map a non-ASCII host with idna.uts46_remap (non-transitional, no STD3 rules) before the existing encode, and declare idna, already installed through aiohttp and yarl, as a direct requirement. IDN hosts that are ordinary (例え.jp) keep their punycode form. Test: one row in the URL refusal table; it fails before this change (allowed, extract awaited) and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): refuse a wrapped URL Python cannot parse url_shape_problem refuses an unparseable URL only at the outer level. For a wrapped or decoded candidate, _single_url_block returned None when urlsplit or IDNA raised, which skipped every family for it. Python rejects https://[::1]@scale.com/leaderboard ("'scale.com' does not appear to be an IPv4 or IPv6 address"), while browsers, httpx and urllib3 read it as host scale.com, so a redirector carrying it in ?u= was served. Refuse such a candidate as malformed_url instead. The realistic URL list from review (48 URLs) shows no new refusal. Test: one row in the URL refusal table; fails before (served), passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): read a wrapper's remainder as a URL however it is spelled A scheme-less remainder became a candidate only when it matched an ASCII-only host regex, which rejects a trailing dot after the TLD, userinfo and full-width or ideographic-dot hosts. So web.archive.org/web/2024/scale.com./leaderboard and web.archive.org/web/2024/x@scale.com/leaderboard were served, as was ?u=scale.com./leaderboard on a redirector. A wrapper fetches its remainder (and Google's cache its cache: target) whatever the spelling, so read each as https://<remainder> and let urlsplit and normalize_hostname decide the host. Other query values still need a scheme or a leading hostname, which may now end with a dot. The README also records the malformed_url refusal of an unparseable wrapped URL from the previous commit. Test: three rows in the URL refusal table; all three fail before (served) and pass after. The realistic URL list from review shows no new refusal. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): match url_prefixes with a query in any parameter order The eight openreview.net prefixes carry ?id=<paper>. As plain string prefixes they match one exact parameter order, so openreview.net/pdf?&id=46UGfq8kMI and openreview.net/pdf?x=1&id=... were served although a server reads both as pdf?id=46UGfq8kMI, and the id appears in no url_patterns rule. Keep the string-prefix check and add a second reading for a prefix with a query: its path must prefix the URL's path, and each of its key=value pairs must be among the URL's parsed query pairs (a value as a prefix), in any order. This only adds matches to the upstream semantics. Test: two rows in the URL refusal table; both fail before (served) and pass after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): collapse slash runs in the canonical URL form remove_dot_segments kept // runs, so arxiv.org//abs/2605.02442, arxiv.org/abs//2605.02442, arxiv.org/%2Fabs/... and the Wayback form of the first were served, and 2605.02442 has no url_patterns rule to catch them. Servers with nginx's default merge_slashes serve such paths as the single-slash page. Collapse runs of / in the canonical form. It is used for prefix and SERP matching only; the fetched URL is unchanged. Not changed: export.arxiv.org and /format/, /src/ and /e-print/ paths of 2605.02442 are gaps in the pinned upstream list, not in its matching. Test: one row in the URL refusal table; fails before, passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): fold Unicode presentation variants before query checks question_words ran \w+ over str.casefold() with no normalization, so a query that repeats 8 words of the question in full-width or mathematical letters, with a zero-width space or soft hyphen inside a word, with a combining accent, or joined by underscores was served. The query patterns had the same gap: humanity<U+00AD>s last exam was not matched. Search engines fold all of these back to the plain words. Words are now runs of letters and digits ([^\W_]+) after NFKC with format characters (Cf) removed and combining marks stripped; the question and the query go through the same function. url_patterns, patterns_case_sensitive and site: operands are checked on the query as written and as folded. Look-alike letters from another script (Cyrillic o) are still not folded; the README says so. Test: test_presentation_variants_of_a_blocked_query_are_refused, five rows, one per mechanism (NFKC, Cf, combining mark, underscore, a pattern with a soft hyphen); all five fail before and pass after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(hle_web_tools): apply ruff-format to the full-width span row The previous commit was made before ruff-format wrapped this row; no behavior change. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): bound title work and cap shown titles Result titles come from the provider with no length bound, and _TITLE_SEPARATOR_RE (\s+ before a separator) backtracked over every blank of a run, quadratic in its length: a 20,000-blank title took 3.9 s and a 50,000-blank one 26.6 s of web_search on the event loop. Each separator also added a variant up to the full title length, so a title of 'a | ' * 2,000 cost 1.1 s. Start the separator match only at the first blank of a run, and cut a title longer than 1,000 characters (with an ellipsis) before it is shown or checked against answer_title_patterns. url_patterns and patterns_case_sensitive still see a result's whole title, which is linear. The same cap applies to the fetched page's Title: line. That title, taken from the heading when the provider sends none, was not counted against max_fetch_chars, so one fetch could return about 960,000 characters. Tests: test_long_result_titles_are_checked_in_linear_time (a blank run and a separator chain, both 20,000 characters) fails before (about 5 s) and passes after; the 60,000-character cap test gains a heading-derived title case, which fails before (the 100,000-character title is shown whole) and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): answer a non-JSON 200 body as a tool error 5142f52df turned a decoded body of the wrong shape into a tool error, but a 200 whose body does not decode still escaped: a gateway's HTML page raised aiohttp's ContentTypeError and a cut-off JSON body raised JSONDecodeError, and both reached the caller as a server error. Return None from the transport for such a body, so the existing "Search failed: ... malformed response" and "Fetch failed for <url>" paths answer it. It is not retried: a 200 is not a transient status. Test: two rows (HTML body, cut-off JSON) in the parametrized malformed-payload test, through the real transport with a stubbed request; both fail before (the exception propagates) and pass after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(hle_web_tools): make the retry-budget test fail when a rejection spends it test_rejected_keys_do_not_spend_the_retry_budget only bounded the total number of requests, which the pre-fix code also met. Two real regressions survived it: a rejected key spending an attempt, and key selection that no longer wraps round to the current key. Replace it with the mixed case: keys key-one, key-two and two attempts; Tavily answers 401, 429, 200. The call must succeed after trying key-one, key-two, key-two with one backoff sleep. Fails on app.py from 498e34bc3 (before eb466ecb9: "Tavily HTTP 429 on /search after 2 attempts"), on the spend mutant (same error) and on the no-wrap mutant (TypeError); passes on HEAD. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): fence a typed session the cap forgets Untyped Agent seeds share the 10,000-session cap with typed sessions. When eviction dropped a live typed session's episode binding, a later seed of that ID for another episode returned 200 and bound it, where the resources-session contract requires 409; the session's question was already gone. When the cap forgets a typed session, mark its ID closed by its bound episode in the session fence. A later seed of the ID then gets 409, and a close from its own episode is still accepted. The README states that the cap must exceed the peak number of open rollouts. Test: test_sessions_past_the_cap_are_forgotten_with_a_warning now seeds the forgotten ID for another episode and expects 409; it fails before (DID NOT RAISE) and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(hle_web_tools): cover the live-session close conflict and the wordless fallback Three regressions survived the session tests: - a close naming another episode of a live typed session was not checked (the contract checks it only for a closed session), so disabling that 409 deleted the session's question unnoticed; - holds_state looked only at the question table, so a close that kept the episode binding passed; - the wordless-question row used the question field, which either of the two guards refuses alone, so dropping the extract_question gate for the last-user-message fallback passed. The contract test now counts the binding as held state and closes a live session with another episode (409, question and binding kept). The wordless case also seeds a last user message of "?!" with no question field. Fails on: the close-conflict mutant (200 != 409), the close-keeps-binding mutant ("close left state"), the gate mutant and app.py from 16cd8c0f5 (wordless message seeded); passes on HEAD. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): refuse a site: operand that is not hostname-shaped The README said a site: operand that is not a valid hostname is refused, but only an operand whose IDNA encoding failed was: site:huggingface.co! and site:exa_mple!.org reached Tavily. Results were still filtered by domain, so nothing leaked, but the refusal and its audit log line that the protocol asks for were missing. Drop trailing punctuation after normalize_hostname (site:huggingface.co! is huggingface.co, refused as domains), and refuse as malformed_url an operand that is then not labels of letters, digits, - and _, optionally after *. Test: two rows in the site-operand table; both fail before (served) and pass after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): strip data URIs that omit the media type RFC 2397 lets a data URI omit its media type (data:;base64,...), and _DATA_URI_RE required one, so such a payload was served whole. Make the media type optional. The README now states the bounds that already applied: at most 16 media-type parameters, and the payload ends at the first whitespace, quote, ) or >, so a base64 payload wrapped over several lines keeps its later lines. Test: a data:;base64, image in the existing data-URI test page; fails before (payload kept) and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): refuse a multi-worker launch with the standard hook c4262af29 documented that the server must run single-worker, but the server did not use SimpleServer.multiple_workers_unsupported_reason, so num_workers=2 passed refuse_unsupported_num_workers and dry runs, and failed only later as uvicorn's 'Attribute "app" not found'. Return the reason (seeded questions, session bindings and the page cache live in one process), so run_webserver refuses num_workers > 1 before uvicorn starts with "hle_tools needs num_workers=1: ...". The README sentence now says so. Test: test_a_launch_with_several_workers_is_refused fails before (DID NOT RAISE) and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): run page checks in worker processes 465d43de4 moved the page work to asyncio.to_thread, but Python's re keeps the GIL for a whole search() or sub() call, so the event loop still stopped for the length of one scan. The same commit widened the scan to the provider's raw page of up to 1,000,000 characters: with a stubbed page of 999,000 characters of 抱抱脸 a 1 ms heartbeat saw a 4.0 s gap (答案: 3.5 s), from the pinned upstream hugging-face/answer-key url_patterns pair. A cache hit rescans up to 500,000 characters the same way. Run build_page and the content families, fresh and on a cache hit, in a module-level ProcessPoolExecutor of two spawn workers; the final-URL check stays on the loop and runs first. The Blocklist is passed with each call (its patterns pickle as source and recompile from re's cache), so a policy changed after start still applies. A pool whose worker died is replaced on the next call. spawn needs the entrypoint to keep its __main__ guard, which app.py has; with app.py as __main__ the workers resolve the functions through __mp_main__ (checked with a copy of app.py run as a script). After: the same probe shows a 0.02 s maximum gap for both pages. The README claim about a worker thread is corrected. Test: test_page_checks_do_not_stall_the_event_loop (300,000 characters of 抱抱脸, heartbeat gap < 0.5 s) fails before (1.25 s gap) and passes after. The 100,000-character heading timing case now warms the pool first, so it does not time the start of the worker processes. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_diamond): list the oversized-page refusal as a third residual The Diamond README said two residual differences from upstream's wrapper remain (the Tavily backend and sandbox-seeded sessions). A third was not disclosed: the protocol caps web_fetch output at 60,000 characters with a truncation indicator, so upstream serves the start of any page, while this server checks the whole page and refuses one longer than max_page_chars (1,000,000 characters) as oversized_page. A clean 1,040,000-character page is refused here and would be served cut upstream; long PDFs reach that size. List it as the third residual a with-tools number should state, and point to it from the server README. Also qualify the server README's redirect claim: the final-URL check relies on Tavily reporting the post-redirect URL in results[].url, which is verified only against a stub, not the live API. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): read format characters both as removed and as a space 4abb964f4 removed format characters (Cf) before splitting a query into words. That catches sp<ZWSP>in, but it glued words joined by a word joiner or a zero-width space: how<U+2060>many<U+2060>odd... became one word and was served, although the span rule refused it before that commit. Whether a search engine drops such a character or splits words at it is not known. Read the text both ways: the question's n-grams come from both readings, a query is checked in both, and the query patterns and site: operands see the as-written, removed and space readings. Test: a word-joiner-separated row in test_presentation_variants_of_a_blocked_query_are_refused; it passes on 2500e8539, fails on 4abb964f4..77d3f0f5a, and passes after. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): import test helpers from their moved nemo_gym paths Main moved nemo_gym.server_utils to nemo_gym.server_runtime.base_server and nemo_gym.testing to nemo_gym.server_runtime.testing (both since 0.8.0, old paths removed in 0.10.0). The tests imported refuse_unsupported_num_workers and check_resources_session_contract through the old paths, so importing them raised the nemo_gym DeprecationWarning. Rewritten with scripts/migrate_imports.py, imports merged by ruff; the README's test description names the new module path. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): pass the core public-API and environment-resources tests Two core unit tests that arrived or changed with the origin/main merge (69c748d082) failed on this branch's files: - test_public_api_facades flagged tests/test_app.py importing maybe_auto_expose from nemo_gym.server_runtime.mcp_auto_exposure; it is exported from nemo_gym.server_runtime. - test_repository_step_environments_declare_their_resources (#5108) flagged hle_web_tools_environment_server, whose single_agent_turn_legacy block named no resources_server. It now names hle_web_tools, as the tavily_search configs do. Before: the two tests, 2 failed. After: tests/unit_tests/ test_public_api_facades.py and test_add_legacy_agent_environment_servers.py 917 passed; resources_servers/hle_web_tools/tests 95 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(hle_web_tools): pin key rotation after a 429 and the release of every response Review thread 4398612291 asked that a 429 followed by a 200 still moves to the next key with backoff, and that every response is released. Both hold, but no test pinned either: a mutant that retries the same key after a 429, and one that releases only 200 responses, each passed the whole Tavily transport selection (22 passed). test_rejected_key_fails_over_and_provider_bodies_stay_private now runs for 401 (no backoff) and 429 (one backoff of 1 s), and asserts that both responses are released. Before (HEAD test, -k 'key or tavily or retry or twenty or malformed'): norotate 22 passed; release_only_200 22 passed. After: norotate fails [429-backoff1]; release_only_200 fails [401-backoff0] and [429-backoff1]; unmutated code passes. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(hle_web_tools): refuse a typed re-seed of a live session for another task Review thread 4398612319 listed, as an acceptance check, that a typed seed of the same resources_session_id with a changed task gets 409. The server already refuses it, but neither check_resources_session_contract nor this file's test changed only the task, so a mutant that compares only the episode (bound[0] != identity[0]) passed the whole file. test_typed_sessions_follow_the_resources_session_contract now seeds the live session again with task_id 2 and asserts 409, with the session's question digest and binding unchanged. Before (HEAD test file): mutant episode_only, 95 passed. After: mutant episode_only fails this test; unmutated code, 96 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): read a Wayback path with no timestamp as naming its target Follow-up to review thread 4405604682 (Wayback spellings). The Wayback unwrapping took the first segment after /web/ as the timestamp, so web.archive.org/web/scale.com/leaderboard was read as https://leaderboard and web.archive.org/web/scale.com as https://web/scale.com, and both were fetched. archive.org redirects such a path to a snapshot of the target (web/scale.com/leaderboard answers 302 to /web/<timestamp>/https://scale.com/leaderboard). The first segment is now read both ways: as a timestamp, and as the start of the URL. A timestamped path keeps its existing reading, so every spelling refused before is still refused; the extra reading only adds a candidate. Before (HEAD app.py, new URL-table rows): web/scale.com/leaderboard and web/scale.com fail (allowed). After: both refused as domains; the timestamped, id_, 2024* and Wikipedia rows unchanged; 98 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): read typographic quotes around a site: operand like ASCII quotes Follow-up to review thread 4405604730 (site: operand parsing). Typographic quotes and guillemets were neither openers nor excluded from the operand, and NFKC does not fold them, so site:“scale.com” became the punycode host xn--scale-hw3b.xn--com-9o0a, which is hostname-shaped and matches no domain: the query was served with no refusal and no audit line. A trailing ” survived as well (site:scale.com” became scale.xn--com-9o0a), since trailing punctuation is stripped after IDNA encoding. The operand regex now treats ‘ ’ ‚ ‛ “ ” „ ‟ « » ‹ › like " and ': they may open the operand and end it. The opener run is now repeatable, so stacked openers (site:((scale.com)), site:«“scale.com”») no longer stop the match. Before (HEAD app.py, new rows): site:“scale.com”, site:scale.com” and site:((scale.com)) are served. After: each is refused as domains; site:example.org, site:“example.org” and site:en.wikipedia.org/wiki/Foo_(bar) are still served; 101 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): fold every default-ignorable character and read ZWSP as a word break Follow-up to review thread 4398612299 (span-rule folding, e075400a3c). Two spellings still got an 8-word question span or a blocked query past both readings of format characters: - Mixing a zero-width space between words with a soft hyphen inside one ("How<ZWSP>many ... no<SHY>nvanishing", "H<SHY>LE<ZWSP>leaderboard"): each reading treats every format character the same way, so each one breaks. Unicode word segmentation (UAX #29) splits at U+200B and ignores U+00AD, and reads both as [how, many, ..., nonvanishing]. - Hangul fillers (U+3164, U+115F, U+FFA0; default-ignorable, category Lo) inside a word ("no<HF>nvanishing", "Hum<HF>anity's Last Exam", "H<HF>LE leaderboard"): only category Cf was folded, and NFKC maps the filler to U+1160, still a letter. fold_text now treats every Default_Ignorable_Code_Point (ranges from DerivedCoreProperties, since unicodedata has no such property) like a format character, and the checks read text three ways: all removed, all as a space, and as word segmentation does (U+200B a space, the rest removed). The readings apply to the question n-grams, the query's words, and the query forms the text families and site: operands see. Before (HEAD app.py, new rows in the presentation-variant table): the four rows mixed-format-characters, mixed-format-pattern, hangul-filler and hangul-filler-pattern are served. After: each is refused with its family; 105 passed. Seeding a 2.24 MB non-ASCII question takes 1.6 s instead of 1.1 s (ASCII: 0.49 s, unchanged); unbounded seed size is already part of the unauthenticated-seed residual. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): also match the canonical form with slash runs merged before dots Follow-up to review thread 4405604669 (equivalent spellings, 25f0c7f83b). The canonical form resolved dot segments and then collapsed slash runs, so arxiv.org/abs/x//%2e%2e/2605.02442 became arxiv.org/abs/x/2605.02442 and was fetched. httpx and urllib3 send that path unchanged, and nginx with its default merge_slashes collapses the run first and then resolves the dots: a throwaway nginx 1.26.3 serves /abs/2605.02442 for it (whether arxiv's front end does the same is not verified). The URL-shaped families now see both orders: dots then slashes (as before) and slashes then dots (nginx). Before (HEAD app.py, new URL-table row): arxiv.org/abs/x//%2e%2e/2605.02442 is allowed. After: refused as url_prefixes; arxiv.org/abs/x/2605.02442 and example.org/a//../b stay allowed; 106 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): find a title's site-name separator after a long blank run Follow-up to review thread 4405604713 (e10df7a73e capped shown titles). The answer-title check ran on the title already cut at 1,000 characters, so a result titled "Quiz answers" + 5,000 blanks + "| Site" lost its separator to the cut and was kept, although the model is shown "Quiz answers ...". Before the cap it was dropped. title_variants now takes the whole title: the title as shown, plus the title up to each separator that starts within the first 1,000 characters, however long the separator's blank run. Each variant stays at most 1,000 characters, so the check stays linear (1,000,000-character titles: under 0.04 s). An answer phrase that lies wholly past the cut is neither shown nor checked; the README says so. Before (HEAD app.py, new row padded-separator): kept. After: dropped; 107 passed. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): state when the session cap forgets a running rollout Review threads 4398612339 (LRU eviction) and 5680046430 (HOLD summary). The README said the cap "must exceed the peak number of rollouts open at once". That holds only for a harness that closes its sessions: OpenCode seeds with its run body and never closes, so held sessions add up, and a running rollout's question is forgotten once more than 10,000 later seeds reach the server, however few rollouts are open at once (probe with the cap at 2: A forgotten after B and C seed, with at most 2 open). The README also said a seeded session keeps its question "for its whole life", but only typed seeds check the fence: a forgotten Agent (untyped) session, or a closed typed ID, takes a later untyped seed through its cookie with another question (HTTP 200). The README and the comment in _install_question now say so, and that this, like an unauthenticated seed flood past the cap that makes other rollouts' searches fail closed, is the same unauthenticated-seed residual as a sandbox-seeded fresh session. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): state every bound of the data-URI regex Review thread 4398612357 (data URIs). The README named the 16-parameter bound only, but the regex also caps the type and subtype at 127 characters, each parameter name at 127 and each value at 256: a data URI with one parameter value of 257 characters is not matched and is served whole. The bounds are what keep the scan linear, so the code is unchanged; the README and the regex comment now state them and that a URI past them is left in place. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): state the page-check pool's cost and what still runs on the loop The README described the page-check pool (b132980c82) without its cost or its scope. Measured at this head, with the Tavily transport stubbed: - The pool is created on the first page check and starts workers on demand, up to two. Each worker imports the server module afresh and sits at about 240 MB RSS, the size of the server process, and the first fetch takes about 1.8 s against 0.002 s afterwards. - web_search's per-result URL, title and snippet checks still run on the event loop: one result whose title (or snippet) is 1,000,000 characters of 抱抱脸 holds the loop 4.2-4.4 s. Whether Tavily ever returns a title or snippet that long is not verified. Both are now stated next to the pool description; the code is unchanged. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): correct the configuration and preset notes - The configuration table omitted tavily_base_url, the one config field it did not list. - The quoted multi-worker refusal hard-coded hle_tools, the Diamond preset's instance name; refuse_unsupported_num_workers prints the instance's own name (hle_web_tools for configs/hle_web_tools.yaml). - "The checks below need TAVILY_API_KEY" pointed at Example data and Tests, which are model-free and pass with the key unset; the key is needed for live runs of the preset. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_diamond): name cap-forgotten sessions in the sandbox-seeded residual The Diamond README's residual said the 409 on re-seeding does not cover a new session. It does not cover a session the server's cap has forgotten either: an untyped seed through that session's cookie installs another question (HTTP 200). The server README says so since 0cd8d9c4e; this keeps the two residual descriptions consistent. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): use the split session locks and fences from main Main's #5083 split ResourcesSessionFence into ResourcesSessionLocks, which runs a session's seed and close one at a time, and ResourcesSessionFences, which refuses a seed after its close until the close retry window ends. The old name no longer exists, so the server failed to import after the merge. Map each call onto the new API without changing behaviour: locked() on the locks, and raise_if_fenced(), get() and fence() on the fences in place of raise_if_closed(), closed_episode() and mark_closed(). The base config already requires a positive close retry window, so the new constructor's check adds no refusal. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(hle_web_tools): declare dependencies in pyproject.toml and uv.lock Main's #4837 moved every component from requirements.txt to a committed uv project and lock, and its component dependency tests now refuse a server that still ships requirements.txt or lacks either file. Move the server's two requirements, nemo-gym[dev] and idna>=3.7, into a pyproject.toml that copies the shared resolver policy, as scripts/component_locks.py expects. The lock was seeded from the calendar server's lock, which has the same Gym dependencies, so it pins the same versions as the other component locks (idna 3.10, mcp 1.29.0) instead of a fresh resolve that would move mcp, fastapi and others to new major or minor releases. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): read a Wayback timestamp segment as a timestamp only Since the timestamp-less Wayback reading was added, every /web/<first>/... path was read both ways, so an ordinary timestamped snapshot also yielded the candidate https://<timestamp>/<url>, which names no page. Each Wayback layer then cost one to three extra candidates and more decode rounds, and a URL at the edge of MAX_URL_CANDIDATES or MAX_URL_DECODE_ROUNDS was refused as malformed_url where it was served before (four timestamped layers around a percent-encoded page, or a snapshot of a page with thirty URL-valued query values). Add the second reading only when the first segment is not timestamp-shaped (digits with an optional two-letter modifier such as id_ and an optional star, or a bare star), which no hostname is. /web/scale.com/x, /web/2024/..., /web/2024id_/... and /web/*/... keep their readings. Candidate counts return to what they were before the second reading existed: two for a plain timestamped snapshot. A URL-table row pins the four-layer case as served. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): protect a short question whole in every ignorable reading QuestionSpans took its span length from the "removed" reading of the question but built n-grams of that length from the "space" and "word_break" readings too. A question under the threshold whose ignorable characters (a variation selector, a zero-width space, a Hangul filler) sit inside a word has more words in the "space" reading than in the "removed" one, so fragments of a single word became blocked n-grams and a query that merely contained such a fragment as a word of its own was refused, against the README's "protected as a whole". No local HLE-Diamond question has fewer than 8 words or readings that differ, so this is a correctness note. Give each reading its own span length min(threshold, words in that reading) and check the distinct lengths longest first, returning the matched length so the refusal rule and the log name the span that actually matched rather than the "removed" reading's length. A test seeds a one-word question with an ignorable character inside the word in two spellings, serves a query repeating one fragment, and refuses the whole question in each spelling with the matched length in the log. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hle_web_tools): check the shown title text at the cut title_variants tried the capped title (with its cut marker) and the title before each separator that starts within the first MAX_TITLE_CHARS characters. A separator starting exactly at the cut was excluded, and the cut marker made the end-anchored answer_title_patterns miss a phrase that ends exactly there, so a title whose text before " | Site" is exactly 1,000 characters and ends in "Quiz answers" was kept and rendered as shown, while the same title one character shorter was dropped. Add the shown text without its cut marker as a variant whenever the title is capped, so the check covers exactly what the model sees. Every variant stays at most MAX_TITLE_CHARS long, so the linear bound holds. A row in the title table pins the separator-at-cut case as dropped. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): count the later sessions that evict a rollout exactly The eviction loop pops the oldest session as soon as the table holds MAX_SESSION_QUESTIONS + 1 entries, so a running rollout's question is gone when the 10,000th later session is seeded, not after "more than 10,000 later seeds". A retried seed of a held session only moves it to the end, so the count is of distinct later sessions, not of seed requests. The log line ("more than %d seeded sessions are held") was already right. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): describe the word-break reading as the server's own The README and the app comment attributed the third ignorable reading to UAX #29 word segmentation in a sentence that names Hangul fillers among the characters it removes. UAX #29 classes U+3164, U+FFA0, U+115F and U+1160 as ALetter, letters it keeps; only format and extend characters are skipped by WB4, and U+200B is a break. Say what the reading does and where it follows the standard, and that the fillers are dropped because NFKC_Casefold drops them. The range list is introduced as including points outside category Cf rather than lying outside it, since most of the listed ranges are Cf, and as listed with adjacent ranges merged. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(hle_web_tools): say why the stray-closer row exists, drop a duplicate row The site:scale.com” row is the only one that catches an implementation reading typographic quotes as a balanced pair only, so it stays, with a comment saying what it pins. The hangul-filler-pattern row never failed without hangul-filler (the set membership) or soft-hyphen-pattern (the pattern path's folding) failing too, since both paths call the same fold_text and text_block folds nothing itself, so it is dropped. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(hle_web_tools): use main's session terms and name the retire release path main's AGENTS.md (since #5083) asks session code, tests and docs to use the names in the key-terminology glossary and add no synonyms. The glossary retires "typed session", "typed seed" and "Environment Server session" for "episode session" and "episode seed", "untyped seed" for "agent-run seed", "retry window" for "close retry window", and "free" for "release". This branch kept the old names in comments, the 422 detail for a malformed episode seed, a test name and fixture ID, docstrings and the README; the merge adapted the code to the split locks and fences but not the words. Rename them throughout. The README also said only a /close_session frees a slot before the cap evicts it: the checkpoint controller's retire of an attempt (retire_session_state) releases every session that attempt seeded too, whatever seed created it, and does not fence the ID, so say so. The "" fixture is test data about data URIs and stays. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(hle_web_tools): wrap the nested Wayback fixture to the line limit Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(hle_web_tools): link the seed-channel residual to its tracking issue The unauthenticated /seed_session residual is now tracked in NVIDIA-NeMo/Gym#5845. Both READMEs point at it in place of the TRACKING_ISSUE_TBD placeholder. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(hle_web_tools): give the true reason the server refuses several workers The README and multiple_workers_unsupported_reason said a rollout's seed and its searches could reach different workers. Session routing (since #3905, and restated by #5709 in the add-benchmark skill) pins a resources request by its cookie session_id and an MCP call by its token sid to the owning worker, so session-ID-keyed state is not what stops a multi-worker launch. What does: app.py defines no module-level app for uvicorn workers to import, and the URL-keyed page cache and the page-check pool are per process. Reword both and make the refusal test assert the new reason and the absence of the old one. Behaviour is unchanged: num_workers > 1 is still refused before uvicorn starts. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(hle_web_tools): refresh uv.lock for main's openai 2.45.0 pin After merging origin/main, scripts/component_locks.py --jobs 8 reported "FAIL resources_servers/hle_web_tools: The lockfile at uv.lock needs to be updated" and ng_test failed at uv sync --locked for the same reason, because main's root pyproject moved openai from 2.44.0 to 2.45.0 (#5464). Regenerated with uv lock --project resources_servers/hle_web_tools; the only change is openai 2.44.0 -> 2.45.0. component_locks now passes 286/286. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(hle_diamond): run opencode_hle_tools as an opencode_agent session The upstream web tools preset extended opencode_legacy_agent.yaml and so ran opencode_sandboxed_agent behind the deprecated legacy_agent relay, the one text preset left on it after #5702 moved the others to agent sessions. It now extends opencode.yaml and overrides the opencode_agent instance the way opencode_search.yaml does for Tavily: - absolute_ip_server_types adds resources_servers, since the sandbox reaches the hle_tools server's MCP endpoint by address. - The environment note moves from the sandbox file /tmp/nemo-gym-environment.txt to the agent's system_prompt, and mcp_tool_timeout_seconds is 300 for a web_fetch of a long page or PDF. - The opencode_config reasoningEffort block goes: a session's opencode_config takes no model options, and the effort travels on the policy model server's extra_body like the other session presets. - tool_servers ([hle_tools]) and network_access (model_and_tools) stay; SingleAgentTurnEnvironmentServer reads the grant from the bound agent and requires expose_tools_over_mcp, which the server instance sets. sandbox_docker_local.yaml composes with the preset again (the override's hle_diamond_opencode_agent block targets opencode_agent); its note names no search backend now, since it serves the Tavily and the upstream web tools preset alike. Tests: the preset leaves LEGACY_PRESETS and joins the agent-session, reasoning-effort and local-Docker parametrisations; the config test asserts the session wiring and holds the agent equal to opencode_search.yaml's apart from tool_servers and system_prompt. No server code changes. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(hle_diamond): describe opencode_hle_tools as an agent-session preset The benchmark README listed opencode_hle_tools among the deprecated relay presets (Scope row, relay paragraphs, effort bullets, Known limitations, Reading sandboxed results, Agent sessions, Reference preset). It now sits with the session presets everywhere, and the Upstream web tools section replaces its Deprecated relay paragraph with the session wiring (episode seed, MCP grant, address exposure, MCP timeout, extra_body effort), a documented run command composed like the other presets (the Docker override composes again), and a note that the 2026-10-26 five-row Docker pilot ran on the relay harness, so no with-tools number is quoted until a pilot is rerun on the session harness. The residual-difference statements (backend, unauthenticated /seed_session under model_and_tools, Gym#5845, oversized pages) are unchanged. The hle_web_tools README's Use in a preset paragraph and the opencode_reference.yaml header say the same. Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>
What does this PR do?
Make the Environment Server own its Resources Server reference instead of discovering it from the agent at runtime. Configuration resolution follows a unique, consumed top-level resources rename; retained originals and copies keep their existing references, and ambiguous renames require an explicit env choice.
For example,
indic/math_500renamesmath_with_judgetoindic_math_500_math_with_judge_resources_server. The env reference now follows that rename even when the agent has no resources binding.indic/ifevalkeeps the English resources alongside an Indic copy, so its env explicitly selects the Indic verifier.resources_serverforsingle_agent_turnand its legacy adapter; use it consistently for sessions, tool grants, verification, cleanup and aggregation.Follows #4952, #4953 and #4954, now on main. No separate issue is needed for this requested follow-up to the existing migration design.
Update after #4954 merged (2026-11-02)
Four commits on top of
00a6819c8; no history was rewritten.1da5791a7merges main (773c056cd). feat(scripts): switch simple_agent relays to single_agent_turn_legacy #4952, chore(configs): move simple_agent configs off the legacy_agent relay #4953 and refactor(configs): retire tool_simulation_agent and non_executing_simple_agent into simple_agent #4954 landed as squashed commits, so this branch's copies of them conflict with main in 290 files. The merge result is this PR's own change,00a6819c8, applied to main. The one real conflict is the generator's comment aboveSWITCHED: the template keeps this PR's explicitresources_serverand chore(configs): move simple_agent configs off the legacy_agent relay #4953's timeout defaults, and the comment now describes only those defaults. The comment about falling back to the agent's resources server is gone with the fallback.1525c46e8adds four bindings. Four configs were switched tosingle_agent_turn_legacyon main after this PR generated its bindings, so they lacked the now-requiredresources_server:resources_servers/beam/configs/beam.yaml→beamresources_servers/mimo_music/configs/mimo_music.yaml→mimo_musicresources_servers/arena/configs/vision_arena_v1.yaml→vision_arena_v1benchmarks/vision_arena_v1/config.yaml→vision_arena_v1_benchmark_resources_server. This benchmark copies its Resources Server through_inherit_fromwhile the parent stays loaded, so the resolver keeps the inherited reference; the child has to override it.Each binding is the Resources Server its agent already uses. The three beam benchmark children inherit the right binding and need no change.
86a11b4fabinds renamed Resources Servers in benchmarks that compose. CI'sgenrm_evalsuite failedtest_mmrb, which loadsgenrm_mmrb1andgenrm_mmrb2together. Both renamegenrm_eval, and their environment servers relied on the resolver to follow the rename. With two renames in one run, the reference is ambiguous and the resolver raisesAmbiguousResourcesRenameError. On main the environment server asked the agent, so this worked. Five environment servers now bind explicitly:benchmarks/genrm_mmrb1/llm_as_a_judge.yaml→genrm_mmrb1_resources_serverbenchmarks/genrm_mmrb2/llm_as_a_judge.yaml→genrm_mmrb2_resources_serverbenchmarks/genrm_rewardbench2/config.yaml→genrm_rewardbench2_resources_serverbenchmarks/hle_diamond/config.yaml→hle_diamond_equivalence_llm_judge_resources_serverbenchmarks/indic/mbpp/config.yaml→indic_mbpp_resources_server, for consistency; it has no sibling that renames the same Resources Server today.genrm_rewardbench2andhle_diamondfailed the same way next to their sibling benchmarks, which CI does not load together. Each binding is the Resources Server its agent already uses.c4e6830cdimports the binding test's parser fromnemo_gym.config.config_typesandnemo_gym.config.global_configinstead of the moved paths (review). No other line this PR adds uses a path listed innemo_gym/_moved_paths.py.Validation on this head:
test_benchmarks,test_cliand the Indic tests: 837 passed. Server suites run from their own directories, as CI does:single_agent_turn_legacy23,simple_agent89,single_agent_turn127, all passed.--checkreports no refusal.genrm_eval's suite in its own environment: 47 passed. The binding, global config, generator, benchmark and Indic tests: 666 passed.indic/math_500is the one benchmark that reusesmath_with_judge_environment_serverinstead of defining its own. Loaded alone, it follows both renames, astest_indic_math_environment_follows_its_resources_renamechecks. Next to anothermath_with_judgebenchmark, the agent rename is ambiguous too, and composition falls back to alegacy_agentrelay, the same as on main. Giving it its own environment server is left for a follow-up.773c056cd) against this head, over all 328 configs with an environment server: 295 environment bindings, 0 changed and 0 unbound. The same 34 configs fail to parse on both sides.dda609f86. It merges cleanly, and its five new commits change no environment server config.Validation
Python 3.13.14:
491 tests passed; all-files hooks and 29 Fern link tests passed. Fern check reported zero errors and one warning: authenticated redirect checks were skipped because no Fern token was configured.
Additional actual-parser comparison against parent
50b0b521c6c048650cdde5acc0734639b92fd058: 1,066 configs checked, 916 pass on both revisions, 150 pre-existing failures unchanged, zero new failures and zero changed effective resources bindings across 306 resolved env bindings. This is a configuration compatibility audit, not full benchmark qualification.The identical resources-only rename probe raises
ServerRefNotFoundErroron the parent and resolves the env to the new resources name here, with the agent's resources binding stillnull. Independent review checked the resolver, runtime binding use and generator refusal logic.Preserved-name integration validation
Rebased onto the updated #4954. The generator retains both explicit Resources Server bindings and Gym's deletion-before-inheritance behavior, including
_copyand string aliases. HLE/APEX harness overrides preserve their original Environment Server names.On the integrated head
7fe55fde5, Python 3.13.14:1,184 tests passed, including the inherited composition/observation fixes and the
_copydeletion-order regression cases. All-files hooks, DCO trailers, and whitespace checks pass. The generator dry-run has no refusal (exit 1: two already-unmigrated Vision Arena relays remain eligible; 22 relays stay for their existing blockers).The six HLE/APEX Pi, OpenCode, and HLE search entrypoints were compared with the pre-#4952 baseline at all four updated stack layers: 30 actual-parser resolutions preserve the original Environment Server names and all effective server/sandbox settings. The integration checks above did not include inference; current-head real-model smoke is recorded below. Broader migration and model-qualification gaps remain.
This is focused integration validation. Full repository runtime CI remains outstanding; current-head representative model smoke is recorded below. The PR stays a draft.
Rollouts
Completed representative real-model HTTP smoke at current head
7fe55fde511c8c02dbbd4299cf805e91cdfc6649using the actual config parser, config-selected resources, simple agent, legacy single-agent env and InferenceProvider against a live model endpoint. Sampling: temperature 0, seed 103, max output 8192 tokens.benchmarks/indic/math_500/config.yamlbenchmarks/indic/ifeval/config.yamlAll four responses were nonempty and completed, with no failure fields or masking. Every traced HTTP call returned 200. Both agents ended with zero active sessions; application servers and the bounded smoke step terminated. Independent review checked raw rollouts, events, source hashes and cleanup receipts.
Source files were verified against the archived commit and loaded Gym module paths were checked. The pinned IndicIFEval checker matched all 30 reconstructed files with no extra Python files. The same four inputs also passed on original commit
2b26f8f; the repeat above followed the concurrent stack integration.Input SHA256:
fc9f51e6fa6d903216c00e7393933e84aae4164253a5a4bc33104a56e32c3984c37cc0abfb892463544010d98ef8e39f00a2c8fffabb4ea3a4d4e42427840e84Two setup attempts failed before their affected inference because the reused runtime lacked dependencies; task-local installation resolved them. Frozen IFEval inputs have recorded hashes/task IDs but no recorded upstream dataset revision. Both Math answers were boxed, and the Bengali-prompt IFEval responses were English (the selected checks do not require Bengali output). This verifies routing/lifecycle for these tasks; it does not establish unboxed extraction behavior, language quality, full CLI startup, tool use, full benchmark scores or pre-migration score parity.
Full runtime CI and parent-stack qualification remain pending; keep this PR as a draft.
Compatibility and benchmark impact
Custom single-agent env configs must now supply an explicit
resources_server; omitted ornullno longer falls back to the agent. Child configs selecting a copied/different resources instance must override the env binding. The shipped backfill preserves the effective bindings observed in the config audit; benchmark scores have not been requalified.Checklist
🤖 Generated with Claude Code