Skip to content

fix(env): own resources bindings and resolve inherited renames - #5108

Merged
weijiac0619 merged 17 commits into
mainfrom
codex/env-resources-bindings
Nov 4, 2026
Merged

weijiac0619 merged 17 commits into
mainfrom
codex/env-resources-bindings

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Stack (bottom → top): main → this PR → #5003 (DeepAgents). #4952, #4953 and #4954 are merged.

What does this PR do?

Make the Environment Server own its Resources Server reference instead of discovering it from the agent at runtime. Configuration resolution follows a unique, consumed top-level resources rename; retained originals and copies keep their existing references, and ambiguous renames require an explicit env choice.

For example, indic/math_500 renames math_with_judge to indic_math_500_math_with_judge_resources_server. The env reference now follows that rename even when the agent has no resources binding. indic/ifeval keeps the English resources alongside an Indic copy, so its env explicitly selects the Indic verifier.

  • Require resources_server for single_agent_turn and its legacy adapter; use it consistently for sessions, tool grants, verification, cleanup and aggregation.
  • Add 299 explicit bindings across 292 shipped YAML configs. Update the migration generator and refuse child overrides that would silently select different resources.
  • Cover copy/rename ambiguity, deleted non-server destinations, string aliases, explicit env overrides, and the real Indic configurations.

Follows #4952, #4953 and #4954, now on main. No separate issue is needed for this requested follow-up to the existing migration design.

Update after #4954 merged (2026-11-02)

Four commits on top of 00a6819c8; no history was rewritten.

  • 1da5791a7 merges main (773c056cd). feat(scripts): switch simple_agent relays to single_agent_turn_legacy #4952, chore(configs): move simple_agent configs off the legacy_agent relay #4953 and refactor(configs): retire tool_simulation_agent and non_executing_simple_agent into simple_agent #4954 landed as squashed commits, so this branch's copies of them conflict with main in 290 files. The merge result is this PR's own change, 00a6819c8, applied to main. The one real conflict is the generator's comment above SWITCHED: the template keeps this PR's explicit resources_server and chore(configs): move simple_agent configs off the legacy_agent relay #4953's timeout defaults, and the comment now describes only those defaults. The comment about falling back to the agent's resources server is gone with the fallback.

  • 1525c46e8 adds four bindings. Four configs were switched to single_agent_turn_legacy on main after this PR generated its bindings, so they lacked the now-required resources_server:

    • resources_servers/beam/configs/beam.yaml → beam
    • resources_servers/mimo_music/configs/mimo_music.yaml → mimo_music
    • resources_servers/arena/configs/vision_arena_v1.yaml → vision_arena_v1
    • benchmarks/vision_arena_v1/config.yaml → vision_arena_v1_benchmark_resources_server. This benchmark copies its Resources Server through _inherit_from while the parent stays loaded, so the resolver keeps the inherited reference; the child has to override it.

    Each binding is the Resources Server its agent already uses. The three beam benchmark children inherit the right binding and need no change.

  • 86a11b4fa binds renamed Resources Servers in benchmarks that compose. CI's genrm_eval suite failed test_mmrb, which loads genrm_mmrb1 and genrm_mmrb2 together. Both rename genrm_eval, and their environment servers relied on the resolver to follow the rename. With two renames in one run, the reference is ambiguous and the resolver raises AmbiguousResourcesRenameError. On main the environment server asked the agent, so this worked. Five environment servers now bind explicitly:

    • benchmarks/genrm_mmrb1/llm_as_a_judge.yaml → genrm_mmrb1_resources_server
    • benchmarks/genrm_mmrb2/llm_as_a_judge.yaml → genrm_mmrb2_resources_server
    • benchmarks/genrm_rewardbench2/config.yaml → genrm_rewardbench2_resources_server
    • benchmarks/hle_diamond/config.yaml → hle_diamond_equivalence_llm_judge_resources_server
    • benchmarks/indic/mbpp/config.yaml → indic_mbpp_resources_server, for consistency; it has no sibling that renames the same Resources Server today.

    genrm_rewardbench2 and hle_diamond failed the same way next to their sibling benchmarks, which CI does not load together. Each binding is the Resources Server its agent already uses.

  • c4e6830cd imports the binding test's parser from nemo_gym.config.config_types and nemo_gym.config.global_config instead of the moved paths (review). No other line this PR adds uses a path listed in nemo_gym/_moved_paths.py.

Validation on this head:

  • The focused tests listed under Validation below, plus test_benchmarks, test_cli and the Indic tests: 837 passed. Server suites run from their own directories, as CI does: single_agent_turn_legacy 23, simple_agent 89, single_agent_turn 127, all passed.
  • The generator's --check reports no refusal.
  • genrm_eval's suite in its own environment: 47 passed. The binding, global config, generator, benchmark and Indic tests: 666 passed.
  • Composition check. Some Resources Servers are renamed by several benchmarks. All 350 pairs of such benchmark configs were loaded together, with a model config, using the actual parser. Main and this head each load 332 pairs, with 0 regressions. The other 18 fail the same way on both sides.
  • indic/math_500 is the one benchmark that reuses math_with_judge_environment_server instead of defining its own. Loaded alone, it follows both renames, as test_indic_math_environment_follows_its_resources_rename checks. Next to another math_with_judge benchmark, the agent rename is ambiguous too, and composition falls back to a legacy_agent relay, the same as on main. Giving it its own environment server is left for a follow-up.
  • Binding audit with the actual parser, main (773c056cd) against this head, over all 328 configs with an environment server: 295 environment bindings, 0 changed and 0 unbound. The same 34 configs fail to parse on both sides.
  • Main has since moved to dda609f86. It merges cleanly, and its five new commits change no environment server config.

Validation

Python 3.13.14:

PYTHONPATH=. python -m pytest --import-mode=importlib \
  tests/unit_tests/test_global_config.py \
  tests/unit_tests/test_environment_resources_binding.py \
  tests/unit_tests/test_add_legacy_agent_environment_servers.py \
  environment_servers/single_agent_turn/tests/test_app.py \
  environment_servers/single_agent_turn_legacy/tests/test_app.py \
  -q -o addopts='' -o log_cli=false --disable-warnings
pre-commit run --all-files
python tests/unit_tests/test_fern_docs_links.py
cd fern && npm run check

491 tests passed; all-files hooks and 29 Fern link tests passed. Fern check reported zero errors and one warning: authenticated redirect checks were skipped because no Fern token was configured.

Additional actual-parser comparison against parent 50b0b521c6c048650cdde5acc0734639b92fd058: 1,066 configs checked, 916 pass on both revisions, 150 pre-existing failures unchanged, zero new failures and zero changed effective resources bindings across 306 resolved env bindings. This is a configuration compatibility audit, not full benchmark qualification.

The identical resources-only rename probe raises ServerRefNotFoundError on the parent and resolves the env to the new resources name here, with the agent's resources binding still null. Independent review checked the resolver, runtime binding use and generator refusal logic.

Preserved-name integration validation

Rebased onto the updated #4954. The generator retains both explicit Resources Server bindings and Gym's deletion-before-inheritance behavior, including _copy and string aliases. HLE/APEX harness overrides preserve their original Environment Server names.

On the integrated head 7fe55fde5, Python 3.13.14:

PYTHONPATH=. python -m pytest --import-mode=importlib \
  tests/unit_tests/test_global_config.py \
  tests/unit_tests/test_environment_resources_binding.py \
  tests/unit_tests/test_add_legacy_agent_environment_servers.py \
  environment_servers/single_agent_turn/tests/test_app.py \
  environment_servers/single_agent_turn_legacy/tests/test_app.py \
  responses_api_agents/simple_agent/tests/test_app.py \
  tests/unit_tests/test_rollout_collection.py \
  tests/unit_tests/test_checkpoint_agent.py \
  tests/unit_tests/test_harness_conformance_ci.py \
  -q -o addopts= -o log_cli=false --disable-warnings --tb=short
pre-commit run --all-files

1,184 tests passed, including the inherited composition/observation fixes and the _copy deletion-order regression cases. All-files hooks, DCO trailers, and whitespace checks pass. The generator dry-run has no refusal (exit 1: two already-unmigrated Vision Arena relays remain eligible; 22 relays stay for their existing blockers).

The six HLE/APEX Pi, OpenCode, and HLE search entrypoints were compared with the pre-#4952 baseline at all four updated stack layers: 30 actual-parser resolutions preserve the original Environment Server names and all effective server/sandbox settings. The integration checks above did not include inference; current-head real-model smoke is recorded below. Broader migration and model-qualification gaps remain.

This is focused integration validation. Full repository runtime CI remains outstanding; current-head representative model smoke is recorded below. The PR stays a draft.

Rollouts

Completed representative real-model HTTP smoke at current head 7fe55fde511c8c02dbbd4299cf805e91cdfc6649 using the actual config parser, config-selected resources, simple agent, legacy single-agent env and InferenceProvider against a live model endpoint. Sampling: temperature 0, seed 103, max output 8192 tokens.

Configuration Frozen tasks Result
benchmarks/indic/math_500/config.yaml First two Hindi tasks, stock generic Math prompt Both completed, reward/library reward 1; renamed resources received both seed/verify/close sequences
benchmarks/indic/ifeval/config.yaml Bengali tasks 1000 and 1001 from retained stock-prepared input Both completed, reward 1; Indic received both seed/verify/close sequences while the concurrently running original English instance received none

All four responses were nonempty and completed, with no failure fields or masking. Every traced HTTP call returned 200. Both agents ended with zero active sessions; application servers and the bounded smoke step terminated. Independent review checked raw rollouts, events, source hashes and cleanup receipts.

Source files were verified against the archived commit and loaded Gym module paths were checked. The pinned IndicIFEval checker matched all 30 reconstructed files with no extra Python files. The same four inputs also passed on original commit 2b26f8f; the repeat above followed the concurrent stack integration.

Input SHA256:

  • Math: fc9f51e6fa6d903216c00e7393933e84aae4164253a5a4bc33104a56e32c3984
  • IFEval: c37cc0abfb892463544010d98ef8e39f00a2c8fffabb4ea3a4d4e42427840e84

Two setup attempts failed before their affected inference because the reused runtime lacked dependencies; task-local installation resolved them. Frozen IFEval inputs have recorded hashes/task IDs but no recorded upstream dataset revision. Both Math answers were boxed, and the Bengali-prompt IFEval responses were English (the selected checks do not require Bengali output). This verifies routing/lifecycle for these tasks; it does not establish unboxed extraction behavior, language quality, full CLI startup, tool use, full benchmark scores or pre-migration score parity.

Full runtime CI and parent-stack qualification remain pending; keep this PR as a draft.

Compatibility and benchmark impact

Custom single-agent env configs must now supply an explicit resources_server; omitted or null no longer falls back to the agent. Child configs selecting a copied/different resources instance must override the env binding. The shipped backfill preserves the effective bindings observed in the config audit; benchmark scores have not been requalified.

Checklist

  • I have read the contributing guidelines.
  • The change is focused.
  • Tests added/updated and pass locally.
  • Documentation updated.
  • All-files pre-commit checks pass.
  • New source files include the required SPDX header.
  • Commits have DCO sign-off.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yaoyu-33 and others added 4 commits October 27, 2026 14:33
…t_turn names none

A legacy_agent relay names no resources server: the agent's /run uses its
own. A single_agent_turn_legacy server that replaces the relay had to pin
one, and a pinned name breaks a benchmark that reaches the server through
an _inherit_from rename of the agent. The rename also renames the
resources server, and Gym follows renames only for agent references, so
the pinned name dangles (for example indic/math_500 through
math_with_judge's server).

resources_server is now optional. Unset, the server binds the resources
server the bound agent names, as the relay did, so it follows the agent
rename. A server that sets none in front of an agent that names none
fails at startup instead of on the first episode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
About 307 configs reach simple_agent through a legacy_agent relay, so a
crash repeats the whole episode. Under single_agent_turn_legacy, it
repeats only the current step, and simple_agent resumes inside its turn
(GYMINT-549).

add_legacy_agent_environment_servers.py gains --switch-simple-agent. It
replaces each relay in front of a simple_agent with a
single_agent_turn_legacy server of the same name, so rows keep their
routing. The server names no resources server, so it binds the agent's,
as the relay did, and follows an agent rename. It grants direct HTTP
access to the resources server's tools, the routes the agent's /run
called.

It resolves each config with its own config_paths, as Gym does, since
instance names are unique only within a run. A relay stays, listed with
its reason, when its agent binds no resources server, declares a grant
simple_agent sessions reject (a required MCP access or a second direct
HTTP access), binds a resources server whose seed_session takes only the
legacy body (19 relays today, read from the server's source so a server
passes once it migrates), or its run cannot load an include. A relay that inherits
another keeps inheriting it, and the script refuses to switch only one
of the two. The default mode is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The pairing guard only compared `_inherit_from` children that carry their own
`legacy_agent` key, so a run that merged a switched relay with a same-name
`legacy_agent` override (the hle and apex_shortlist opencode/pi recipes) or a
bare `_inherit_from` child exited 0 with two environment server types.

- Resolve every run after the planned switches. A relay that a run merges with
  a legacy_agent server stays on the relay; when the relay it inherits switches,
  the script refuses (exit 2) and names the config. Top-level nulls now remove
  instances, as in Gym.
- Keep relays whose resources server reads the row's rollout coordinates
  (genrm_compare), whose implementation is not under resources_servers/, or
  that set keys the switched stanza would drop.
- Build every rewrite before writing any file; a stanza that cannot be found
  exits 2 instead of raising StopIteration.
- Dedupe scanned paths after resolving them.
- Give the hle and apex_shortlist opencode/pi recipes a legacy_agent server of
  their own, so their benchmarks can switch.
- Fix the docstring and docs examples, document the new stay reasons, note how
  to inherit the single_agent_turn templates, and test the legacy adapter with
  resources_server unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
When two files of a run declare the same relay (a base and a partial
override of it) and a later config merges it with a legacy_agent server,
split_servers found two switching entries for the instance itself and looked
for a different inherited relay that was not there, raising StopIteration.
Every declaration of the instance now stays on the relay; the script still
refuses (exit 2) only when a different relay the instance inherits switches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 force-pushed the codex/env-resources-bindings branch from 7fe55fd to 43f6d53 Compare October 27, 2026 18:59
@yaoyu-33
yaoyu-33 removed this pull request from stack #4955 October 27, 2026 19:02
@yaoyu-33
yaoyu-33 added this pull request to stack #5146 October 27, 2026 19:03
@yaoyu-33
yaoyu-33 marked this pull request as ready for review October 27, 2026 19:55
@yaoyu-33
yaoyu-33 requested review from a team as code owners October 27, 2026 19:56
yaoyu-33 and others added 9 commits October 27, 2026 16:04
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Runs scripts/add_legacy_agent_environment_servers.py --switch-simple-agent
over the repository (GYMINT-549). 287 relays in front of simple_agent in
279 config files become single_agent_turn_legacy servers of the same
name, so a crash repeats only the current step instead of the whole
episode, and simple_agent resumes inside its turn.

20 relays stay, as the script reports:
- 19 whose resources server seeds only the legacy body (workplace_assistant,
  openenv, newton_bench, indirect_prompt_injection,
  example_session_state_mgmt, and the SWE example configs); they move
  once their servers accept the typed seed.
- benchmarks/nemotron_3_ultra/benchmarks_no_external.yaml, whose include
  benchmarks/ruler/config_nemotron_3_256k.yaml does not exist on main.

Tests and docs that read these configs follow the new key. The
super35 container build now covers single_agent_turn_legacy too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Regenerated with the fixed --switch-simple-agent: genrm_compare's /verify needs
the row's _ng_rollout_index and cohort keys, which single_agent_turn_legacy does
not forward, so both genrm servers stay on legacy_agent. hle and apex_shortlist
still switch now that their opencode/pi recipes declare their own relay. The
YAML is byte-identical to a fresh generator run: 278 files, 285 relays switched,
22 kept.

The docs said every shipped simple_agent config uses single_agent_turn_legacy;
22 stay on the relay, so say "most" and name why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The switched configs add single_agent_turn_legacy to the core batch's resolved
server set, so the prebuilt dependency check now reports 16 environment paths.
CI on Linux showed "16 server environment paths checked" against the old
expectation of 15; macOS Bash 3.2 stops these shell tests before the count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
With model-call capture on, a simple_agent episode run through an
Environment Server (single_agent_turn, single_agent_turn_legacy) lost
its executed tool records: the session kept only the trajectory's
invocations and gaps, so the collected ng_trajectory had no tool_calls
and ng_perf.num_tool_calls was 0, while /run reported them.

The session now adds a ToolCallObservation for each executed tool call
to the agent observations it returns at close. Rollout collection
already merges these into ng_trajectory.tool_calls and rejoins each
call's output from the invocation conversation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 2ef4896)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…_legacy

A `--agent-type` swap onto an environment fronted by single_agent_turn_legacy kept
that server type even when the incoming agent does not implement agent sessions,
so every episode failed with NotImplementedError from /v1/agent_sessions.
Composition now rewrites such a server to a legacy_agent relay that calls the
agent's own /run, keeping its entrypoint, agent_server and host/port.

AGENT_SESSION_AGENT_TYPES in global_config is the single list of session-capable
agent types; the legacy-relay migration script reads it instead of its own copy,
and a unit test keeps it in step with the agent apps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit d156ef2)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ple_agent into simple_agent

Both agents make one model call and pass its tool calls to the verifier
unexecuted. simple_agent with execute_tools: false does the same as an
agent session, so its configs run under single_agent_turn_legacy and
resume inside the turn after a crash (GYMINT-217, GYMINT-147).

14 configs move: image_tools_pivot, swe_pivot, structured_outputs_v4,
seven single_step_tool_use_with_argument_comparison configs and four
terminal_multi_harness configs. Each agent keeps its instance name and
becomes simple_agent with execute_tools: false; its relay becomes a
single_agent_turn_legacy server, which simple_agent needs since it
refuses /run without tool execution; allowed_agents gains simple_agent.

The pivot-dataset skill's template and reference config writer emit
the same pairing. Both agents stay, marked deprecated, so configs
outside the repository keep working until a later release removes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ent configs

With execute_tools: false the agent never calls a tool, so the direct_http grant
on the 14 converted configs was unused, and the pivot skill's template already
omits it. It also made a silent mis-score possible: `--agent-type simple_agent`
is now allowed on these servers, composition does not carry execute_tools over,
and the swapped-in agent would execute tools through the grant against verifiers
that score one unexecuted call. Without the grant it fails loudly instead.

A repository test now asserts that no execute_tools: false simple_agent sits
behind a legacy_agent relay (its /run refuses) or gets a tool grant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@weijiac0619
weijiac0619 enabled stacked auto-merge November 3, 2026 06:05
@weijiac0619

Copy link
Copy Markdown
Contributor

Hi @ananthsub , the import issue is fixed and CI is green. It still needs a code-owner approval because it touches resources_servers/litmus_agent/configs/. Could you approve when you have a moment?

@weijiac0619
weijiac0619 added this pull request to the merge queue Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41683812971

yaoyu-33 added a commit that referenced this pull request Nov 3, 2026
Merge codex/env-resources-bindings at c4e6830 to resolve the GenRM MMRB configuration ambiguity that failed server-suite shard 0. Preserve the DeepAgents diff unchanged.

Validation: GenRM suite 47 passed; configuration, resources binding and migration generator tests 359 passed (the local port test passed outside the sandbox); pre-commit run --all-files and git diff --check passed. No new model rollout: this merge imports existing config bindings and a test import correction, with no agent or verifier code changes.
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41693344268

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41694085699

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41695039879

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41699905303

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41704097016

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41705943440

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41709778427

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41711148823

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Nov 3, 2026
@weijiac0619
weijiac0619 added this pull request to the merge queue Nov 3, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41743893874

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41750498064

Merged via the queue into main with commit b9ddce5 Nov 4, 2026
55 of 58 checks passed
@weijiac0619
weijiac0619 deleted the codex/env-resources-bindings branch November 4, 2026 01:47
gwarmstrong added a commit to gwarmstrong/Gym that referenced this pull request Nov 4, 2026
NVIDIA-NeMo#5108 made the Environment Server's resources_server required again and
dropped the fallback to the agent's. The startup check for tool servers
now compares against the configured resources_server; the agent fallback
for tool_servers stays.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
yaoyu-33 added a commit to yaoyu-33/Gym that referenced this pull request Nov 5, 2026
single_agent_turn servers now require resources_server (NVIDIA-NeMo#5108). Bind
critpt_resources_server on the example server and the renamed
critpt_benchmark_resources_server on the benchmark server, the servers
their agents already use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yfw added a commit that referenced this pull request Nov 6, 2026
…erver block

Gym #5108 made resources_server a required field of single_agent_turn_legacy: the Environment Server no
longer takes it from its agent. configs/agentic_mme.yaml (the environment's own agent) and
benchmarks/agentic_mme/server_tools.yaml (simple_agent) left it out, so gym env validate failed for both with
'resources_server: Field required'. The base block now names agentic_mme, and every pairing in
benchmarks/agentic_mme/ inherits it. opencode.yaml no longer repeats it.

gym env validate passes for all 12 Agentic-MME configurations (each benchmark config, the no-internet overlay,
the search overlay and the six retrieval overlays).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
rimashahbazyan added a commit that referenced this pull request Nov 9, 2026
…s tests

Two core unit tests that arrived or changed with the origin/main merge
(69c748d) failed on this branch's files:

- test_public_api_facades flagged tests/test_app.py importing
  maybe_auto_expose from nemo_gym.server_runtime.mcp_auto_exposure; it is
  exported from nemo_gym.server_runtime.
- test_repository_step_environments_declare_their_resources (#5108)
  flagged hle_web_tools_environment_server, whose single_agent_turn_legacy
  block named no resources_server. It now names hle_web_tools, as the
  tavily_search configs do.

Before: the two tests, 2 failed. After: tests/unit_tests/
test_public_api_facades.py and test_add_legacy_agent_environment_servers.py
917 passed; resources_servers/hle_web_tools/tests 95 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fernandol-nvidia pushed a commit to fernandol-nvidia/Gym that referenced this pull request Nov 17, 2026
…eset (NVIDIA-NeMo#5176)

* feat(hle_web_tools): upstream HLE web_search and web_fetch behind the blocklist over MCP

Add resources_servers/hle_web_tools, a tool server that implements section 3 of
centerforaisafety/hle docs/evaluation-with-tools.md (commit 22ed307): exactly two
MCP tools, web_search (at most 20 results) and web_fetch (Markdown capped at
60,000 characters with a truncation indicator, no media payloads), with Tavily as
the search and extraction backend and the keys held server-side. The pinned
upstream docs/blocklist.json is shipped byte-exact and digest-checked at start;
all seven rule families are enforced with the protocol's semantics, plus query
refusal for blocked site: domains and verbatim question spans (8 consecutive
words, configurable; the question arrives through /seed_session), fetch-URL
checks after percent-decoding archive and proxy wrappers, title-and-Markdown
re-checks of fetched and cached pages, and a log line per blocked call naming
the family and rule.

Add benchmarks/hle_diamond/opencode_hle_tools.yaml, the OpenCode preset with this
server as its only tool server, document it and the residual Tavily-backend
difference in the Diamond README, and extend the Diamond preset tests to it.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(hle_web_tools): refuse web_search from a session with no seeded question

The plain POST /web_search route stays served next to /mcp under Gym's MCP
exposure, and a call without the rollout's cookie or token lands in a fresh
session that carries no question, so the verbatim-span rule was silently
skipped (probe: the same 8-word query refused with the rollout cookie, served
with no cookie and no token). Fail closed instead: web_search refuses any
session that was not seeded with a question under the new
require_seeded_question flag (default on), logged as family=unseeded_session;
a seed without a question says its searches will be refused. The other
families already applied on the direct route, and web_fetch has no
session-dependent rule.

Add the model-free test (seed, then a cookie-less POST /web_search is refused
and the backend is never called; the flag off restores the old behaviour),
and reword the module docstring and both READMEs to describe the enforced
behaviour rather than claiming the agent reaches the server only over MCP.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): refuse re-seeding a seeded session with another question

A session now keeps the question it was seeded with: a retried seed with the
same words is served again, a seed that would change or drop the question
answers HTTP 409 (ResourcesSessionConflictError), so a caller holding a
session cannot swap the verbatim-span check out from under it. QuestionSpans
keeps a SHA-256 of the normalized words, never the text, to tell the two
apart.

This does not close the fresh-session path: /seed_session is unauthenticated
and network_access model_and_tools admits the whole tool-server host, so a
sandbox can seed a new session with a question of its choice and search from
it without the span rule. The server README and the opencode_reference.yaml
header now state that residual and what closing it needs (an authenticated
seed channel the sandbox never sees).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
(cherry picked from commit 27187af4bc522d498ca64039b14570f41434aae2)

* fix(hle_web_tools): import nemo_gym names from their new paths

main moved the modules this server imported (nemo_gym.base_resources_server,
nemo_gym.server_utils names, nemo_gym.mcp_auto_exposure) and serves the old
paths only through deprecation shims removed in nemo_gym 0.10.0. Rewritten
with scripts/migrate_imports.py to the public API (nemo_gym.client,
nemo_gym.server_runtime) and, for the MCP test, the defining module, as
the other servers on main do; the README names the new request path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* docs(hle_diamond): say hle_web_tools exists as its own preset

opencode_reference.yaml and the README's Reference preset section, both
from main (#5177), said the reference preset would switch to the
hle_web_tools server once it existed. This branch adds that server, so
the sentence would turn into an unfulfilled promise on merge. The
reference preset keeps the Tavily tools here; the wording now names
opencode_hle_tools as the preset that runs hle_web_tools (on the science
image) and leaves running it on the reference image and egress policy to
a follow-up change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): end the Tavily call when every key is rejected

A key pool whose keys Tavily all rejects (401, 432, 433) spun forever in
the key-selection loop without an await, freezing the worker's event
loop: one configured key, a key listed twice, or two distinct rejected
keys were enough. A rejected key also spent the transient retry budget,
so [bad, bad, bad, good] with three attempts never reached the good key.

The pool is now deduplicated when the backend is built, rejected keys
move on without spending an attempt, the next key comes from a bounded
helper over the keys still accepted, and the call raises a RuntimeError
naming the status once every key is rejected. Transient failures keep
their backoff and budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): turn malformed Tavily bodies into tool errors

web_search read Tavily's payload unguarded: a body that is not an
object, `results: null`, a non-object item or a non-string title or
content raised AttributeError or TypeError and reached the model as an
HTTP 500. web_fetch had the same `results: null` gap and rendered a
non-string raw_content as page text.

Both tools now read the results list through one guarded helper.
web_search answers a malformed body with a "Search failed" tool error,
skips non-object items and treats non-string title and content as
empty; web_fetch reports any malformed body, item or raw_content as
"Fetch failed", and guards failed_results the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): add the example data CI validates

CI's changed-server step runs ng_test with should_validate_data=true,
which requires data/example.jsonl (5 rows), data/example_metrics.json and
data/example_rollouts.jsonl, plus an agent with an example dataset in the
server config; hle_web_tools had none of them.

Following resources_servers/tavily_search: the config gains a
simple_agent with an example dataset behind a single_agent_turn_legacy
environment server over direct HTTP, example.jsonl holds 5 synthetic
general-knowledge rows (no HLE text, each marked by a placeholder field)
whose tools are web_search and web_fetch with the server's own schemas,
example_metrics.json comes from gym dataset collate in
example_validation mode, and example_rollouts.jsonl holds placeholder
rows since no live model or Tavily call was made. The server stays
verified: false. The README says so, and the update-readme-table hook
adds the server to the environment list now that its config has an
agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* docs(hle_web_tools): say the server must run single-worker

Seeded questions and the page cache are per process, so with
num_workers > 1 a rollout's seed and its searches could land in
different workers; the module also defines no worker app, so such a
launch fails to start. Rather than add the multi-worker entrypoint
footer, the README states the single-worker requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): refuse URL spellings the policy could misread

The URL check certified URLs it had not fully checked and matched the
page-level families on the raw string only. Now url_block:

- refuses as malformed_url any URL longer than 4,096 characters, with a
  backslash, whitespace or control character, or with any @ in the
  authority (an empty https://@host included);
- fails closed when candidate expansion stops with candidates unexplored,
  at the 64-candidate cap or after the original plus four rounds, so 63
  decoy query values or a fourth decoded form no longer hide a target;
- runs url_prefixes and serp_url_patterns on a canonical form as well
  (IDNA lower-cased host, trailing dot, userinfo and default port
  dropped, dot segments resolved), so :443, a trailing dot or /x/../
  no longer evade a prefix;
- treats every *.archive.org host as a wrapper and reads https:/,
  https:/// and https: remainders as https://;
- reads wrapped and decoded URLs the way WHATWG fetchers do (backslash
  as /, tab and newlines deleted);
- takes each embedded http(s):/ up to the next one instead of through a
  lookahead to the end of the string, so extraction is linear.

site: operands are read as host characters only, so (site:a OR
site:b), site:a|site:b and site:b, yield bare hosts, and an operand
whose IDNA encoding fails is refused instead of skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): check the whole fetched page before it is cut

Fetched content was sanitized and cut at max_cached_page_chars before
the content families ran, so a blocked span past 500,000 characters was
served and cached, and a cache cap below max_fetch_chars served a cut
page as truncated=False. Now:

- a fresh page is checked against url_patterns and
  patterns_case_sensitive on the provider's raw Markdown, before data
  URIs are removed or anything is cut, and on the sanitized Markdown;
  a refused page is never cached;
- the scan is bounded: a page longer than max_page_chars (new,
  default 1,000,000) is refused unread as oversized_page, and the regex
  work runs in a worker thread instead of on the event loop;
- max_cached_page_chars no longer cuts the page: a longer page is
  served whole but not cached, so cache-hit rechecks see the complete
  sanitized Markdown;
- data URIs with media-type parameters or a percent-encoded payload are
  removed too, the payload stops at the Markdown destination's end so
  following prose is kept, and every repetition is bounded so the
  regex stays linear (the unbounded suggested form is quadratic);
- the heading fallback for the title is linear on long blank runs
  (100,000 characters took 42 s) and drops a closing # sequence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): bind typed sessions to their episode and fence closes

A typed seed (an Environment Server's resources_session_id, episode_id,
task_id and task_data) kept only the question, so the same ID could be
seeded or closed for another episode or task, and a seed replayed after a
close recreated a searchable session. Typed seeds are now validated as
ResourcesSeedSessionRequest, their episode and task are recorded next to
the question, and seed and close go through ResourcesSessionFence, as in
example_single_tool_call: a conflicting seed or close answers 409, and a
seed after its session closed is refused for the retry window. The
identity map shares the 10,000-session cap with the questions and is
evicted with them. Legacy Agent seeds are unchanged.

The test runs check_resources_session_contract with keeps_state=True.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): treat a question with no words as no question

A seed whose question has no word characters ("?!", or an empty user
message) was stored as seeded with span_words=0, which disabled the
verbatim-span rule while require_seeded_question still saw a seeded
session. extract_question now returns None for such text (the question
field falls back to the last user message first), so the session stays
unseeded and its searches fail closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): log a warning when the session cap forgets a question

Past MAX_SESSION_QUESTIONS (10,000) seeded sessions the least recently
seeded one was dropped silently, and its later searches were refused as
unseeded_session with nothing pointing at the cause. The eviction now logs
a WARNING naming the session and the cap. The test lowers the cap to one
and also checks that the question and the episode binding leave together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* docs(hle_web_tools): record the MIT license of the vendored blocklist

blocklist.json is an unmodified copy of docs/blocklist.json from
centerforaisafety/hle at 22ed3074b1e7b134bcbc09028d0ba320839b0655, which
is MIT (Copyright (c) 2025 centerforaisafety). The JSON stays byte-exact
for its digest check, so the upstream LICENSE at that revision is
reproduced unchanged in the sibling LICENSE-HLE, the file gets a row and
a note under Vendored Components in ATTRIBUTIONS.md, and the server
README gains a Licensing section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* docs(hle_diamond): name the sandbox-seeded session residual

Both READMEs called the Tavily backend the one residual difference from
upstream's wrapper, and the Diamond section said a session not seeded
with the question is refused "so the span rule cannot be skipped". Both
overstate the policy: /seed_session is unauthenticated, so a sandbox
process can seed a fresh session with a question of its choice and
search from it; the 409 on re-seeding covers existing sessions only, and
the session and MCP-token signing key derives from public config.

The server README now names that residual in its own paragraph, the
Diamond section lists it next to the backend as the second residual, the
fail-closed sentences are scoped to unseeded sessions, and the server
README's dangling pointer to the Diamond "Reference preset" section now
points at "Upstream web tools", where the residual is listed. Closing it
needs an authenticated seed channel; both READMEs carry the marker
TRACKING_ISSUE_TBD for the tracking issue, to be replaced by its number
before push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>

* fix(hle_web_tools): declare the task-data schema CI validates

The core unit tests require every resources server to ship a task_data.py,
and the committed example rows to validate against it. hle_web_tools had
none, so TestSchemaPresence and TestRepoDataMatchesSchemas failed.

Declare both keys the example rows carry: the optional question that
/seed_session reads, and the placeholder note that marks the rows as
synthetic. Keeping the note in the data makes each row say what it is;
the schema records that the server never reads it.

tests/unit_tests/test_task_data.py: 2 failed, 421 passed before;
425 passed after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): map hosts the way UTS #46 fetchers do

normalize_hostname used Python's IDNA 2003 codec, which leaves code
points unassigned in Unicode 3.2 unmapped. Browsers (WHATWG URL) and
yarl, which aiohttp uses, map a host with UTS #46 first, so
https://sc\U0001f130le.com/leaderboard was allowed by the policy but
fetched as scale.com. The same held for lastexam.ai and arxiv.org
spelled with such letters.

Map a non-ASCII host with idna.uts46_remap (non-transitional, no STD3
rules) before the existing encode, and declare idna, already installed
through aiohttp and yarl, as a direct requirement. IDN hosts that are
ordinary (例え.jp) keep their punycode form.

Test: one row in the URL refusal table; it fails before this change
(allowed, extract awaited) and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): refuse a wrapped URL Python cannot parse

url_shape_problem refuses an unparseable URL only at the outer level.
For a wrapped or decoded candidate, _single_url_block returned None when
urlsplit or IDNA raised, which skipped every family for it. Python
rejects https://[::1]@scale.com/leaderboard ("'scale.com' does not
appear to be an IPv4 or IPv6 address"), while browsers, httpx and
urllib3 read it as host scale.com, so a redirector carrying it in
?u= was served.

Refuse such a candidate as malformed_url instead. The realistic URL list
from review (48 URLs) shows no new refusal.

Test: one row in the URL refusal table; fails before (served), passes
after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): read a wrapper's remainder as a URL however it is spelled

A scheme-less remainder became a candidate only when it matched an
ASCII-only host regex, which rejects a trailing dot after the TLD,
userinfo and full-width or ideographic-dot hosts. So
web.archive.org/web/2024/scale.com./leaderboard and
web.archive.org/web/2024/x@scale.com/leaderboard were served, as was
?u=scale.com./leaderboard on a redirector.

A wrapper fetches its remainder (and Google's cache its cache: target)
whatever the spelling, so read each as https://<remainder> and let
urlsplit and normalize_hostname decide the host. Other query values
still need a scheme or a leading hostname, which may now end with a dot.
The README also records the malformed_url refusal of an unparseable
wrapped URL from the previous commit.

Test: three rows in the URL refusal table; all three fail before
(served) and pass after. The realistic URL list from review shows no
new refusal.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): match url_prefixes with a query in any parameter order

The eight openreview.net prefixes carry ?id=<paper>. As plain string
prefixes they match one exact parameter order, so
openreview.net/pdf?&id=46UGfq8kMI and openreview.net/pdf?x=1&id=...
were served although a server reads both as pdf?id=46UGfq8kMI, and the
id appears in no url_patterns rule.

Keep the string-prefix check and add a second reading for a prefix with
a query: its path must prefix the URL's path, and each of its key=value
pairs must be among the URL's parsed query pairs (a value as a prefix),
in any order. This only adds matches to the upstream semantics.

Test: two rows in the URL refusal table; both fail before (served) and
pass after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): collapse slash runs in the canonical URL form

remove_dot_segments kept // runs, so arxiv.org//abs/2605.02442,
arxiv.org/abs//2605.02442, arxiv.org/%2Fabs/... and the Wayback form
of the first were served, and 2605.02442 has no url_patterns rule to
catch them. Servers with nginx's default merge_slashes serve such paths
as the single-slash page.

Collapse runs of / in the canonical form. It is used for prefix and SERP
matching only; the fetched URL is unchanged.

Not changed: export.arxiv.org and /format/, /src/ and /e-print/ paths of
2605.02442 are gaps in the pinned upstream list, not in its matching.

Test: one row in the URL refusal table; fails before, passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): fold Unicode presentation variants before query checks

question_words ran \w+ over str.casefold() with no normalization, so a
query that repeats 8 words of the question in full-width or mathematical
letters, with a zero-width space or soft hyphen inside a word, with a
combining accent, or joined by underscores was served. The query
patterns had the same gap: humanity<U+00AD>s last exam was not matched.
Search engines fold all of these back to the plain words.

Words are now runs of letters and digits ([^\W_]+) after NFKC with
format characters (Cf) removed and combining marks stripped; the
question and the query go through the same function. url_patterns,
patterns_case_sensitive and site: operands are checked on the query as
written and as folded. Look-alike letters from another script (Cyrillic
o) are still not folded; the README says so.

Test: test_presentation_variants_of_a_blocked_query_are_refused, five
rows, one per mechanism (NFKC, Cf, combining mark, underscore, a pattern
with a soft hyphen); all five fail before and pass after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(hle_web_tools): apply ruff-format to the full-width span row

The previous commit was made before ruff-format wrapped this row; no
behavior change.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): bound title work and cap shown titles

Result titles come from the provider with no length bound, and
_TITLE_SEPARATOR_RE (\s+ before a separator) backtracked over every
blank of a run, quadratic in its length: a 20,000-blank title took
3.9 s and a 50,000-blank one 26.6 s of web_search on the event loop.
Each separator also added a variant up to the full title length, so a
title of 'a | ' * 2,000 cost 1.1 s.

Start the separator match only at the first blank of a run, and cut a
title longer than 1,000 characters (with an ellipsis) before it is shown
or checked against answer_title_patterns. url_patterns and
patterns_case_sensitive still see a result's whole title, which is
linear.

The same cap applies to the fetched page's Title: line. That title,
taken from the heading when the provider sends none, was not counted
against max_fetch_chars, so one fetch could return about 960,000
characters.

Tests: test_long_result_titles_are_checked_in_linear_time (a blank run
and a separator chain, both 20,000 characters) fails before (about 5 s)
and passes after; the 60,000-character cap test gains a heading-derived
title case, which fails before (the 100,000-character title is shown
whole) and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): answer a non-JSON 200 body as a tool error

5142f52df turned a decoded body of the wrong shape into a tool error,
but a 200 whose body does not decode still escaped: a gateway's HTML
page raised aiohttp's ContentTypeError and a cut-off JSON body raised
JSONDecodeError, and both reached the caller as a server error.

Return None from the transport for such a body, so the existing
"Search failed: ... malformed response" and "Fetch failed for <url>"
paths answer it. It is not retried: a 200 is not a transient status.

Test: two rows (HTML body, cut-off JSON) in the parametrized
malformed-payload test, through the real transport with a stubbed
request; both fail before (the exception propagates) and pass after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(hle_web_tools): make the retry-budget test fail when a rejection spends it

test_rejected_keys_do_not_spend_the_retry_budget only bounded the total
number of requests, which the pre-fix code also met. Two real
regressions survived it: a rejected key spending an attempt, and key
selection that no longer wraps round to the current key.

Replace it with the mixed case: keys key-one, key-two and two attempts;
Tavily answers 401, 429, 200. The call must succeed after trying
key-one, key-two, key-two with one backoff sleep.

Fails on app.py from 498e34bc3 (before eb466ecb9: "Tavily HTTP 429 on
/search after 2 attempts"), on the spend mutant (same error) and on the
no-wrap mutant (TypeError); passes on HEAD.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): fence a typed session the cap forgets

Untyped Agent seeds share the 10,000-session cap with typed sessions.
When eviction dropped a live typed session's episode binding, a later
seed of that ID for another episode returned 200 and bound it, where
the resources-session contract requires 409; the session's question
was already gone.

When the cap forgets a typed session, mark its ID closed by its bound
episode in the session fence. A later seed of the ID then gets 409, and
a close from its own episode is still accepted. The README states that
the cap must exceed the peak number of open rollouts.

Test: test_sessions_past_the_cap_are_forgotten_with_a_warning now seeds
the forgotten ID for another episode and expects 409; it fails before
(DID NOT RAISE) and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(hle_web_tools): cover the live-session close conflict and the wordless fallback

Three regressions survived the session tests:
- a close naming another episode of a live typed session was not
  checked (the contract checks it only for a closed session), so
  disabling that 409 deleted the session's question unnoticed;
- holds_state looked only at the question table, so a close that kept
  the episode binding passed;
- the wordless-question row used the question field, which either of the
  two guards refuses alone, so dropping the extract_question gate for the
  last-user-message fallback passed.

The contract test now counts the binding as held state and closes a live
session with another episode (409, question and binding kept). The
wordless case also seeds a last user message of "?!" with no question
field.

Fails on: the close-conflict mutant (200 != 409), the close-keeps-binding
mutant ("close left state"), the gate mutant and app.py from 16cd8c0f5
(wordless message seeded); passes on HEAD.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): refuse a site: operand that is not hostname-shaped

The README said a site: operand that is not a valid hostname is refused,
but only an operand whose IDNA encoding failed was: site:huggingface.co!
and site:exa_mple!.org reached Tavily. Results were still filtered by
domain, so nothing leaked, but the refusal and its audit log line that
the protocol asks for were missing.

Drop trailing punctuation after normalize_hostname (site:huggingface.co!
is huggingface.co, refused as domains), and refuse as malformed_url an
operand that is then not labels of letters, digits, - and _, optionally
after *.

Test: two rows in the site-operand table; both fail before (served) and
pass after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): strip data URIs that omit the media type

RFC 2397 lets a data URI omit its media type (data:;base64,...), and
_DATA_URI_RE required one, so such a payload was served whole.

Make the media type optional. The README now states the bounds that
already applied: at most 16 media-type parameters, and the payload ends
at the first whitespace, quote, ) or >, so a base64 payload wrapped over
several lines keeps its later lines.

Test: a data:;base64, image in the existing data-URI test page; fails
before (payload kept) and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): refuse a multi-worker launch with the standard hook

c4262af29 documented that the server must run single-worker, but the
server did not use SimpleServer.multiple_workers_unsupported_reason, so
num_workers=2 passed refuse_unsupported_num_workers and dry runs, and
failed only later as uvicorn's 'Attribute "app" not found'.

Return the reason (seeded questions, session bindings and the page cache
live in one process), so run_webserver refuses num_workers > 1 before
uvicorn starts with "hle_tools needs num_workers=1: ...". The README
sentence now says so.

Test: test_a_launch_with_several_workers_is_refused fails before (DID
NOT RAISE) and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): run page checks in worker processes

465d43de4 moved the page work to asyncio.to_thread, but Python's re keeps
the GIL for a whole search() or sub() call, so the event loop still
stopped for the length of one scan. The same commit widened the scan to
the provider's raw page of up to 1,000,000 characters: with a stubbed
page of 999,000 characters of 抱抱脸 a 1 ms heartbeat saw a 4.0 s gap
(答案: 3.5 s), from the pinned upstream hugging-face/answer-key
url_patterns pair. A cache hit rescans up to 500,000 characters the same
way.

Run build_page and the content families, fresh and on a cache hit, in a
module-level ProcessPoolExecutor of two spawn workers; the final-URL
check stays on the loop and runs first. The Blocklist is passed with each
call (its patterns pickle as source and recompile from re's cache), so a
policy changed after start still applies. A pool whose worker died is
replaced on the next call. spawn needs the entrypoint to keep its
__main__ guard, which app.py has; with app.py as __main__ the workers
resolve the functions through __mp_main__ (checked with a copy of app.py
run as a script).

After: the same probe shows a 0.02 s maximum gap for both pages. The
README claim about a worker thread is corrected.

Test: test_page_checks_do_not_stall_the_event_loop (300,000 characters of
抱抱脸, heartbeat gap < 0.5 s) fails before (1.25 s gap) and passes after.
The 100,000-character heading timing case now warms the pool first, so it
does not time the start of the worker processes.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_diamond): list the oversized-page refusal as a third residual

The Diamond README said two residual differences from upstream's wrapper
remain (the Tavily backend and sandbox-seeded sessions). A third was not
disclosed: the protocol caps web_fetch output at 60,000 characters with a
truncation indicator, so upstream serves the start of any page, while this
server checks the whole page and refuses one longer than max_page_chars
(1,000,000 characters) as oversized_page. A clean 1,040,000-character page
is refused here and would be served cut upstream; long PDFs reach that
size.

List it as the third residual a with-tools number should state, and point
to it from the server README. Also qualify the server README's redirect
claim: the final-URL check relies on Tavily reporting the post-redirect
URL in results[].url, which is verified only against a stub, not the live
API.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): read format characters both as removed and as a space

4abb964f4 removed format characters (Cf) before splitting a query into
words. That catches sp<ZWSP>in, but it glued words joined by a word
joiner or a zero-width space: how<U+2060>many<U+2060>odd... became one
word and was served, although the span rule refused it before that
commit. Whether a search engine drops such a character or splits words
at it is not known.

Read the text both ways: the question's n-grams come from both readings,
a query is checked in both, and the query patterns and site: operands
see the as-written, removed and space readings.

Test: a word-joiner-separated row in
test_presentation_variants_of_a_blocked_query_are_refused; it passes on
2500e8539, fails on 4abb964f4..77d3f0f5a, and passes after.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): import test helpers from their moved nemo_gym paths

Main moved nemo_gym.server_utils to nemo_gym.server_runtime.base_server and
nemo_gym.testing to nemo_gym.server_runtime.testing (both since 0.8.0, old
paths removed in 0.10.0). The tests imported refuse_unsupported_num_workers
and check_resources_session_contract through the old paths, so importing
them raised the nemo_gym DeprecationWarning. Rewritten with
scripts/migrate_imports.py, imports merged by ruff; the README's test
description names the new module path.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): pass the core public-API and environment-resources tests

Two core unit tests that arrived or changed with the origin/main merge
(69c748d082) failed on this branch's files:

- test_public_api_facades flagged tests/test_app.py importing
  maybe_auto_expose from nemo_gym.server_runtime.mcp_auto_exposure; it is
  exported from nemo_gym.server_runtime.
- test_repository_step_environments_declare_their_resources (#5108)
  flagged hle_web_tools_environment_server, whose single_agent_turn_legacy
  block named no resources_server. It now names hle_web_tools, as the
  tavily_search configs do.

Before: the two tests, 2 failed. After: tests/unit_tests/
test_public_api_facades.py and test_add_legacy_agent_environment_servers.py
917 passed; resources_servers/hle_web_tools/tests 95 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(hle_web_tools): pin key rotation after a 429 and the release of every response

Review thread 4398612291 asked that a 429 followed by a 200 still moves to
the next key with backoff, and that every response is released. Both hold,
but no test pinned either: a mutant that retries the same key after a 429,
and one that releases only 200 responses, each passed the whole Tavily
transport selection (22 passed).

test_rejected_key_fails_over_and_provider_bodies_stay_private now runs for
401 (no backoff) and 429 (one backoff of 1 s), and asserts that both
responses are released.

Before (HEAD test, -k 'key or tavily or retry or twenty or malformed'):
norotate 22 passed; release_only_200 22 passed.
After: norotate fails [429-backoff1]; release_only_200 fails
[401-backoff0] and [429-backoff1]; unmutated code passes.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(hle_web_tools): refuse a typed re-seed of a live session for another task

Review thread 4398612319 listed, as an acceptance check, that a typed seed
of the same resources_session_id with a changed task gets 409. The server
already refuses it, but neither check_resources_session_contract nor this
file's test changed only the task, so a mutant that compares only the
episode (bound[0] != identity[0]) passed the whole file.

test_typed_sessions_follow_the_resources_session_contract now seeds the
live session again with task_id 2 and asserts 409, with the session's
question digest and binding unchanged.

Before (HEAD test file): mutant episode_only, 95 passed.
After: mutant episode_only fails this test; unmutated code, 96 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): read a Wayback path with no timestamp as naming its target

Follow-up to review thread 4405604682 (Wayback spellings). The Wayback
unwrapping took the first segment after /web/ as the timestamp, so
web.archive.org/web/scale.com/leaderboard was read as https://leaderboard
and web.archive.org/web/scale.com as https://web/scale.com, and both were
fetched. archive.org redirects such a path to a snapshot of the target
(web/scale.com/leaderboard answers 302 to /web/<timestamp>/https://scale.com/leaderboard).

The first segment is now read both ways: as a timestamp, and as the start
of the URL. A timestamped path keeps its existing reading, so every
spelling refused before is still refused; the extra reading only adds a
candidate.

Before (HEAD app.py, new URL-table rows): web/scale.com/leaderboard and
web/scale.com fail (allowed). After: both refused as domains; the
timestamped, id_, 2024* and Wikipedia rows unchanged; 98 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): read typographic quotes around a site: operand like ASCII quotes

Follow-up to review thread 4405604730 (site: operand parsing). Typographic
quotes and guillemets were neither openers nor excluded from the operand,
and NFKC does not fold them, so site:“scale.com” became the punycode host
xn--scale-hw3b.xn--com-9o0a, which is hostname-shaped and matches no
domain: the query was served with no refusal and no audit line. A
trailing ” survived as well (site:scale.com” became scale.xn--com-9o0a),
since trailing punctuation is stripped after IDNA encoding.

The operand regex now treats ‘ ’ ‚ ‛ “ ” „ ‟ « » ‹ › like " and ': they may
open the operand and end it. The opener run is now repeatable, so stacked
openers (site:((scale.com)), site:«“scale.com”») no longer stop the match.

Before (HEAD app.py, new rows): site:“scale.com”, site:scale.com” and
site:((scale.com)) are served. After: each is refused as domains;
site:example.org, site:“example.org” and site:en.wikipedia.org/wiki/Foo_(bar)
are still served; 101 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): fold every default-ignorable character and read ZWSP as a word break

Follow-up to review thread 4398612299 (span-rule folding, e075400a3c). Two
spellings still got an 8-word question span or a blocked query past both
readings of format characters:

- Mixing a zero-width space between words with a soft hyphen inside one
  ("How<ZWSP>many ... no<SHY>nvanishing", "H<SHY>LE<ZWSP>leaderboard"):
  each reading treats every format character the same way, so each one
  breaks. Unicode word segmentation (UAX #29) splits at U+200B and ignores
  U+00AD, and reads both as [how, many, ..., nonvanishing].
- Hangul fillers (U+3164, U+115F, U+FFA0; default-ignorable, category Lo)
  inside a word ("no<HF>nvanishing", "Hum<HF>anity's Last Exam",
  "H<HF>LE leaderboard"): only category Cf was folded, and NFKC maps the
  filler to U+1160, still a letter.

fold_text now treats every Default_Ignorable_Code_Point (ranges from
DerivedCoreProperties, since unicodedata has no such property) like a
format character, and the checks read text three ways: all removed, all
as a space, and as word segmentation does (U+200B a space, the rest
removed). The readings apply to the question n-grams, the query's words,
and the query forms the text families and site: operands see.

Before (HEAD app.py, new rows in the presentation-variant table): the four
rows mixed-format-characters, mixed-format-pattern, hangul-filler and
hangul-filler-pattern are served. After: each is refused with its family;
105 passed. Seeding a 2.24 MB non-ASCII question takes 1.6 s instead of
1.1 s (ASCII: 0.49 s, unchanged); unbounded seed size is already part of
the unauthenticated-seed residual.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): also match the canonical form with slash runs merged before dots

Follow-up to review thread 4405604669 (equivalent spellings, 25f0c7f83b).
The canonical form resolved dot segments and then collapsed slash runs, so
arxiv.org/abs/x//%2e%2e/2605.02442 became arxiv.org/abs/x/2605.02442 and
was fetched. httpx and urllib3 send that path unchanged, and nginx with its
default merge_slashes collapses the run first and then resolves the dots:
a throwaway nginx 1.26.3 serves /abs/2605.02442 for it (whether arxiv's
front end does the same is not verified).

The URL-shaped families now see both orders: dots then slashes (as before)
and slashes then dots (nginx).

Before (HEAD app.py, new URL-table row): arxiv.org/abs/x//%2e%2e/2605.02442
is allowed. After: refused as url_prefixes; arxiv.org/abs/x/2605.02442 and
example.org/a//../b stay allowed; 106 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): find a title's site-name separator after a long blank run

Follow-up to review thread 4405604713 (e10df7a73e capped shown titles). The
answer-title check ran on the title already cut at 1,000 characters, so a
result titled "Quiz answers" + 5,000 blanks + "| Site" lost its separator
to the cut and was kept, although the model is shown "Quiz answers ...".
Before the cap it was dropped.

title_variants now takes the whole title: the title as shown, plus the
title up to each separator that starts within the first 1,000 characters,
however long the separator's blank run. Each variant stays at most 1,000
characters, so the check stays linear (1,000,000-character titles: under
0.04 s). An answer phrase that lies wholly past the cut is neither shown
nor checked; the README says so.

Before (HEAD app.py, new row padded-separator): kept. After: dropped;
107 passed.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): state when the session cap forgets a running rollout

Review threads 4398612339 (LRU eviction) and 5680046430 (HOLD summary).
The README said the cap "must exceed the peak number of rollouts open at
once". That holds only for a harness that closes its sessions: OpenCode
seeds with its run body and never closes, so held sessions add up, and a
running rollout's question is forgotten once more than 10,000 later seeds
reach the server, however few rollouts are open at once (probe with the cap
at 2: A forgotten after B and C seed, with at most 2 open).

The README also said a seeded session keeps its question "for its whole
life", but only typed seeds check the fence: a forgotten Agent (untyped)
session, or a closed typed ID, takes a later untyped seed through its
cookie with another question (HTTP 200). The README and the comment in
_install_question now say so, and that this, like an unauthenticated seed
flood past the cap that makes other rollouts' searches fail closed, is the
same unauthenticated-seed residual as a sandbox-seeded fresh session.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): state every bound of the data-URI regex

Review thread 4398612357 (data URIs). The README named the 16-parameter
bound only, but the regex also caps the type and subtype at 127 characters,
each parameter name at 127 and each value at 256: a data URI with one
parameter value of 257 characters is not matched and is served whole. The
bounds are what keep the scan linear, so the code is unchanged; the README
and the regex comment now state them and that a URI past them is left in
place.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): state the page-check pool's cost and what still runs on the loop

The README described the page-check pool (b132980c82) without its cost or
its scope. Measured at this head, with the Tavily transport stubbed:

- The pool is created on the first page check and starts workers on
  demand, up to two. Each worker imports the server module afresh and sits
  at about 240 MB RSS, the size of the server process, and the first fetch
  takes about 1.8 s against 0.002 s afterwards.
- web_search's per-result URL, title and snippet checks still run on the
  event loop: one result whose title (or snippet) is 1,000,000 characters
  of 抱抱脸 holds the loop 4.2-4.4 s. Whether Tavily ever returns a title or
  snippet that long is not verified.

Both are now stated next to the pool description; the code is unchanged.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): correct the configuration and preset notes

- The configuration table omitted tavily_base_url, the one config field it
  did not list.
- The quoted multi-worker refusal hard-coded hle_tools, the Diamond
  preset's instance name; refuse_unsupported_num_workers prints the
  instance's own name (hle_web_tools for configs/hle_web_tools.yaml).
- "The checks below need TAVILY_API_KEY" pointed at Example data and Tests,
  which are model-free and pass with the key unset; the key is needed for
  live runs of the preset.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_diamond): name cap-forgotten sessions in the sandbox-seeded residual

The Diamond README's residual said the 409 on re-seeding does not cover a
new session. It does not cover a session the server's cap has forgotten
either: an untyped seed through that session's cookie installs another
question (HTTP 200). The server README says so since 0cd8d9c4e; this keeps
the two residual descriptions consistent.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): use the split session locks and fences from main

Main's #5083 split ResourcesSessionFence into ResourcesSessionLocks,
which runs a session's seed and close one at a time, and
ResourcesSessionFences, which refuses a seed after its close until the
close retry window ends. The old name no longer exists, so the server
failed to import after the merge.

Map each call onto the new API without changing behaviour: locked() on
the locks, and raise_if_fenced(), get() and fence() on the fences in
place of raise_if_closed(), closed_episode() and mark_closed(). The base
config already requires a positive close retry window, so the new
constructor's check adds no refusal.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(hle_web_tools): declare dependencies in pyproject.toml and uv.lock

Main's #4837 moved every component from requirements.txt to a committed
uv project and lock, and its component dependency tests now refuse a
server that still ships requirements.txt or lacks either file.

Move the server's two requirements, nemo-gym[dev] and idna>=3.7, into a
pyproject.toml that copies the shared resolver policy, as
scripts/component_locks.py expects. The lock was seeded from the
calendar server's lock, which has the same Gym dependencies, so it pins
the same versions as the other component locks (idna 3.10, mcp 1.29.0)
instead of a fresh resolve that would move mcp, fastapi and others to
new major or minor releases.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): read a Wayback timestamp segment as a timestamp only

Since the timestamp-less Wayback reading was added, every /web/<first>/...
path was read both ways, so an ordinary timestamped snapshot also yielded
the candidate https://<timestamp>/<url>, which names no page. Each Wayback
layer then cost one to three extra candidates and more decode rounds, and a
URL at the edge of MAX_URL_CANDIDATES or MAX_URL_DECODE_ROUNDS was refused
as malformed_url where it was served before (four timestamped layers around
a percent-encoded page, or a snapshot of a page with thirty URL-valued query
values).

Add the second reading only when the first segment is not timestamp-shaped
(digits with an optional two-letter modifier such as id_ and an optional
star, or a bare star), which no hostname is. /web/scale.com/x, /web/2024/...,
/web/2024id_/... and /web/*/... keep their readings. Candidate counts return
to what they were before the second reading existed: two for a plain
timestamped snapshot. A URL-table row pins the four-layer case as served.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): protect a short question whole in every ignorable reading

QuestionSpans took its span length from the "removed" reading of the
question but built n-grams of that length from the "space" and "word_break"
readings too. A question under the threshold whose ignorable characters
(a variation selector, a zero-width space, a Hangul filler) sit inside a word
has more words in the "space" reading than in the "removed" one, so
fragments of a single word became blocked n-grams and a query that merely
contained such a fragment as a word of its own was refused, against the
README's "protected as a whole". No local HLE-Diamond question has fewer
than 8 words or readings that differ, so this is a correctness note.

Give each reading its own span length min(threshold, words in that reading)
and check the distinct lengths longest first, returning the matched length
so the refusal rule and the log name the span that actually matched rather
than the "removed" reading's length. A test seeds a one-word question with
an ignorable character inside the word in two spellings, serves a query
repeating one fragment, and refuses the whole question in each spelling
with the matched length in the log.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hle_web_tools): check the shown title text at the cut

title_variants tried the capped title (with its cut marker) and the title
before each separator that starts within the first MAX_TITLE_CHARS
characters. A separator starting exactly at the cut was excluded, and the
cut marker made the end-anchored answer_title_patterns miss a phrase that
ends exactly there, so a title whose text before " | Site" is exactly 1,000
characters and ends in "Quiz answers" was kept and rendered as shown,
while the same title one character shorter was dropped.

Add the shown text without its cut marker as a variant whenever the title
is capped, so the check covers exactly what the model sees. Every variant
stays at most MAX_TITLE_CHARS long, so the linear bound holds. A row in the
title table pins the separator-at-cut case as dropped.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): count the later sessions that evict a rollout exactly

The eviction loop pops the oldest session as soon as the table holds
MAX_SESSION_QUESTIONS + 1 entries, so a running rollout's question is gone
when the 10,000th later session is seeded, not after "more than 10,000
later seeds". A retried seed of a held session only moves it to the end,
so the count is of distinct later sessions, not of seed requests. The log
line ("more than %d seeded sessions are held") was already right.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): describe the word-break reading as the server's own

The README and the app comment attributed the third ignorable reading to
UAX #29 word segmentation in a sentence that names Hangul fillers among
the characters it removes. UAX #29 classes U+3164, U+FFA0, U+115F and
U+1160 as ALetter, letters it keeps; only format and extend characters
are skipped by WB4, and U+200B is a break. Say what the reading does and
where it follows the standard, and that the fillers are dropped because
NFKC_Casefold drops them. The range list is introduced as including
points outside category Cf rather than lying outside it, since most of the
listed ranges are Cf, and as listed with adjacent ranges merged.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(hle_web_tools): say why the stray-closer row exists, drop a duplicate row

The site:scale.com” row is the only one that catches an implementation
reading typographic quotes as a balanced pair only, so it stays, with a
comment saying what it pins. The hangul-filler-pattern row never failed
without hangul-filler (the set membership) or soft-hyphen-pattern (the
pattern path's folding) failing too, since both paths call the same
fold_text and text_block folds nothing itself, so it is dropped.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(hle_web_tools): use main's session terms and name the retire release path

main's AGENTS.md (since #5083) asks session code, tests and docs to use the
names in the key-terminology glossary and add no synonyms. The glossary
retires "typed session", "typed seed" and "Environment Server session" for
"episode session" and "episode seed", "untyped seed" for "agent-run seed",
"retry window" for "close retry window", and "free" for "release". This
branch kept the old names in comments, the 422 detail for a malformed
episode seed, a test name and fixture ID, docstrings and the README; the
merge adapted the code to the split locks and fences but not the words.

Rename them throughout. The README also said only a /close_session frees a
slot before the cap evicts it: the checkpoint controller's retire of an
attempt (retire_session_state) releases every session that attempt seeded
too, whatever seed created it, and does not fence the ID, so say so. The
"![untyped](data:...)" fixture is test data about data URIs and stays.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(hle_web_tools): wrap the nested Wayback fixture to the line limit

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(hle_web_tools): link the seed-channel residual to its tracking issue

The unauthenticated /seed_session residual is now tracked in
NVIDIA-NeMo/Gym#5845. Both READMEs point at it in place of the
TRACKING_ISSUE_TBD placeholder.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(hle_web_tools): give the true reason the server refuses several workers

The README and multiple_workers_unsupported_reason said a rollout's seed and
its searches could reach different workers. Session routing (since #3905, and
restated by #5709 in the add-benchmark skill) pins a resources request by its
cookie session_id and an MCP call by its token sid to the owning worker, so
session-ID-keyed state is not what stops a multi-worker launch. What does:
app.py defines no module-level app for uvicorn workers to import, and the
URL-keyed page cache and the page-check pool are per process. Reword both and
make the refusal test assert the new reason and the absence of the old one.
Behaviour is unchanged: num_workers > 1 is still refused before uvicorn starts.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(hle_web_tools): refresh uv.lock for main's openai 2.45.0 pin

After merging origin/main, scripts/component_locks.py --jobs 8 reported
"FAIL resources_servers/hle_web_tools: The lockfile at uv.lock needs to be
updated" and ng_test failed at uv sync --locked for the same reason, because
main's root pyproject moved openai from 2.44.0 to 2.45.0 (#5464). Regenerated
with uv lock --project resources_servers/hle_web_tools; the only change is
openai 2.44.0 -> 2.45.0. component_locks now passes 286/286.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(hle_diamond): run opencode_hle_tools as an opencode_agent session

The upstream web tools preset extended opencode_legacy_agent.yaml and so
ran opencode_sandboxed_agent behind the deprecated legacy_agent relay,
the one text preset left on it after #5702 moved the others to agent
sessions. It now extends opencode.yaml and overrides the opencode_agent
instance the way opencode_search.yaml does for Tavily:

- absolute_ip_server_types adds resources_servers, since the sandbox
  reaches the hle_tools server's MCP endpoint by address.
- The environment note moves from the sandbox file
  /tmp/nemo-gym-environment.txt to the agent's system_prompt, and
  mcp_tool_timeout_seconds is 300 for a web_fetch of a long page or PDF.
- The opencode_config reasoningEffort block goes: a session's
  opencode_config takes no model options, and the effort travels on the
  policy model server's extra_body like the other session presets.
- tool_servers ([hle_tools]) and network_access (model_and_tools) stay;
  SingleAgentTurnEnvironmentServer reads the grant from the bound agent
  and requires expose_tools_over_mcp, which the server instance sets.

sandbox_docker_local.yaml composes with the preset again (the override's
hle_diamond_opencode_agent block targets opencode_agent); its note names
no search backend now, since it serves the Tavily and the upstream web
tools preset alike.

Tests: the preset leaves LEGACY_PRESETS and joins the agent-session,
reasoning-effort and local-Docker parametrisations; the config test
asserts the session wiring and holds the agent equal to
opencode_search.yaml's apart from tool_servers and system_prompt. No
server code changes.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(hle_diamond): describe opencode_hle_tools as an agent-session preset

The benchmark README listed opencode_hle_tools among the deprecated
relay presets (Scope row, relay paragraphs, effort bullets, Known
limitations, Reading sandboxed results, Agent sessions, Reference
preset). It now sits with the session presets everywhere, and the
Upstream web tools section replaces its Deprecated relay paragraph with
the session wiring (episode seed, MCP grant, address exposure, MCP
timeout, extra_body effort), a documented run command composed like the
other presets (the Docker override composes again), and a note that the
2026-10-26 five-row Docker pilot ran on the relay harness, so no
with-tools number is quoted until a pilot is rerun on the session
harness. The residual-difference statements (backend, unauthenticated
/seed_session under model_and_tools, Gym#5845, oversized pages) are
unchanged. The hle_web_tools README's Use in a preset paragraph and the
opencode_reference.yaml header say the same.

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>

This branch was previously deployed

1 inactive deployment
public — c4e6830c Deployed Nov 3, 2026 by copy-pr-bot[bot] via release / finalize / notify #9446
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:config Hydra configuration, schemas, composition, and compatibility bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants