Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d2578c0
feat(environment): bind the agent's resources server when single_agen…
yaoyu-33 Oct 23, 2026
6c84687
feat(scripts): switch simple_agent relays to single_agent_turn_legacy
yaoyu-33 Oct 23, 2026
48e8af1
fix(scripts): keep relays the switch would split or cannot drive
yaoyu-33 Oct 23, 2026
f22839c
fix(scripts): keep every same-name declaration of a split relay
yaoyu-33 Oct 23, 2026
07650ac
fix(config): preserve benchmark relay names in harness overrides
yaoyu-33 Oct 27, 2026
8ac8075
chore(configs): move simple_agent configs off the legacy_agent relay
yaoyu-33 Oct 23, 2026
3950e9c
fix(configs): keep genrm_compare on the relay and soften the docs claim
yaoyu-33 Oct 23, 2026
31f1c2a
test(super35): expect 16 core server environments after the switch
yaoyu-33 Oct 23, 2026
7c401e6
fix(agent): report simple_agent session tool calls as observations
yaoyu-33 Oct 23, 2026
00788d9
fix(config): relay session-less agents swapped onto single_agent_turn…
yaoyu-33 Oct 23, 2026
d68eebb
refactor(configs): retire tool_simulation_agent and non_executing_sim…
yaoyu-33 Oct 23, 2026
daf0ca2
fix(configs): drop the unused tool grant from non-executing simple_ag…
yaoyu-33 Oct 23, 2026
00a6819
fix(env): own resources bindings and resolve inherited renames
yaoyu-33 Oct 27, 2026
1da5791
Merge main (773c056cd) into codex/env-resources-bindings
weijiac0619 Nov 2, 2026
1525c46
fix(configs): bind Resources Servers for environment servers switched…
weijiac0619 Nov 2, 2026
86a11b4
fix(configs): bind renamed Resources Servers in benchmarks that compose
weijiac0619 Nov 3, 2026
c4e6830
test(config): import the binding test's parser from nemo_gym.config
weijiac0619 Nov 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions benchmarks/aalcr/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ aalcr_benchmark_environment_server:
environment_servers:
single_agent_turn_legacy:
entrypoint: app.py
resources_server:
type: resources_servers
name: aalcr_benchmark_resources_server
agent_server:
type: responses_api_agents
name: aalcr_benchmark_simple_agent
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/aalcr/config_v1_1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ aalcr_benchmark_environment_server:
environment_servers:
single_agent_turn_legacy:
entrypoint: app.py
resources_server:
type: resources_servers
name: aalcr_benchmark_resources_server
agent_server:
type: responses_api_agents
name: aalcr_benchmark_simple_agent
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/aegis_v4_safety/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,8 @@ aegis_v4_safety_benchmark_environment_server:
_inherit_from: aegis_v4_safety_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aegis_v4_safety_benchmark_resources_server
agent_server:
name: aegis_v4_safety_benchmark_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/agentif/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,8 @@ agentif_benchmark_environment_server:
_inherit_from: agentif_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: agentif
agent_server:
name: agentif_benchmark_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/aime24-x/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ aime24-x_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aime24-x_math_with_judge_resources_server
agent_server:
name: aime24-x_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/aime24/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,8 @@ aime24_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aime24_math_with_judge_resources_server
agent_server:
name: aime24_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/aime25-x/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ aime25-x_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aime25-x_math_with_judge_resources_server
agent_server:
name: aime25-x_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/aime25/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,8 @@ aime25_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aime25_math_with_judge_resources_server
agent_server:
name: aime25_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/aime26/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,8 @@ aime26_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: aime26_math_with_judge_resources_server
agent_server:
name: aime26_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/answer-judge/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ answer_judge_math_proof_judgement_environment_server:
_inherit_from: math_proof_judgement_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: answer_judge_math_proof_judgement_resources_server
agent_server:
name: answer_judge_math_proof_judgement_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/apex25/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,8 @@ apex25_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: apex25_math_with_judge_resources_server
agent_server:
name: apex25_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/apex25/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,8 @@ apex25_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: apex25_math_with_judge_resources_server
agent_server:
name: apex25_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/apex_shortlist/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ apex_shortlist_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: apex_shortlist_math_with_judge_resources_server
agent_server:
name: apex_shortlist_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arena_hard/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ arena_hard_arena_judge_environment_server:
_inherit_from: arena_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arena_hard_arena_judge_resources_server
agent_server:
name: arena_hard_arena_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arena_hard_v2/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ arena_hard_v2_arena_judge_environment_server:
_inherit_from: arena_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arena_hard_v2_arena_judge_resources_server
agent_server:
name: arena_hard_v2_arena_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0426/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ arxivmath_0426_math_with_autograder_environment_server:
_inherit_from: math_with_autograder_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0426_math_with_autograder_resources_server
agent_server:
name: arxivmath_0426_math_with_autograder_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0426/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ arxivmath_0426_math_with_autograder_environment_server:
_inherit_from: math_with_autograder_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0426_math_with_autograder_resources_server
agent_server:
name: arxivmath_0426_math_with_autograder_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0526/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,8 @@ arxivmath_0526_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0526_math_with_judge_resources_server
agent_server:
name: arxivmath_0526_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0526/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,8 @@ arxivmath_0526_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0526_math_with_judge_resources_server
agent_server:
name: arxivmath_0526_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0626/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,8 @@ arxivmath_0626_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0626_math_with_judge_resources_server
agent_server:
name: arxivmath_0626_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/arxivmath_0626/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,8 @@ arxivmath_0626_math_with_judge_environment_server:
_inherit_from: math_with_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: arxivmath_0626_math_with_judge_resources_server
agent_server:
name: arxivmath_0626_math_with_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/asr_leaderboard/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ asr_leaderboard_asr_with_pc_environment_server:
_inherit_from: asr_with_pc_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: asr_leaderboard_asr_with_pc_resources_server
agent_server:
name: asr_leaderboard_asr_with_pc_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/assaybench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ assaybench_assaybench_environment_server:
_inherit_from: assaybench_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: assaybench_assaybench_resources_server
agent_server:
name: assaybench_assaybench_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/bigcodebench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ bigcodebench_benchmark_environment_server:
environment_servers:
single_agent_turn_legacy:
entrypoint: app.py
resources_server:
type: resources_servers
name: bigcodebench
agent_server:
type: responses_api_agents
name: bigcodebench_benchmark_simple_agent
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/birdbench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,8 @@ birdbench_bird_sql_environment_server:
_inherit_from: bird_sql_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: birdbench_bird_sql_resources_server
agent_server:
name: birdbench_bird_sql_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0426/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ brokenarxiv_0426_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0426_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0426_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0426/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ brokenarxiv_0426_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0426_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0426_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0526/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,8 @@ brokenarxiv_0526_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0526_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0526_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0526/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,8 @@ brokenarxiv_0526_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0526_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0526_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0626/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,8 @@ brokenarxiv_0626_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0626_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0626_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/brokenarxiv_0626/config_with_system.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,8 @@ brokenarxiv_0626_false_statement_judge_environment_server:
_inherit_from: false_statement_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: brokenarxiv_0626_false_statement_judge_resources_server
agent_server:
name: brokenarxiv_0626_false_statement_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/bunsenbench_chemistry_mcq/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,8 @@ bunsenbench_chemistry_mcq_benchmark_environment_server:
_inherit_from: bunsenbench_chemistry_mcq_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: bunsenbench_chemistry_mcq_resources_server
agent_server:
name: bunsenbench_chemistry_mcq_benchmark_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/chemreason_bench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ chemreason_bench_chemreason_bench_environment_server:
_inherit_from: chemreason_bench_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: chemreason_bench_chemreason_bench_resources_server
agent_server:
name: chemreason_bench_chemreason_bench_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/financebench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,5 +51,8 @@ financebench_environment_server:
_inherit_from: equivalence_llm_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: financebench_judge_resources_server
agent_server:
name: financebench_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/flores200/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ flores200_wmt_translation_environment_server:
_inherit_from: wmt_translation_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: flores200_wmt_translation_resources_server
agent_server:
name: flores200_wmt_translation_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/frontierscience_olympiad/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,8 @@ frontierscience_olympiad_frontierscience_judge_environment_server:
_inherit_from: frontierscience_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: frontierscience_olympiad_frontierscience_judge_resources_server
agent_server:
name: frontierscience_olympiad_frontierscience_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/frontierscience_research/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,8 @@ frontierscience_research_frontierscience_judge_environment_server:
_inherit_from: frontierscience_judge_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: frontierscience_research_frontierscience_judge_resources_server
agent_server:
name: frontierscience_research_frontierscience_judge_simple_agent
3 changes: 3 additions & 0 deletions benchmarks/fukuyamabench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ fukuyamabench_benchmark_environment_server:
_inherit_from: fukuyamabench_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: fukuyamabench
agent_server:
name: fukuyamabench_benchmark_agent
3 changes: 3 additions & 0 deletions benchmarks/genrm_hs3/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ genrm_hs3_environment_server:
_inherit_from: genrm_eval_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: genrm_hs3_resources_server
agent_server:
name: genrm_hs3_benchmark_agent
3 changes: 3 additions & 0 deletions benchmarks/genrm_hs4/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ genrm_hs4_environment_server:
_inherit_from: genrm_eval_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: genrm_hs4_resources_server
agent_server:
name: genrm_hs4_benchmark_agent
3 changes: 3 additions & 0 deletions benchmarks/genrm_judgebench/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ genrm_judgebench_environment_server:
_inherit_from: genrm_eval_environment_server
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: genrm_judgebench_resources_server
agent_server:
name: genrm_judgebench_benchmark_agent
7 changes: 7 additions & 0 deletions benchmarks/genrm_mmrb1/llm_as_a_judge.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,10 @@ genrm_mmrb1_resources_server:
resources_servers:
genrm_eval:
verdict_format: brackets

genrm_mmrb1_environment_server:
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: genrm_mmrb1_resources_server
7 changes: 7 additions & 0 deletions benchmarks/genrm_mmrb2/llm_as_a_judge.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,10 @@ genrm_mmrb2_resources_server:
resources_servers:
genrm_eval:
verdict_format: json_ab

genrm_mmrb2_environment_server:
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: genrm_mmrb2_resources_server
Loading
Loading