Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
277 commits
Select commit Hold shift + click to select a range
3ea4728
docs(fern): add v0.5.0 version snapshot for GA release (#2389)
kajalj22 Aug 6, 2026
34086bc
fix(docs): remove absolute internal URL from 0.5.0 release notes (#2397)
kajalj22 Aug 7, 2026
53541c7
docs(fern): remove NeMo Evaluator from Ecosystem page (#2413)
sephmard Aug 7, 2026
25e5898
docs(readme): add v0.5.0 to News section (#2405)
kajalj22 Aug 7, 2026
c84b7ed
perf(vllm_model): default to eight server workers (#2369)
youngeunkwon0405 Aug 7, 2026
076b673
bump: version 0.5.0 → 0.5.1 on main (#2422)
kajalj22 Aug 7, 2026
ef7699f
SWEBench resources server (#2421)
bxyu-nvidia Aug 7, 2026
e406044
Update Super 3.5 evals (#2423)
bxyu-nvidia Aug 7, 2026
df51e61
OpenCode Sandboxed agent server (#2424)
bxyu-nvidia Aug 7, 2026
d831411
[Unvalidated] SWE Bench Verified+Multilingual OpenCode benchmark (#2426)
bxyu-nvidia Aug 7, 2026
3021389
chore: update claude review to FW-CI-templates v1.8.5 (#2197)
kajalj22 Aug 7, 2026
fa5520c
feat(docker): install enroot and declare bash in Gym container (#2392)
kajalj22 Aug 7, 2026
60c736a
fix(local_vllm_model): put the server venv's bin on the Ray actor's P…
ananthsub Aug 7, 2026
de2dbda
feat: add end-to-end synthetic conversational tool-use generation and…
jkyi-nvidia Aug 8, 2026
ce3be4e
fix(opensandbox): make exec resilient to execd bind race and dead bac…
hemildesai Aug 8, 2026
a3223d3
feat: provider neutral cvdp sandbox update (#2076)
cmunley1 Aug 10, 2026
62e16b0
docs(fern): restore tutorial path redirects (#2323)
lbliii Aug 10, 2026
49c8558
fix(pinchbench): startup validation, non-clean exit handling, FEP-132…
laszkiewiczp Aug 10, 2026
3260c98
docs: improve model-call capture guide (#2415)
ritaneves Aug 10, 2026
88d1692
fix(openai_utils): accept hosted-tool responses api output items (#2438)
Atharva-Kanherkar Aug 10, 2026
abc21eb
[rollout-observability][3/7] Add OpenClaw rollout observations (#2115)
Glorf Aug 10, 2026
f499d49
docs: Add equivalence match verification guide. (#2294)
ffrujeri Aug 10, 2026
2087acb
fix(aviary_agent): add type discriminators to the mocked `/verify` re…
ananthsub Aug 10, 2026
abcb0a0
Add LiteLLM provider into ‎responses_api_models (#990)
imxj Aug 10, 2026
649abd2
fix(opensandbox): send the API key on proxied execd routes (#2462)
hemildesai Aug 10, 2026
7c37593
fix(swe_agents): fix exit-code capture and final-failure exit in open…
michal2409 Aug 11, 2026
3b31312
fix(swe_agents): forward max_output_tokens to the OpenHands llm confi…
michal2409 Aug 11, 2026
6a2fc8d
[rollout-observability][4/7] Add Hermes rollout observations (#2117)
Glorf Aug 11, 2026
a62c300
feat(docs) recipes (#2471)
ritaneves Aug 11, 2026
48b66ad
[rollout-observability][5/7] Add Pi rollout observations (#2118)
Glorf Aug 11, 2026
8514905
security: remove bundled royalty-bearing codec binaries (#2376)
kajalj22 Aug 11, 2026
aec151a
ci(gym): add shared CPU CI contract (#2271)
ko3n1g Aug 11, 2026
fa437ef
OpenCodeSandboxedAgent and SWEBench servers hardening (#2479)
bxyu-nvidia Aug 11, 2026
0051ad8
GlobalConfig resolves overrides from recursive config_paths correctly…
bxyu-nvidia Aug 11, 2026
c696231
Super 3.5 evals 20260811 update (#2480)
bxyu-nvidia Aug 11, 2026
1c56cba
[1/4] feat(environments): define manifest contract (#2417)
Glorf Aug 11, 2026
ab2d9c4
contributing guide updates (#881)
vadam5 Aug 11, 2026
28ee482
PivotRL parallel tool calls for single step argument comparison env (…
jkyi-nvidia Aug 11, 2026
bf1b7a6
build(sandbox): add httpx-aiohttp to the sandbox extra (#2487)
hemildesai Aug 11, 2026
572f035
fix(responses): handle omitted Chat tools (#2493)
ananthsub Aug 11, 2026
bb13047
fix(responses): validate token metadata as a bundle (#2488)
ananthsub Aug 11, 2026
7a9eb18
security: remove dependencies flagged in legal review (#2492)
kajalj22 Aug 11, 2026
24961cf
chore: route PR reviews to owning teams via CODEOWNERS (#2439)
anwithk Aug 11, 2026
8759321
Misc infra 20260811 (#2502)
bxyu-nvidia Aug 12, 2026
c83320b
fix(ci): lazy-load optional gprof2dot dependency (#2499)
cmunley1 Aug 12, 2026
92e5957
Add back Super 3.5 evals args to CLI rather than config (#2503)
bxyu-nvidia Aug 12, 2026
1999869
[Super 3.5 evals] Remove unnecessary curls (#2504)
bxyu-nvidia Aug 12, 2026
8b6bb9c
fix(responses): preserve reasoning effort across conversions (#2489)
ananthsub Aug 12, 2026
7afd126
fix(chat): support audio, files, and custom tools (#2491)
ananthsub Aug 12, 2026
cc5d668
fix(responses): preserve structured function outputs (#2490)
ananthsub Aug 12, 2026
2440427
feat(finance_agent_v2): add Vals Finance Agent v2 (FABv2) environment…
ushnish-de Aug 12, 2026
8b7aab9
fix(responses): remove reasoning_item from the boundary set, and name…
ananthsub Aug 12, 2026
47a0665
test(responses): check the item union and boundary sets against the i…
ananthsub Aug 12, 2026
2133abf
docs(finance_agent_v2): configure the tool API keys in env.yaml in th…
ushnish-de Aug 12, 2026
3bcf8f5
fix(deps): isolate OpenShell SDK from sandbox extra (#2432)
hemildesai Aug 12, 2026
99a21de
[2/4] feat(environments): add static manifest validation (#2441)
Glorf Aug 12, 2026
c918e14
[3/4] feat(environments): add profile-aware scaffolding (#2442)
Glorf Aug 12, 2026
12ebfc6
revert: route PR reviews to owning teams via CODEOWNERS (#2510)
ritaneves Aug 12, 2026
d2946f9
Env variable injection for basic SLURM orchestration (#2354)
oyilmaz-nvidia Aug 12, 2026
24a3878
Container mount and output parameter basic SLURM orchestration (#2419)
oyilmaz-nvidia Aug 12, 2026
ab9820c
[4/4] feat(environments): add onboarding catalog and commands (#2443)
Glorf Aug 12, 2026
d980441
fix(cli): fail a dry run when a server's venv setup exits non-zero (#…
ananthsub Aug 12, 2026
16637f4
[rollout-observability][6/7] Add OpenCode rollout observations (#2119)
Glorf Aug 12, 2026
349244f
fix(pinchbench): OpenClaw timeout configuration — invalid agents.list…
laszkiewiczp Aug 12, 2026
5c2e95b
feat: add stable public API surface to nemo_gym package (#1871)
ananthsub Aug 12, 2026
c86feed
fix(ci): add expected answer to wheel test fixture (#2516)
cmunley1 Aug 12, 2026
d4fd5e8
ci: AUT-1474 retry transient full-suite failures (#2511)
svcnemo-autobot Aug 12, 2026
f5c555f
fix: make the stdin mock iterable (#2508)
tdene Aug 13, 2026
93aeccd
fix(global_config): fail fast when the parent openai version violates…
michal2409 Aug 13, 2026
4a4688c
ci: AUT-1501 bump SSO preflight to v1.8.10 (#2544)
svcnemo-autobot Aug 13, 2026
4558578
feat(osworld): add OpenSandbox backend with scoped dependency policy …
JeffPengCoder Aug 13, 2026
4ba2d4f
fix(responses_converter): tool_call related fixes (#2515)
pachmu Aug 13, 2026
5290910
feat(sandbox): interactive PTY sessions for the OpenSandbox provider …
hemildesai Aug 13, 2026
5cc2e76
fix(sandbox): silence OpenSandbox SDK info logs (#2550)
hemildesai Aug 13, 2026
e781dc7
[browsecomp harness] adding a memory tool (#2140)
ritugala Aug 13, 2026
9eeeee5
Super 3.5 evals update 20260813 (#2549)
bxyu-nvidia Aug 13, 2026
b64a755
[Super 3.5 evals] SWE Bench Verified fixes; Rollout collection prints…
bxyu-nvidia Aug 15, 2026
ef74828
feat(benchmarks): add HLE vision (multimodal) variant (#2529)
gnalbandyan Aug 15, 2026
a6c97ce
[Super 3.5 evals] Update Aug 15 2026 (#2565)
bxyu-nvidia Aug 16, 2026
b0261ab
feat: prime-agent integration (#2406)
cmunley1 Aug 17, 2026
6c6740f
Refactor(finance_agent): one loop for both finance agent benchmarks (…
ushnish-de Aug 17, 2026
6030d4e
fix: update langgraph agents (#2149)
cmunley1 Aug 17, 2026
b50cfb1
feat(vllm-model): pin sampling params for on-policy training (#2190)
cmunley1 Aug 17, 2026
6a9f75e
fix(hermes_agent): configurable api key and chat template args (#1794)
cmunley1 Aug 17, 2026
03be216
Fix global_config: Preserve sibling order when outer config_paths ove…
ushnish-de Aug 17, 2026
120d904
feat(apptainer provider): support custom apptainer binpath (#1957)
cmunley1 Aug 17, 2026
7f3fc38
Fix global_config: keep sibling subtrees contiguous when flattening c…
ushnish-de Aug 17, 2026
54cee4f
[Super 3.5 evals] Update Aug 17 2026 (#2579)
bxyu-nvidia Aug 17, 2026
2e341db
VCQA Agent (AC) (#1997)
kbhardwaj-nvidia Aug 17, 2026
90741f3
feat(agents): add Cline CLI agent harness (#2563)
cline-test-bot Aug 18, 2026
95d9b57
Add ruler2 benchmark + ruler2 resource server (RULER v2) (#1165)
gwarmstrong Aug 18, 2026
df30df8
litmus benchmarking and profiling for Tier 1,2 and 5 ADME questions (…
OliviaViessmann Aug 18, 2026
b1e0f67
fix: preserve unknown response usage details in responses converters …
ananthsub Aug 18, 2026
4d565e1
docs: redirect retired 0.1.x and pre-nest GRPO URLs (#2518)
lbliii Aug 18, 2026
09afdc9
fix(vllm_model): prefer inline token metadata over prompt tokenizatio…
ananthsub Aug 18, 2026
098abe4
fix(docker): support non-root runtime execution (#2560)
ko3n1g Aug 18, 2026
c44bb7d
Add bbq resource server: BBQ-style social-bias comparative QA with a …
nickson-quak Aug 18, 2026
dbc9e04
ci: add 5-minute timeout to apt-get install step in unit tests (#2603)
kajalj22 Aug 18, 2026
082ebb1
feat: overridable results_dir/cache_dir roots; anchor SWE artifacts a…
michal2409 Aug 18, 2026
c3bac96
feat(video): support video inputs in vLLM responses (#2324)
ehosseiniasl Aug 19, 2026
3a5afd9
fix(workplace-assistant): install Jupyter for SDG notebook (#2590)
shashank3959 Aug 19, 2026
1f972cb
fix(opensandbox): surface OOM status after exec 502 (#2604)
hemildesai Aug 19, 2026
2251ef7
feat: option to skip verification (#2279)
macandro96 Aug 19, 2026
3dea6d4
feat(opensandbox): configure network policy (#2607)
cmunley1 Aug 19, 2026
2ea1b2b
ci: retry apt-get from Actions side instead of apt's own timeout (#2626)
kajalj22 Aug 20, 2026
b3fc4dc
feat(openair): add a 5G congestion-control environment (#2564)
adv-andrew Aug 20, 2026
9f0277c
[Super 3.5 evals] Update Aug 19 2026 (#2627)
bxyu-nvidia Aug 20, 2026
2f91507
feat: pass `max_tokens` to Hermes (#2614)
elisam0 Aug 20, 2026
0dff3d1
feat(spartqa): add SpartQA spatial-reasoning benchmark (#2072)
pachmu Aug 20, 2026
0fbef59
feat(longmemeval): add LongMemEval long-term-memory QA benchmark (#2302)
pachmu Aug 20, 2026
fbd379f
feat(pinchbench): add custom provider headers (#2598)
giuliolovisotto Aug 20, 2026
88b5e0e
feat: MLFlow exporter (#2606)
marta-sd Aug 20, 2026
267fd27
docs: Update README with detailed IHEval metrics and usage instructio…
pachmu Aug 20, 2026
500203a
fix(cli): report workload lookup errors by catalog kind (#2625)
qiaochuz-nv Aug 20, 2026
509f18d
Multi-Instance vLLM Support in gym eval submit (#2482)
oyilmaz-nvidia Aug 20, 2026
72091ad
feat(citation_if): add citation_if resources server for citation inst…
knayaka Aug 20, 2026
82f8c1e
fix: skip rewriting an up-to-date dataset metrics sidecar (#2634)
gchlebus Aug 20, 2026
4a58218
Add image tools agent and PivotRL verifier (#2624)
DanialTaheri Aug 20, 2026
0951c9b
fix(ci): make image tools agent discoverable (#2643)
ananthsub Aug 21, 2026
1db4bab
ci: add Gym GPU test workflow (#2630)
chtruong814 Aug 21, 2026
354babf
feat(token-id-capture): capture training tokens from external harness…
ananthsub Aug 21, 2026
a7c4c7b
fix(harbor): reuse one HTTP client per episode for vLLM session affin…
puneeshkhanna Aug 21, 2026
945116e
feat(token-id-capture): chain a rollout's calls into one response (#2…
ananthsub Aug 21, 2026
2910b96
fix(vllm pd): let the caller name the rollouts file (#2648)
laszkiewiczp Aug 21, 2026
ce5850d
feat(token-id-capture): deliver rebuilt trajectories safely (#2126)
ananthsub Aug 21, 2026
63aea8c
feat(openclaw_agent): improve openclaw (#2649)
elisam0 Aug 21, 2026
fa0d251
[Super 3.5 evals] Update Aug 21, 2026 (#2658)
bxyu-nvidia Aug 21, 2026
3286a8f
fix(openclaw_agent): salvage the partial trace when a run is cut shor…
elisam0 Aug 22, 2026
d005573
feat: add OpenClaw to AnyTerminal (#2650)
elisam0 Aug 22, 2026
07d6b0d
fix: secret redaction anyterminal (#2651)
elisam0 Aug 22, 2026
48d5b9c
feat: strands agent (#2420)
cmunley1 Aug 23, 2026
67a196a
[Super 3.5 evals] Update Aug 23, 2026 (#2701)
bxyu-nvidia Aug 24, 2026
5832390
Mboubdir/super 3dot5 pipecleaning (#2618)
ka00ri Aug 24, 2026
e9555a7
feat(agent): add simple agent with context compaction (#2616)
aroshanghias-nvd Aug 24, 2026
e87d5d9
fix(eval): reject -i/--input in end-to-end eval run instead of silent…
adil-a Aug 24, 2026
a2c39b6
ci: add Fireworks inference provider e2e (#2712)
chtruong814 Aug 24, 2026
89918cb
test: add GPU end-to-end CI coverage (#2662)
balasaajay Aug 24, 2026
07bb0f5
fix(openshell): keep generated sandbox names routable (#2671)
cwing-nvidia Aug 25, 2026
bacf85d
fix(mini_swe_agent_2): remove duplicate datasets key in config (#2743)
kajalj22 Aug 25, 2026
a2b56cb
docs: update standalone Python runtime examples (#2716)
cwing-nvidia Aug 25, 2026
b24d666
docs: clarify ARC-AGI setup and execution (#2735)
qiaochuz-nv Aug 25, 2026
bd19a8c
ci: stop running unit tests and container build twice per PR (#2707)
kajalj22 Aug 25, 2026
054f4e7
fix: clear failure sidecar on fresh rollout runs (#2723)
ananthsub Aug 25, 2026
233d3c0
feat: agentic math environments (#2151)
cmunley1 Aug 25, 2026
c026994
docs: note /ok to test requirement for CI triggering (#2759)
kajalj22 Aug 25, 2026
3af5828
fix(evalplus): handle non-picklable MBPP outputs (#2747)
ananthsub Aug 25, 2026
810d165
fix(longmt): derive runtime caches from Gym config (#2762)
ananthsub Aug 25, 2026
ef505dd
docs: training on rollouts from an external agent harness (#2341)
ananthsub Aug 25, 2026
438570a
feat: declare failure_reason on BaseVerifyResponse (#2552)
tdene Aug 25, 2026
9a87438
[rollout-observability][7/7] Add SWE agent rollout observations (#2120)
Glorf Aug 25, 2026
6a95a1d
Multi-node multi-instance vLLM via gym eval submit (#2597)
oyilmaz-nvidia Aug 25, 2026
00859d1
feat: judge endpoint resiliency (#2383)
tdene Aug 25, 2026
0071b5a
fix: flaky fireworks-e2e test (#2753)
kajalj22 Aug 25, 2026
286ea94
fix: bump Pillow to 12.3.0 in osworld_agent (#2751)
kajalj22 Aug 25, 2026
82673c7
feat: aviary environments (#1466)
cmunley1 Aug 25, 2026
ce3a06d
fix(docs): environment concepts page (#2708)
cmunley1 Aug 25, 2026
1298567
fix aviary readme (#2773)
cmunley1 Aug 25, 2026
0c8faef
feat(finance_sec_search): local backend to find SEC filings by full-t…
ushnish-de Aug 26, 2026
cae7594
feat(opensandbox): add run-scoped cleanup job (#2559)
hemildesai Aug 26, 2026
e8fa681
feat(sandbox): detached PTY exec — run long commands without holding …
hemildesai Aug 26, 2026
db5a15d
[Super 3.5 evals] Update Aug 25, 2026 (#2780)
bxyu-nvidia Aug 26, 2026
501a752
docs: fix single-step tutorial response output (#2702)
ananthsub Aug 26, 2026
e3133c3
feat(nemotron_3.5_super): report the model under a caller-supplied na…
laszkiewiczp Aug 26, 2026
e985256
fix(browsecomp): accept both vLLM /tokenize response shapes (#2789)
mjmikulski Aug 26, 2026
d743de8
Add rollout quality verification (#2705)
giuliolovisotto Aug 26, 2026
310d880
feat(sandbox): dedicated short timeout for background status polls (#…
hemildesai Aug 26, 2026
bb923fe
fix(opensandbox): let a caller launch the cleanup job from anywhere (…
laszkiewiczp Aug 26, 2026
91b3a14
[Browsecomp] Replacing shell access to pocket-shell (no exec) + Movin…
ritugala Aug 26, 2026
b793621
Super 3.5 vllm model tuning (Qwen3.5-122B-A10B, Inkling-Small) (#2599)
fsiino-nvidia Aug 26, 2026
65129dd
chore(deps): pin openai to 2.44.0 (#2456)
ananthsub Aug 26, 2026
f8219a5
update pivot dataset generation skill with parallel tool calls capabi…
jkyi-nvidia Aug 26, 2026
32c52f0
fix: preserve HTTP errors across process boundaries (#2726)
ananthsub Aug 26, 2026
52b8080
docs: align single-step tutorial with verifier scaffold (#2766)
ananthsub Aug 26, 2026
391e425
fix(local_vllm_model): generic configs cannot satisfy their own start…
qiaochuz-nv Aug 26, 2026
da8c2c7
fix(opensandbox): make sandbox terminate idempotent on not-found (#2760)
hemildesai Aug 26, 2026
23bd8fa
ci: temporarily disable container_build and gpu_e2e_tests jobs (#2801)
kajalj22 Aug 26, 2026
e8b26bb
test: align CI-shape tests with temporarily disabled container/gpu jo…
adil-a Aug 26, 2026
0d9f915
feat(local_vllm_model): support vLLM 0.25.1 (#2754)
ananthsub Aug 26, 2026
90579bd
perf(server_utils): avoid repeated config work in agent runs (#1404)
ananthsub Aug 26, 2026
43612db
feat(telemetry): add optional OpenTelemetry instrumentation via nemo-…
ahmadki Aug 26, 2026
4a5220b
feat(rollout): agent_map override + pre-dispatch agent validation (#2…
kajalj22 Aug 26, 2026
b3bc01b
feat(rollout): task_source resolution and run-level fan-out (#2713)
kajalj22 Aug 26, 2026
bf55b8e
feat(data): collate stamps task_source; datasets declarable on resour…
kajalj22 Aug 26, 2026
f2e0097
feat: add switchyard_model server for router-backed evals with condit…
linj-glitch Aug 26, 2026
5bf22a0
fix(scicode): write generated code to a file instead of passing it vi…
jubick1337 Aug 26, 2026
ebc81b0
[Unvalidated] Checkpoint in progress TerminalBench 2.1 benchmark; Upd…
bxyu-nvidia Aug 27, 2026
c8e67b0
feat: nemo fabric integration (#2591)
cmunley1 Aug 27, 2026
19dec43
[Performance] Rollout collection _preprocess_rows_from_config avoid d…
bxyu-nvidia Aug 27, 2026
ddb44c9
docs(fern): document rollout health checks (#2784)
giuliolovisotto Aug 27, 2026
11a9391
SWE Agents Opencode update (#2817)
sdevare-nv Aug 27, 2026
afdcc17
fix(code_gen): make LCB mock sys.stdin.buffer a real byte stream (#2824)
fgalko-oss Aug 27, 2026
4de6fe6
feat(swebench,opencode): inject CPU parallelism-cap env vars derived …
hemildesai Aug 27, 2026
e609d6c
feat: update fabric configs and output (#2821)
cmunley1 Aug 27, 2026
5de32d3
ci: enforce test coverage gate before merge, not just after (#2813)
kajalj22 Aug 27, 2026
bc84bdf
Update enroot readme (#2328)
oyilmaz-nvidia Aug 27, 2026
ce9af1d
feat(sandbox): add E2B sandbox provider (#2301)
max-sudolabs Aug 27, 2026
b622652
fix(workplace-assistant): install wcwidth for SDG notebook (#2829)
shashank3959 Aug 27, 2026
bc433e5
feat(observability): add per-rollout and per-run aggregate observabil…
mlazuka Aug 27, 2026
731e8de
feat(token-id-capture): record served response IDs (#2675)
ananthsub Aug 27, 2026
bf05c20
ci: scheduled (every 4h) Gym CI with manual trigger and single gate (…
kajalj22 Aug 27, 2026
9eeb7f0
feat(token-id-capture): attribute terminal model calls (#2676)
ananthsub Aug 27, 2026
d71871e
feat: terminus 2 agent integration (#2457)
cmunley1 Aug 27, 2026
42244c2
ci: AUT-2161 route external jobs through ephemeral v2 (#2818)
svcnemo-autobot Aug 28, 2026
dbfdfd8
Validate the local SEC sqlite index and its metadata sidecar at start…
ushnish-de Aug 28, 2026
ed5ee57
Record agent /run failures as row-associated failures (#2017)
laszkiewiczp Aug 28, 2026
61dea1f
Add repeat-level variability statistics to reward profiling (#2474)
mpatel31415 Aug 28, 2026
c02c6f9
feat: allow to swap agents when parsing the config (#2640)
marta-sd Aug 28, 2026
dc97949
feat: add --agent-type CLI flag to enable agent swaping (#2641)
marta-sd Aug 28, 2026
0b0f25e
feat: add safeguard for agent - resources server compatibility (#2661)
marta-sd Aug 28, 2026
f7bca4d
fix: rename agent instance after swapping the type (#2757)
marta-sd Aug 28, 2026
38938dd
fix(cli): add `--all` flag for `gym env test` (#2825)
marta-sd Aug 28, 2026
c8940a2
fix(cli): support top-level --json flag (#2820)
marta-sd Aug 28, 2026
4758612
feat (AgentIF): agentic instruction-following benchmark implementatio…
pachmu Aug 28, 2026
788fcae
fix: remove local openai related fix (#2839)
sdevare-nv Aug 28, 2026
685ce96
docs: fix wrong Hydra override key in reverify-rollouts tutorial (#2842)
mpatel31415 Aug 28, 2026
4b5f190
feat(token-id-capture): resolve each call's parent at request time (#…
ananthsub Aug 28, 2026
bb5c8d3
ci: AUT-2168 pin Slack rejection handling (#2834)
svcnemo-autobot Aug 28, 2026
e9094e6
feat(vllm-model): supply the previous call's exact training tokens (#…
ananthsub Aug 28, 2026
958875f
feat(data): per-server task_data schemas, gym env schema, collate val…
adil-a Aug 28, 2026
5b0aae0
feat(data): task_data schemas for all 108 resources servers (#2800)
adil-a Aug 28, 2026
b369322
feat(data): task_data validation gating: auto mode, misplaced-field c…
adil-a Aug 28, 2026
c5cab82
feat(telemetry): instrument the rollout path across Gym's server proc…
ahmadki Aug 28, 2026
40db35c
release: bump main to 0.7.0rc0 (#2847)
ananthsub Aug 28, 2026
701d961
feat(benchmarks): explicit agent pin on benchmark datasets; scaffold …
adil-a Aug 28, 2026
8a77bf7
add LMArena v3 and v2 proxy benchmarks (#2837)
jveronvialard Aug 28, 2026
d42a109
[Super 3.5 evals] Update Aug 28, 2026 (#2856)
bxyu-nvidia Aug 28, 2026
c760f10
Retry PTY takeover attach when execd's eviction times out (#2840)
hemildesai Aug 28, 2026
68cb996
[OpenSandbox] Ignore PTY reattach error (#2861)
bxyu-nvidia Aug 28, 2026
40b1001
ci: AUT-2179 temporarily disable Fireworks E2E test (#2871)
svcnemo-autobot Aug 30, 2026
1a12168
perf: serialize non-streaming responses with orjson in model server (…
ananthsub Aug 30, 2026
e044a8c
Make sidecar failure routing opt-in and loud about it (#2883)
laszkiewiczp Aug 31, 2026
26a7c67
perf: use httptools for HTTP parsing (#2869)
ananthsub Aug 31, 2026
26c6e66
docs(fern): clarify trajectory capability matrix (#2887)
Glorf Aug 31, 2026
44181d3
Mpatelka/gym compare v0 (#2635)
mpatel31415 Aug 31, 2026
023b370
fix(responses-converter): strip strict from function tools before eng…
pthombre Aug 31, 2026
b6457a5
docs: fix switchyard.mdx inaccuracies (nvbug 6674632) (#2897)
linj-glitch Aug 31, 2026
a8e0085
feat: Add SWE-bench Pro resources server. (#2498)
ffrujeri Aug 31, 2026
b1bbb57
fix(workplace-assistant): release session state after verification (#…
ananthsub Aug 31, 2026
5356cae
Remove redundancy from swe benchmark prompt (#2876)
sdevare-nv Aug 31, 2026
a2c5222
Add ViBench resources server + agent (PRD-to-web-app benchmark) (#2674)
deepbuilder Aug 31, 2026
33bcc83
[Super 3.5 Evals] Update Aug 31, 2026 (#2912)
bxyu-nvidia Sep 1, 2026
91b644e
[Unvalidated] Terminus 2 (#2913)
bxyu-nvidia Sep 1, 2026
02d978f
Add README for vibench_agent (#2906)
deepbuilder Sep 1, 2026
59b33de
fix: gym list benchmarks docs (#2819)
marta-sd Sep 1, 2026
cd6acf7
feat(cli): add strict mode for gym env validate command (#2844)
marta-sd Sep 1, 2026
5f1e687
[GDPVal] Merge main into gdpval-devel
Kh4L Sep 1, 2026
f6ac565
[GDPVal] Type chat_template_kwargs, thinking, and output_config in th…
Kh4L Sep 1, 2026
5048681
[GDPVal] Forward failure routing through multistage dispatch and drop…
Kh4L Sep 1, 2026
27858c7
[GDPVal] Canonicalize provider-bound tool history for the strict chat…
Kh4L Sep 2, 2026
ed9677a
[GDPVal] Keep audio formats open and gate request chat_template_kwarg…
Kh4L Sep 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
85 changes: 71 additions & 14 deletions .agents/skills/nemo-gym-pivot-datasets/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ name: nemo-gym-pivot-datasets
description: >-
Use when creating, validating, or documenting Nemo Gym pivot datasets from rollout,
trajectory, chat-completion, Responses API, or tool-call artifacts. Covers Gym
Responses-style row conversion, pivot selection, single-step tool-use configs,
agent_ref alignment, verifier knobs, expected-action row contracts, and train/eval usage.
Responses-style row conversion, reconstructing model calls from flattened rollout output,
parallel tool-call (function_call_batch) labels, reasoning placement and label leakage,
pivot selection, single-step tool-use configs, agent_ref alignment, verifier knobs,
expected-action row contracts, and train/eval usage.
---

# Nemo Gym Pivot Datasets
Expand All @@ -28,31 +30,46 @@ pieces needed by Gym's Responses-style row format.
## Core Workflow

1. Inspect the source data shape and count the candidate assistant decision points.
2. Identify the semantic fields needed for each pivot:
2. Reconstruct model calls from the source's flattened output list, and cross-check the count
against whatever per-trajectory model-call count the source records. A decision point is one whole
model call, so this step decides what a row even is.
3. Identify the semantic fields needed for each pivot:
- model-call input context before the pivot action
- available tools at that decision point
- expected assistant action
- reward/verifier target if it is separate from the demonstrated action
- optional provenance such as task id, source trajectory id, rollout id, uuid, depth, and original metadata
3. Convert each accepted decision point into one pivot JSONL row.
4. Generate or update the matching Gym config so the pivot-format JSONL can be used directly.
5. Validate with the bundled validator and, when available, the target Gym resource-server models.
6. Write metrics that make skipped rows, action types, tool names, depth, and provenance coverage easy to inspect.
4. Convert each accepted decision point into one pivot JSONL row.
5. Generate or update the matching Gym config so the pivot-format JSONL can be used directly.
6. Validate with the bundled validator and, when available, the target Gym resource-server models.
7. Write metrics that make skipped rows, action types, tool names, depth, and provenance coverage easy
to inspect, including the parallel / single / chat split and a batch-size histogram.

## Row Shape

Read [references/row-contract.md](references/row-contract.md) when implementing or reviewing a
converter. For `single_step_tool_use_with_argument_comparison`, the essential row fields are:

- `responses_create_params`: Responses API-style input and tool specs for the model call.
- `expected_action`: one `function_call` or one `message`.
- `agent_ref`: row-level agent routing that matches the generated config.
- `expected_action`: one `function_call`, one `function_call_batch` (parallel calls emitted in a
single response), or one `message`.
- `agent_ref`: row-level agent routing that matches the generated config. Optional when the config
routes the dataset instead.

Do not copy optional null fields into `responses_create_params`; omit them unless the target
contract explicitly wants them.

`expected_action` is singular. If a source assistant turn has more than one tool call, filter that
turn out of the pivot dataset and keep it only in a skipped-row audit if it needs review.
## One Model Call, One Row

A pivot covers one whole model call. No calls is a `message`, exactly one call is a `function_call`,
and two or more calls in the same response are one `function_call_batch` — never split across rows,
never dropped, and never a one-element batch. Tool calls take precedence over assistant text, so a
turn that narrates *and* calls is labelled with its calls.

Reconstructing "one model call" from a flattened output list is the hard part, and the same pass
decides which reasoning belongs to the prefix and which would leak the label. Read
[references/rollout-artifact-pitfalls.md](references/rollout-artifact-pitfalls.md) before writing a
converter against raw rollout artifacts.

## Conversion Patterns

Expand All @@ -67,6 +84,13 @@ to borrow from rather than canonical commands to run unchanged:
- `chat_messages_to_pivot_dataset_reference.py`: chat-completion messages to pivot rows.
- `conversational_messages_to_pivot_dataset_reference.py`: conversational message trajectories to pivot rows with reasoning/provenance handling.
- `tool_messages_to_pivot_dataset_reference.py`: message/tool-use style rows to pivot rows.
- `responses_output_to_pivot_dataset_reference.py`: Gym rollout artifacts whose trajectory is a flat
list of Responses API output items. Start here — it is self-contained and runnable, and it is the
only one that has to reconstruct model calls, so it demonstrates segmentation, parallel batch
labels, reasoning placement and `reference_output` together.

The four message-list converters were copied from real pipelines; two of them import modules that do
not exist in this repository and cannot be run as written.

## Pivot Selection

Expand All @@ -81,6 +105,10 @@ initial policy. Use at least 8 sampled local rollouts per candidate as the defau
with mixed local rewards, discard all-1 and all-0 reward groups, and if data is abundant, drop the
easiest/high-pass-rate pivots first so training concentrates on hard but learnable states.

When every source trajectory carries the same reward, the source-task filter has nothing to act on.
Report that in the metrics rather than skipping it silently, keep the full set, and rely on local
on-policy profiling for selection.

## Config And Training

Read [references/config-training-and-agent-ref.md](references/config-training-and-agent-ref.md)
Expand All @@ -90,7 +118,8 @@ Key points:

- The pivot JSONL is the training/eval dataset; point the config's train dataset entry directly at it.
- `agent_ref.name` in each row must match the agent block used by the config unless the launcher overrides routing intentionally.
- `word_count_similarity_threshold` is the main string-argument matching knob for the single-step tool-use verifier.
- `word_count_similarity_threshold` is the main string-argument matching knob for the single-step tool-use verifier. Its maximum useful value is 0.5, not 1.0: identical strings score exactly 0.5, so anything higher rejects every multi-word string argument.
- `parallel_tool_call_rewarding` must be on for `function_call_batch` datasets, or the number of calls a response makes is not part of the verdict.
- Use `tool_choice: "auto"` for these rows; `tool_choice: "required"` can route some inference engines into structured decoding paths.
- Validate configs and datasets together; a valid JSONL file can still be unusable if the agent/resource-server names do not line up.

Expand All @@ -115,5 +144,33 @@ Use `--require-field` and `--require-any-field` only when a dataset-specific wor
provenance checks. Provenance is useful for debugging and filtering, but it is not required by the
resource-server request model.

The validator accepts both supported expected-action types by default (`function_call` and `message`)
and prints an end summary split between tool-call and message pivots.
The validator accepts all three expected-action types (`function_call`, `function_call_batch` and
`message`) and prints an end summary split between parallel, single and chat pivots with a
batch-size histogram. Pass `--no-require-agent-ref` for datasets routed by their config rather than
by the row.

Two further scripts, for the cases the base validator cannot cover:

```bash
# Once a single validation pass is measured in tens of minutes, shard it by byte range.
python scripts/validate_pivot_dataset_sharded.py --path /path/to/pivot.jsonl \
--workers 32 --gym-repo /path/to/Gym --config /path/to/agent_config.yaml

# Whenever the converter had to reconstruct model calls from a flattened output list.
# With --source-path alone this reports source anomalies instead of auditing a pivot file.
python scripts/audit_pivot_turn_boundaries.py --source-path /path/to/rollouts.jsonl \
--pivot-path /path/to/pivot.jsonl
```

## Reference Loading

Load a reference only when the task needs that detail:

- [references/row-contract.md](references/row-contract.md) — writing or reviewing row fields,
`expected_action` types, provenance.
- [references/conversion-patterns.md](references/conversion-patterns.md) — the source is not already
in pivot shape and needs mapping by meaning.
- [references/rollout-artifact-pitfalls.md](references/rollout-artifact-pitfalls.md) — converting
from raw rollout artifacts, reasoning placement, validating or running very large pivot files.
- [references/config-training-and-agent-ref.md](references/config-training-and-agent-ref.md) —
writing the Gym YAML, verifier knobs, threshold tuning, pivot selection for training.
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ For `single_step_tool_use_with_argument_comparison`, the commonly tuned config b
tool_call_comparator_config:
word_count_similarity_threshold: 0.1
floating_point_comparison_threshold: 1.0e-6
# For datasets with function_call_batch labels:
parallel_tool_call_rewarding: true
allow_subset: true
allow_superset: true
parallel_tool_call_reward_mode: f1
```

Knobs:
Expand All @@ -58,13 +63,67 @@ Knobs:
split on whitespace, lowercase tokens, count token multiplicities, and compute
`intersection_count / (expected_word_total + actual_word_total)`. Lower is more permissive;
higher is stricter. This is a word-overlap threshold in the current code, not a standard IoU.

**0.5 is the ceiling, not 1.0.** The denominator is a sum, not a union, so two *identical* strings
score exactly 0.5. Any threshold above 0.5 rejects every multi-word string argument, including a
perfect reproduction — set 0.7 expecting "strict" and you get "reject everything". Note also that
the check only applies when both sides have at least two words, so short arguments are unaffected
by any threshold.

**Tune from the pairs that change verdict, not the mean.** Re-score recorded rollouts offline at
several thresholds rather than re-running inference, then read the argument pairs whose verdict
actually flips and decide which side each belongs on. A mean reward moves smoothly and tells you
nothing about whether the cut is in the right place.

**Known limitation.** An argument that is itself a JSON string — a tool whose `arguments` parameter
nests escaped JSON — is word-matched rather than compared structurally. Shared keys and ids
dominate the overlap, so a wrong enum value inside it can still pass at any usable threshold.
Either unwrap nested JSON in the converter, or accept that those arguments are only loosely
checked. No threshold setting fixes this.
- `floating_point_comparison_threshold`: absolute tolerance for float argument comparison.

### Parallel Tool-Call Knobs

`parallel_tool_call_rewarding` is the master switch and defaults to **false**. While it is off the
number of calls is not part of the verdict: every expected call must still match, but surplus calls
cost nothing. Turn it on for any dataset carrying `function_call_batch` labels — counting calls is
the point of those rows.

Once on, two stages apply. `allow_subset` and `allow_superset` are the cardinality gate, deciding
which response *shapes* are admissible at all; both default to false, so the call count must match
exactly. `parallel_tool_call_reward_mode` then decides how much credit an admissible response earns:
`binary_strict` (all or nothing), `fractional` (matched fraction), or `f1`
(`2 * matched / (expected + actual)`).

Measured against an expected set of two calls:

| response | switch off | on, `f1`, both gates open |
| --- | ---: | ---: |
| both calls, either order | 1.000 | 1.000 |
| only one of the two | 0.000 | 0.667 |
| both calls plus a surplus call | 1.000 | 0.800 |
| right count, one call wrong | 0.000 | 0.500 |

Prefer `f1` for RL. It is the only mode that penalizes missing and surplus calls symmetrically; with
permissive gates the other two reward a policy that emits one easy call and stops, or that spams
every plausible call. `resources_servers/single_step_tool_use_with_argument_comparison/configs/parallel_tool_calls_single_step_tool_use_with_argument_comparison.yaml`
is the maintained worked example.

### Re-Scoring Without Re-Running Inference

`gym eval reverify` recomputes rewards for recorded rollouts against an updated resources server,
which is how to compare verifier settings without paying for inference again. Note that a server
which does not declare `REVERIFY_MODE` defaults to `UNKNOWN`, and reverification then requires
`--force` and writes `unsafe_`-prefixed output;
`single_step_tool_use_with_argument_comparison` currently does not declare it even though its
`verify()` is a pure function of the request.

Use `tool_choice: "auto"` in pivot rows for this workflow. Avoid `tool_choice: "required"` because
some inference engines, including vLLM paths, can treat it as a structured decoding request rather
than ordinary tool-choice behavior.

Keep each pivot row to one `function_call` expected action or one `message` expected action.
Each pivot row carries exactly one `expected_action`, and that action covers the whole model call:
`function_call` for a single call, `function_call_batch` for a parallel set, `message` for text.

## Pivot Selection For Training

Expand All @@ -86,6 +145,11 @@ Default profiling rule:
Source-task mixed reward is useful but not required. A source model and downstream policy can be far
apart, so downstream local reward variance is the stronger selection signal when it is available.

When every source trajectory carries the same reward — a filtered success-only export, for instance —
this filter has nothing to act on. Say so explicitly in the metrics rather than skipping it silently,
keep the full unfiltered set, and move selection to local on-policy profiling, which is the stronger
signal anyway.

Reference: [Yi et al., "PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost"
(arXiv:2603.21383v1, 2026)](https://arxiv.org/html/2603.21383v1).

Expand All @@ -97,5 +161,10 @@ Before training:
- `agent_ref.name` matches the config.
- expected tool names exist in `responses_create_params.tools`.
- no `responses_create_params.metadata: null` rows are present.
- no one-element `function_call_batch` rows are present, and `parallel_tool_calls` is `true` wherever
batch labels are.
- every label self-scores 1.0 under the comparator this config builds.
- the boundary and reasoning-leakage audit is clean when the converter had to segment a flattened
output list.
- optional provenance is present when needed for debugging, filtering, or later analysis.
- selection metrics show how many candidates were kept, skipped, all-success, all-failure, and mixed.
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,27 @@ Common mappings:

- system/developer/user turns become Responses `message` input items.
- prior assistant text becomes completed assistant message output items.
- prior assistant reasoning can be preserved as completed reasoning items when the target agent/model path accepts it.
- prior assistant reasoning can be preserved as completed reasoning items when the target agent/model
path accepts it. Only reasoning from *earlier* turns: the pivot turn's own reasoning is part of the
action being predicted, and replaying it leaks the answer. A replayed reasoning item needs an `id`.
See [rollout-artifact-pitfalls.md](rollout-artifact-pitfalls.md).
- prior tool results become `function_call_output` items with the matching `call_id`.
- prior assistant tool calls become completed `function_call` items with `name`, `arguments`, `call_id`, and `status`.

The pivot action itself must not be included in the input prefix.

## Segmenting A Flattened Output List

A rollout artifact stores the trajectory as one flat item list, so "one model call" has to be
reconstructed before it can become a pivot. The portable rule: a tool-call batch ends only at a tool
result or a message, because the environment executes every call of one response before returning
any result. Reasoning never ends a model call. Cross-check the reconstructed count against whatever
per-trajectory model-call count the source records.

Get this wrong and a parallel batch is split across two rows, each labelled with part of what the
model did. Read [rollout-artifact-pitfalls.md](rollout-artifact-pitfalls.md) before writing the
converter, and check the result with `scripts/audit_pivot_turn_boundaries.py`.

## Tools

Preserve the tool schema visible at the pivot state. For function tools, prefer the normalized
Expand All @@ -55,15 +70,37 @@ For `single_step_tool_use_with_argument_comparison`, use these canonical action
{"type": "function_call", "name": "tool_name", "arguments": "{\"x\": 1}"}
```

```json
{"type": "function_call_batch", "calls": [{"type": "function_call", "name": "a", "arguments": "{}"},
{"type": "function_call", "name": "b", "arguments": "{}"}]}
```

```json
{"type": "message", "content": "final assistant text"}
```

Keep `arguments` as a JSON string for function calls. Validate that each argument string decodes as
JSON before writing the row.

`expected_action` is singular. If a source assistant action contains more than one tool call, filter
it out of the pivot dataset and keep it only in a skipped-row audit if it needs review.
Pick the type from what the turn did, not from a preference:

- no calls, text only → `message`
- exactly one call → `function_call`
- two or more calls in one response → `function_call_batch`, never a one-element batch

Tool calls take precedence over assistant text: a turn that narrates *and* calls is labelled with its
calls, and the text belongs in provenance. This is not a style choice — it mirrors
`resources_servers/single_step_tool_use_with_argument_comparison/common/response_utils.py::extract_action`,
which is what the verifier runs on the policy's response. A label built any other way is compared
against a shape the response side cannot produce. In one source, roughly a fifth of all pivots came
from turns that narrated and called in the same response.

## Carry The Model Call As reference_output

`expected_action` is the reduced label the verifier compares. Carrying the model call verbatim
alongside it — reasoning, assistant message, tool calls, in emission order — keeps the narration a
narrate-and-call label drops and the reasoning the prefix withholds. Carry the one call, not the
whole trajectory.

## Provenance

Expand All @@ -89,7 +126,11 @@ Write skipped-row audit artifacts when rows are rejected. Track at least:
- malformed JSON arguments
- missing tool schema
- unsupported action type
- assistant actions excluded because they contain more than one tool call
- turns whose arguments do not decode as JSON
- turns whose expected tool is missing from the tool list

Write count-and-percent summaries for action type, tool name, depth, source reward group, and any
selection filters. This catches silent data-shape changes before training.
selection filters. This catches silent data-shape changes before training. For tool-call datasets
also report the parallel / single / chat split with a batch-size histogram, the number of turns that
narrated and called in the same response, and the number of trajectories whose segmentation
disagreed with the source's own model-call count.
Loading
Loading