Repository navigation
docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute deliverables dir - #4655
Closed
qiaochuz-nv wants to merge 1 commit into
Closed
qiaochuz-nv wants to merge 1 commit into
qiaochuz-nv wants to merge 1 commit into
Conversation
Contributor
Author
|
/ok to test 321faa4 |
… deliverables dir The GDPVal run recipe could not work as written on main: - The benchmark config defaults persist_deliverables_dir to the relative output/gdpval/deliverables, which the Stirrup agent rejects at startup. - With an absolute PERSIST_DELIVERABLES_DIR, every task fails with "GDPval requires the Apptainer sandbox" because GDPVAL_CONTAINER_PATH is unset; the run then stops with "None of the N dispatched rollouts produced a result" and no aggregate metrics. - Since Ray became an optional extra (#4350), a `uv sync --extra dev` install stops before the run because the Stirrup agent needs Ray. Add a "Prerequisites for every run" section to the benchmark README and the config's run recipe. Correct the Stirrup README, which still called Apptainer optional, said the agent falls back to a local sandbox, and installed Gym without Ray, and fix the aggregate-metrics filename (<stem>_aggregate_metrics.json) in both READMEs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
qiaochuz-nv
force-pushed
the
qiaochuz/nvb-6649458-gdpval-no-metrics-exit0
branch
from
October 15, 2026 22:27
321faa4 to
76937de
Compare
Contributor
Author
|
/ok to test 76937de |
Contributor
Author
|
Closing: as @ananthsub noted, this collides with #4178, which reworks GDPval. The run prerequisites documented here (the |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Documents the three prerequisites a GDPVal run needs, so the benchmark README's recipe works as written on
main:ray_enabled = True, and since build(deps): make Ray an optional dependency #4350 Ray is an optional extra. The Stirrup README still installs withuv sync --extra dev, which leaves Ray out, sogym eval run --benchmark gdpvalstops before any task.persist_deliverables_dirto the relativeoutput/gdpval/deliverables, which the Stirrup agent rejects at startup. Only the multi-stage section said to setPERSIST_DELIVERABLES_DIR.responses_api_agents/stirrup_agent/containers/gdpval.def. The benchmark README and the config's run recipe never said so. Without the image, every task fails withGDPval requires the Apptainer sandboxand the run produces no aggregate metrics. This is the NVBug's report.It also states that the README's commands run from the repository root, which every
cdand$PWDin the README already assumes.Fixes NVBug 6649458.
Closes #4501.
Cause
tasks/gdpval.pyget_exec_provider,app.pytool setup). The Stirrup README still carried Add Stirrup agent + GDPVal eval/RL environment #1090's wording from when the sandbox was optional: "(Optional) Install Apptainer" and "By default the agent uses a local sandbox".persist_deliverables_dirdefault and the agent's absolute-path check both date from Add Stirrup agent + GDPVal eval/RL environment #1090.--extra rayon the install pages. The GDPVal and Stirrup READMEs were not updated.<output>_metrics.json. Gym writes<output stem>_aggregate_metrics.json.main. Record agent /run failures as row-associated failures #2017 made a run with zero scored rollouts raiseNone of the N dispatched rollouts produced a result. This PR changes no runtime code.Fix
Docs only, 3 files:
benchmarks/gdpval/README.md: new Prerequisites for every run section.uv sync --extra dev --extra ray..sif, with a link to the existing verification step, andexport GDPVAL_CONTAINER_PATH=....export PERSIST_DELIVERABLES_DIR=...as an absolute path.benchmarks/gdpval/config.yaml: the header run recipe installs the extra and exports both variables.responses_api_agents/stirrup_agent/README.md: the install step adds--extra ray; Apptainer is required fortask: gdpvaland the agent never falls back to the local backend; the Quick Start command sets both variables; the config table notes the absolute-path requirement; the metrics filename is fixed.Not changed: the relative config default. Resolving it inside the agent would disagree with the multi-stage driver, which reads the same key in another process and also requires an absolute path (
resources_servers/gdpval/transport_assignment.py).Before the fix
Gym
717c04a3b(origin/main). README recipe as written, from the repository root, with a hosted policy and--limit 2, in a checkout installed per the Stirrup README (uv sync --extra dev):Same command after
uv sync --extra dev --extra ray:With an absolute
PERSIST_DELIVERABLES_DIRand still no sandbox, the NVBug's state. The run now fails loudly through #2017's guard:After the fix
Gym
76937de3e(this branch, on717c04a3b). Following the new prerequisites, with the image built by the README'sapptainer buildline and--limit 1:Validation
gdpval(nmfw_tests, follow-up MR). Onmain717c04a3bit passes 1/3 (only the no-sandbox case, thanks to Record agent /run failures as row-associated failures #2017); on this branch it passes 3/3 (1063 s). Its cases:mean/rewardin_aggregate_metrics.json.gdpval.sifbuilt fromcontainers/gdpval.defwith Apptainer 1.5.4:all 419 AA v2 pins present. It passesverify_gdpval_sandbox.py.tests/unit_tests/test_fern_docs_links.py: 24 passed. The change is outsidefern/; this ran as a guard.pre-commit run --all-files: passed.nvidia/nvidia/Nemotron-3-Nano-30B-A3Band the default judge panel. In the run above, 28code_execand 42stateful_python_code_execcalls ran in the sandbox, the model deliveredSample.xlsx, and the panel'sclaude-opus-4.8judge scored 38 rubric criteria for a reward of 0.68 withinvalid_judge_response=false. Earlier runs of the same task on this change scored 0.02, 0.68, 0.72, 0.30 and 0.68 at temperature 1.0; the spread is the model's, and the plumbing assertions held every time.Compatibility
None. This changes docs and a YAML comment only. Runs that already install Ray and set both variables behave the same.
🤖 Generated with Claude Code