Skip to content

docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute deliverables dir - #4655

Closed
qiaochuz-nv wants to merge 1 commit into
mainfrom
qiaochuz/nvb-6649458-gdpval-no-metrics-exit0
Closed

qiaochuz-nv wants to merge 1 commit into
mainfrom
qiaochuz/nvb-6649458-gdpval-no-metrics-exit0

Conversation

@qiaochuz-nv

@qiaochuz-nv qiaochuz-nv commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Documents the three prerequisites a GDPVal run needs, so the benchmark README's recipe works as written on main:

  • Ray extra. The Stirrup agent declares ray_enabled = True, and since build(deps): make Ray an optional dependency #4350 Ray is an optional extra. The Stirrup README still installs with uv sync --extra dev, which leaves Ray out, so gym eval run --benchmark gdpval stops before any task.
  • Absolute deliverables directory. The config defaults persist_deliverables_dir to the relative output/gdpval/deliverables, which the Stirrup agent rejects at startup. Only the multi-stage section said to set PERSIST_DELIVERABLES_DIR.
  • Apptainer sandbox. GDPVal executes code only inside the Apptainer image built from responses_api_agents/stirrup_agent/containers/gdpval.def. The benchmark README and the config's run recipe never said so. Without the image, every task fails with GDPval requires the Apptainer sandbox and the run produces no aggregate metrics. This is the NVBug's report.

It also states that the README's commands run from the repository root, which every cd and $PWD in the README already assumes.

Fixes NVBug 6649458.
Closes #4501.

Cause

Fix

Docs only, 3 files:

  • benchmarks/gdpval/README.md: new Prerequisites for every run section.
    • Ray: uv sync --extra dev --extra ray.
    • Build the .sif, with a link to the existing verification step, and export GDPVAL_CONTAINER_PATH=....
    • export PERSIST_DELIVERABLES_DIR=... as an absolute path.
    • Judge-only runs need Ray and the deliverables directory, not the sandbox.
    • Commands run from the repository root.
    • The aggregate-metrics filename is fixed.
  • benchmarks/gdpval/config.yaml: the header run recipe installs the extra and exports both variables.
  • responses_api_agents/stirrup_agent/README.md: the install step adds --extra ray; Apptainer is required for task: gdpval and the agent never falls back to the local backend; the Quick Start command sets both variables; the config table notes the absolute-path requirement; the metrics filename is fixed.

Not changed: the relative config default. Resolving it inside the agent would disagree with the multi-stage driver, which reads the same key in another process and also requires an absolute path (resources_servers/gdpval/transport_assignment.py).

Before the fix

Gym 717c04a3b (origin/main). README recipe as written, from the repository root, with a hosted policy and --limit 2, in a checkout installed per the Stirrup README (uv sync --extra dev):

gym eval prepare --benchmark gdpval
gym eval run --model-type openai_model --benchmark gdpval --split benchmark \
  --output results/gdpval_rubric.jsonl --limit 2
Error: These servers use Ray, but Ray isn't installed where `gym` runs:
gdpval_stirrup_agent. Install with:
  pip install nemo-gym[ray]
exit 1

Same command after uv sync --extra dev --extra ray:

pydantic_core._pydantic_core.ValidationError: 1 validation error for StirrupAgentWrapper
  Value error, persist_deliverables_dir must be an absolute path (got 'output/gdpval/deliverables'). Relative paths resolve differently in the agent vs. the resources server.
exit 1

With an absolute PERSIST_DELIVERABLES_DIR and still no sandbox, the NVBug's state. The run now fails loudly through #2017's guard:

(gdpval_stirrup_agent) [stirrup-legitimate] 83d10b06-...: RuntimeError: GDPval requires the Apptainer sandbox: set `gdpval_container_path` to the .sif built from responses_api_agents/stirrup_agent/containers/gdpval.def. ...
RuntimeError: None of the 2 dispatched rollouts produced a result {'legitimate': 2}. Inspect .../gdpval_rubric_failures.jsonl; the run has no score to report.

After the fix

Gym 76937de3e (this branch, on 717c04a3b). Following the new prerequisites, with the image built by the README's apptainer build line and --limit 1:

uv sync --extra dev --extra ray
export GDPVAL_CONTAINER_PATH=$PWD/responses_api_agents/stirrup_agent/containers/gdpval.sif
export PERSIST_DELIVERABLES_DIR=$PWD/output/gdpval/deliverables
gym eval prepare --benchmark gdpval
gym eval run --model-type openai_model --benchmark gdpval --split benchmark \
  --output results/gdpval_rubric.jsonl --limit 1
(gdpval_stirrup_agent) [gdpval] Using Apptainer container /abs/path/to/gdpval.sif for task 83d10b06-26d1-4636-a32c-23f92c57f30b
Finished rollout collection! View results at:
Aggregate metrics: results/gdpval_rubric_aggregate_metrics.json
exit 0

gdpval_rubric_aggregate_metrics.json key_metrics:
  {"mean/reward": 0.68, "mean/invalid_judge_response": 0.0, "rubric/aggregate_rows_total": 1, "rubric/aggregate_rows_included": 1, ...}

Validation

  • New QA testcase gdpval (nmfw_tests, follow-up MR). On main 717c04a3b it passes 1/3 (only the no-sandbox case, thanks to Record agent /run failures as row-associated failures #2017); on this branch it passes 3/3 (1063 s). Its cases:
    1. The README states all three prerequisites.
    2. Without the sandbox, the README run exits nonzero, every row fails with the sandbox error, and no metrics file is written.
    3. With the image, the README run scores one task. This asserts successful tool calls in the sandbox, a valid rubric verdict, reward equal to the judge's overall score, persisted deliverables, and mean/reward in _aggregate_metrics.json.
  • gdpval.sif built from containers/gdpval.def with Apptainer 1.5.4: all 419 AA v2 pins present. It passes verify_gdpval_sandbox.py.
  • tests/unit_tests/test_fern_docs_links.py: 24 passed. The change is outside fern/; this ran as a guard.
  • pre-commit run --all-files: passed.
  • Rollout evidence: the sandboxed QA runs use policy nvidia/nvidia/Nemotron-3-Nano-30B-A3B and the default judge panel. In the run above, 28 code_exec and 42 stateful_python_code_exec calls ran in the sandbox, the model delivered Sample.xlsx, and the panel's claude-opus-4.8 judge scored 38 rubric criteria for a reward of 0.68 with invalid_judge_response=false. Earlier runs of the same task on this change scored 0.02, 0.68, 0.72, 0.30 and 0.68 at temperature 1.0; the spread is the model's, and the plumbing assertions held every time.

Compatibility

None. This changes docs and a YAML comment only. Runs that already install Ray and set both variables behave the same.

🤖 Generated with Claude Code

@qiaochuz-nv qiaochuz-nv added bug Something isn't working QA_fix Fix implemented by a QA team member area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers needs-review PR is ready for code review and waiting on a reviewer complexity:low Localized change in one scope with a small, straightforward review surface labels Oct 15, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@qiaochuz-nv
qiaochuz-nv requested a review from yaoyu-33 October 15, 2026 21:54
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 321faa4

… deliverables dir

The GDPVal run recipe could not work as written on main:

- The benchmark config defaults persist_deliverables_dir to the relative
  output/gdpval/deliverables, which the Stirrup agent rejects at startup.
- With an absolute PERSIST_DELIVERABLES_DIR, every task fails with
  "GDPval requires the Apptainer sandbox" because GDPVAL_CONTAINER_PATH is
  unset; the run then stops with "None of the N dispatched rollouts
  produced a result" and no aggregate metrics.
- Since Ray became an optional extra (#4350), a `uv sync --extra dev`
  install stops before the run because the Stirrup agent needs Ray.

Add a "Prerequisites for every run" section to the benchmark README and the
config's run recipe. Correct the Stirrup README, which still called
Apptainer optional, said the agent falls back to a local sandbox, and
installed Gym without Ray, and fix the aggregate-metrics filename
(<stem>_aggregate_metrics.json) in both READMEs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
@qiaochuz-nv
qiaochuz-nv force-pushed the qiaochuz/nvb-6649458-gdpval-no-metrics-exit0 branch from 321faa4 to 76937de Compare October 15, 2026 22:27
@qiaochuz-nv qiaochuz-nv added docs Documentation-only updates or documentation debt and removed bug Something isn't working labels Oct 15, 2026
@qiaochuz-nv qiaochuz-nv changed the title fix(gdpval): document run prerequisites and prepare benchmarks from any cwd docs(gdpval): document the Ray extra, Apptainer sandbox, and absolute deliverables dir Oct 15, 2026
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

/ok to test 76937de

@ananthsub ananthsub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to collide with #4178, which will significantly change gpdval

@ananthsub
ananthsub requested a review from marta-sd October 15, 2026 23:19
@qiaochuz-nv

Copy link
Copy Markdown
Contributor Author

Closing: as @ananthsub noted, this collides with #4178, which reworks GDPval. The run prerequisites documented here (the ray extra, an absolute PERSIST_DELIVERABLES_DIR, and GDPVAL_CONTAINER_PATH) belong in that rework's docs. The NVBug's reported exit-0-with-no-metrics is already fixed on main by #2017 and is being closed as verified.

@qiaochuz-nv
qiaochuz-nv deleted the qiaochuz/nvb-6649458-gdpval-no-metrics-exit0 branch October 15, 2026 23:48

This branch was previously deployed

1 inactive deployment
public — 76937de3 Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5580
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:low Localized change in one scope with a small, straightforward review surface docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer QA_fix Fix implemented by a QA team member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[NVBug 6649458] Document GDPval Apptainer prerequisite; don't exit 0 with no metrics

2 participants