Skip to content

Add PinchBench agentic benchmark integration - #1810

Merged
gchlebus merged 8 commits into
NVIDIA-NeMo:mainfrom
hkumar92:hshivkumar/add-pinchbench
Jul 14, 2026
Merged

gchlebus merged 8 commits into
NVIDIA-NeMo:mainfrom
hkumar92:hshivkumar/add-pinchbench

Conversation

@hkumar92

Copy link
Copy Markdown
Contributor

Adds PinchBench — an LLM-as-OpenClaw-agent benchmark over 147 real-world tasks (calendar, email triage, CSV/log analysis, coding, research, writing), graded by deterministic checks, an LLM judge, or both — as an agent-server integration under responses_api_agents/pinchbench/.

How

  • Agent-server level (same shape as swe_agents / harbor_agent): OpenClaw + PinchBench's own rollout/grading harness is the orchestration, so we wrap it rather than reimplement it.
  • One sandbox per task via the provider-neutral nemo_gym.sandbox API (apptainer / opensandbox) — each task gets its own ~/.openclaw + OpenClaw gateway, which is cliff-proof (a wedged task can't affect others; a gateway shared across all 147 tasks hits WorkspaceVanishedError).
  • Skill not vendored: Dockerfile.benchmark clones skill@v2.0.0 at build and applies a small NVIDIA endpoint-integration patch (scripts/lib_agent.py only: custom OpenAI-compatible provider, JUDGE_BASE_URL/JUDGE_API_KEY routing, brave/tavily web-search config, transcript-path resolution).
  • Returns continuous reward + grading_breakdown + grading_notes + the full OpenClaw transcript in raw_rollout.

Validation

Vs vanilla standalone PinchBench on Nemotron-3-Nano-30B-A3B (n=3, max_tokens=65536, temperature=1.0): Gym 0.583 ± 0.025 vs vanilla 0.564 ± 0.013 → parity. The apptainer-sandbox path was reproduced on a 10-task subset (~0.66); a full n=3 sandbox run is in progress.

Notes

  • OpenClaw pinned 2026.6.5. Evaluation-only (OpenClaw exposes no token IDs/logprobs).
  • DCO signed off; co-authored with @AWarno (rollout usage accounting).

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Jun 29, 2026
@@ -0,0 +1,5 @@
{"responses_create_params":{"input":[{"role":"user","content":"<placeholder; real prompt loaded from the task .md at run time>"}]},"verifier_metadata":{"task_id":"task_sanity"}}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could u upload 5 example rollouts? e.g. https://docs.nvidia.com/nemo/gym/main/contribute/environments/new-environment#4-generate-example-rollouts

Also, is there a way to include the real prompts here rather than placeholders? At the least, a task index per row that is used to load each task? I think we need some control over which tasks we run, e.g. if we wanted to run a subset or something

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI,

  • Added data/example_rollouts.jsonl — 5 example rollouts (the 5 example tasks through the agent: sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89).
  • Embedded each task's real prompt (the skill .md ## Prompt section) in input instead of the placeholder in example rollouts.
  • On task control: each row carries verifier_metadata.task_id, which is the selector — run_task.sh runs benchmark.py --suite <task_id>, and you run a subset by including only the rows you want. The full task (prompt + assets + grading) loads from the skill by task_id, so the JSONL stays small.

@gchlebus gchlebus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Jul 14, 2026
Haran Kumar Shiv Kumar and others added 8 commits July 14, 2026 13:30
PinchBench (https://github.com/pinchbench/skill) measures an LLM as the brain of an
OpenClaw agent across 147 real-world tasks (calendar, email, CSV/log analysis, coding,
research, writing), graded by deterministic checks, an LLM judge, or both.

Integrated at the agent-server level (like swe_agents / harbor_agent): OpenClaw plus
PinchBench's own rollout/grading harness is the orchestration, wrapped rather than
reimplemented. One sandbox per task (via nemo_gym.sandbox; apptainer / opensandbox) is
the isolation boundary -- each task gets its own ~/.openclaw + OpenClaw gateway, avoiding
the shared-workspace WorkspaceVanishedError cliff. The skill is not vendored: the image
clones skill@v2.0.0 at build and applies a small NVIDIA endpoint-integration patch
(scripts/lib_agent.py: custom OpenAI-compatible provider, JUDGE_BASE_URL/JUDGE_API_KEY
routing, brave/tavily web-search config, transcript-path resolution).

Validated vs vanilla standalone PinchBench on Nemotron-3-Nano-30B-A3B (n=3, max_tokens
65536, temperature 1.0): Gym 0.583 +/- 0.025 vs vanilla 0.564 +/- 0.013 -> parity.

Co-Authored-By: Anna Warno <awarno@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Address review on data/example.jsonl:
- Add data/example_rollouts.jsonl: 5 example rollouts (the 5 example tasks run
  through the agent — sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89).
- Embed each task's real prompt (the skill .md `## Prompt` section) in `input`
  instead of a placeholder; dataset_preprocess.py now extracts it.
- Clarify that `verifier_metadata.task_id` is the authoritative per-row selector:
  run_task.sh runs `benchmark.py --suite <task_id>`, and a subset is run by
  including only those rows. README + docstring updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Fix a concurrency bug: run_task.sh started the gateway on the fixed default port
(18789). Under a provider that shares the host network namespace (apptainer),
concurrent sandboxes then collide -- only the first gateway binds, the rest fail
with EADDRINUSE -> no agent -> empty transcript -> reward 0. Since the agent's
default max_concurrent is 4, the default config was broken under apptainer.

Give each sandbox a unique gateway port (`openclaw config set gateway.port` +
`--port`); the in-sandbox `openclaw agent` client reads gateway.port from the
per-sandbox $HOME/.openclaw, so gateway and client agree. Harmless for opensandbox
(each container has its own netns, so the port was never shared). Verified with 3
concurrent models (1 sandbox each): all produce real scores, zero EADDRINUSE.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Failed tasks were returned as reward-0 rows in the main jsonl, so
resume-after-walltime counted in-flight kills as done and never retried
them. Classify failures like stirrup_agent: signal-killed sandbox execs
are kill_shaped (not persisted, re-dispatched on resume), per-task
timeouts are terminal sidecar rows, everything else goes to the sidecar
with bounded retry.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
The sentinel-free-row regression test fails on the pre-fix agent (a
failed task produced a row the dispatcher writes to the main jsonl);
the rest pin kill_shaped/timeout/legitimate routing and that scored
rollouts stay sentinel-free. Placeholder keys in the existing test
fixture carry detect-secrets allowlist pragmas.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Drop the inline allowlist pragmas (test fixture lines stay as authored)
and register the placeholder keys in the detect-secrets baseline instead;
compress the added comments and remove an unused test helper.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
@hkumar92
hkumar92 force-pushed the hshivkumar/add-pinchbench branch from a2a7d31 to 76583bb Compare July 14, 2026 17:33
@gchlebus
gchlebus merged commit f2d3741 into NVIDIA-NeMo:main Jul 14, 2026
12 of 15 checks passed
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Jul 14, 2026
laszkiewiczp added a commit that referenced this pull request Aug 3, 2026
#2259)

## Summary

PinchBench shipped as an agent server
(`responses_api_agents/pinchbench`, #1810) but has no entry under
`benchmarks/`, so it is neither discoverable via `gym list benchmarks`
nor preparable via `gym eval prepare`. This adds the benchmark
definition.

- `config.yaml` chains `config_paths` to the agent config and narrows it
to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2`
shape, the existing precedent for a benchmark chaining to a
`responses_api_agents/` config.
- `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref
`Dockerfile.benchmark` bakes into the per-task image, and writes all 147
manifest tasks.
- `num_repeats: 3`, matching the n=3 parity validation in the agent
README (reward is continuous, and temperature 1.0 + live web search make
a single pass noisy).

## Notes for reviewers

**Run-time interpolations.** The agent config interpolates eight knobs
(`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not
resolve on their own, so `gym eval prepare` would fail on a bare config.
They are declared top-level with `oc.env:...,null` defaults. This does
not break the documented `+sandbox_image=...` invocation: Hydra parses
CLI args against an empty base *before* `config_paths` loads, and CLI
merges last. A `null` still reaching the agent fails fast, since those
fields are required `str` in `PinchBenchAgentConfig`.

**Listing metadata on the base node.** `domain`/`description`/`value`
are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml`
rather than the benchmark config, because adding them to the inheriting
node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth
knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so
`ng_list_benchmarks` renders such a key fine and only the real
prepare/run path trips the check. The `README.md` row is regenerated by
the `update-readme-table` hook.

**Drift guards.** Preparation fails loudly if the manifest is not 147
tasks, or if the regenerated rows do not byte-match the committed
`responses_api_agents/pinchbench/data/example.jsonl` — so upstream
prompt drift and divergence from `dataset_preprocess.py` cannot pass
silently.

## Verification

- `ng_prepare_benchmark
"+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via
both the live-clone path and `PINCHBENCH_SKILL_DIR`
- Rows validated: all carry a non-empty `responses_create_params.input`,
`task_id`s unique
- `ng_list_benchmarks` lists `pinchbench` with domain `agent` and
`num_repeats` 3
- Drift guard negative-tested (tampered prompt raises)
- `pre-commit run` clean on all touched files
- Generated JSONL is covered by the existing `benchmarks/.gitignore`

## Related

Cherry-picked to `r0.5.0` in a companion PR (that release branch carries
the agent; `r0.4.0` does not, as PinchBench landed on main after the
cut).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
svcnvidia-nemo-ci pushed a commit that referenced this pull request Aug 3, 2026
#2259)

## Summary

PinchBench shipped as an agent server
(`responses_api_agents/pinchbench`, #1810) but has no entry under
`benchmarks/`, so it is neither discoverable via `gym list benchmarks`
nor preparable via `gym eval prepare`. This adds the benchmark
definition.

- `config.yaml` chains `config_paths` to the agent config and narrows it
to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2`
shape, the existing precedent for a benchmark chaining to a
`responses_api_agents/` config.
- `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref
`Dockerfile.benchmark` bakes into the per-task image, and writes all 147
manifest tasks.
- `num_repeats: 3`, matching the n=3 parity validation in the agent
README (reward is continuous, and temperature 1.0 + live web search make
a single pass noisy).

## Notes for reviewers

**Run-time interpolations.** The agent config interpolates eight knobs
(`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not
resolve on their own, so `gym eval prepare` would fail on a bare config.
They are declared top-level with `oc.env:...,null` defaults. This does
not break the documented `+sandbox_image=...` invocation: Hydra parses
CLI args against an empty base *before* `config_paths` loads, and CLI
merges last. A `null` still reaching the agent fails fast, since those
fields are required `str` in `PinchBenchAgentConfig`.

**Listing metadata on the base node.** `domain`/`description`/`value`
are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml`
rather than the benchmark config, because adding them to the inheriting
node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth
knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so
`ng_list_benchmarks` renders such a key fine and only the real
prepare/run path trips the check. The `README.md` row is regenerated by
the `update-readme-table` hook.

**Drift guards.** Preparation fails loudly if the manifest is not 147
tasks, or if the regenerated rows do not byte-match the committed
`responses_api_agents/pinchbench/data/example.jsonl` — so upstream
prompt drift and divergence from `dataset_preprocess.py` cannot pass
silently.

## Verification

- `ng_prepare_benchmark
"+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via
both the live-clone path and `PINCHBENCH_SKILL_DIR`
- Rows validated: all carry a non-empty `responses_create_params.input`,
`task_id`s unique
- `ng_list_benchmarks` lists `pinchbench` with domain `agent` and
`num_repeats` 3
- Drift guard negative-tested (tampered prompt raises)
- `pre-commit run` clean on all touched files
- Generated JSONL is covered by the existing `benchmarks/.gitignore`

## Related

Cherry-picked to `r0.5.0` in a companion PR (that release branch carries
the agent; `r0.4.0` does not, as PinchBench landed on main after the
cut).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
OlegSudakov pushed a commit to OlegSudakov/Gym that referenced this pull request Aug 7, 2026
NVIDIA-NeMo#2259)

## Summary

PinchBench shipped as an agent server
(`responses_api_agents/pinchbench`, NVIDIA-NeMo#1810) but has no entry under
`benchmarks/`, so it is neither discoverable via `gym list benchmarks`
nor preparable via `gym eval prepare`. This adds the benchmark
definition.

- `config.yaml` chains `config_paths` to the agent config and narrows it
to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2`
shape, the existing precedent for a benchmark chaining to a
`responses_api_agents/` config.
- `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref
`Dockerfile.benchmark` bakes into the per-task image, and writes all 147
manifest tasks.
- `num_repeats: 3`, matching the n=3 parity validation in the agent
README (reward is continuous, and temperature 1.0 + live web search make
a single pass noisy).

## Notes for reviewers

**Run-time interpolations.** The agent config interpolates eight knobs
(`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not
resolve on their own, so `gym eval prepare` would fail on a bare config.
They are declared top-level with `oc.env:...,null` defaults. This does
not break the documented `+sandbox_image=...` invocation: Hydra parses
CLI args against an empty base *before* `config_paths` loads, and CLI
merges last. A `null` still reaching the agent fails fast, since those
fields are required `str` in `PinchBenchAgentConfig`.

**Listing metadata on the base node.** `domain`/`description`/`value`
are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml`
rather than the benchmark config, because adding them to the inheriting
node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth
knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so
`ng_list_benchmarks` renders such a key fine and only the real
prepare/run path trips the check. The `README.md` row is regenerated by
the `update-readme-table` hook.

**Drift guards.** Preparation fails loudly if the manifest is not 147
tasks, or if the regenerated rows do not byte-match the committed
`responses_api_agents/pinchbench/data/example.jsonl` — so upstream
prompt drift and divergence from `dataset_preprocess.py` cannot pass
silently.

## Verification

- `ng_prepare_benchmark
"+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via
both the live-clone path and `PINCHBENCH_SKILL_DIR`
- Rows validated: all carry a non-empty `responses_create_params.input`,
`task_id`s unique
- `ng_list_benchmarks` lists `pinchbench` with domain `agent` and
`num_repeats` 3
- Drift guard negative-tested (tampered prompt raises)
- `pre-commit run` clean on all touched files
- Generated JSONL is covered by the existing `benchmarks/.gitignore`

## Related

Cherry-picked to `r0.5.0` in a companion PR (that release branch carries
the agent; `r0.4.0` does not, as PinchBench landed on main after the
cut).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Issue reported or requested by someone from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants