Repository navigation
Add PinchBench agentic benchmark integration - #1810
Merged
Merged
Conversation
cmunley1
reviewed
Jun 29, 2026
| @@ -0,0 +1,5 @@ | |||
| {"responses_create_params":{"input":[{"role":"user","content":"<placeholder; real prompt loaded from the task .md at run time>"}]},"verifier_metadata":{"task_id":"task_sanity"}} | |||
Contributor
There was a problem hiding this comment.
could u upload 5 example rollouts? e.g. https://docs.nvidia.com/nemo/gym/main/contribute/environments/new-environment#4-generate-example-rollouts
Also, is there a way to include the real prompts here rather than placeholders? At the least, a task index per row that is used to load each task? I think we need some control over which tasks we run, e.g. if we wanted to run a subset or something
Contributor
Author
There was a problem hiding this comment.
HI,
- Added data/example_rollouts.jsonl — 5 example rollouts (the 5 example tasks through the agent: sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89).
- Embedded each task's real prompt (the skill .md ## Prompt section) in input instead of the placeholder in example rollouts.
- On task control: each row carries verifier_metadata.task_id, which is the selector — run_task.sh runs benchmark.py --suite <task_id>, and you run a subset by including only the rows you want. The full task (prompt + assets + grading) loads from the skill by task_id, so the JSONL stays small.
7 tasks
This was referenced Jul 10, 2026
hkumar92
force-pushed
the
hshivkumar/add-pinchbench
branch
from
July 14, 2026 12:05
f7063e1 to
8ae8f4b
Compare
PinchBench (https://github.com/pinchbench/skill) measures an LLM as the brain of an OpenClaw agent across 147 real-world tasks (calendar, email, CSV/log analysis, coding, research, writing), graded by deterministic checks, an LLM judge, or both. Integrated at the agent-server level (like swe_agents / harbor_agent): OpenClaw plus PinchBench's own rollout/grading harness is the orchestration, wrapped rather than reimplemented. One sandbox per task (via nemo_gym.sandbox; apptainer / opensandbox) is the isolation boundary -- each task gets its own ~/.openclaw + OpenClaw gateway, avoiding the shared-workspace WorkspaceVanishedError cliff. The skill is not vendored: the image clones skill@v2.0.0 at build and applies a small NVIDIA endpoint-integration patch (scripts/lib_agent.py: custom OpenAI-compatible provider, JUDGE_BASE_URL/JUDGE_API_KEY routing, brave/tavily web-search config, transcript-path resolution). Validated vs vanilla standalone PinchBench on Nemotron-3-Nano-30B-A3B (n=3, max_tokens 65536, temperature 1.0): Gym 0.583 +/- 0.025 vs vanilla 0.564 +/- 0.013 -> parity. Co-Authored-By: Anna Warno <awarno@nvidia.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Address review on data/example.jsonl: - Add data/example_rollouts.jsonl: 5 example rollouts (the 5 example tasks run through the agent — sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89). - Embed each task's real prompt (the skill .md `## Prompt` section) in `input` instead of a placeholder; dataset_preprocess.py now extracts it. - Clarify that `verifier_metadata.task_id` is the authoritative per-row selector: run_task.sh runs `benchmark.py --suite <task_id>`, and a subset is run by including only those rows. README + docstring updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Fix a concurrency bug: run_task.sh started the gateway on the fixed default port (18789). Under a provider that shares the host network namespace (apptainer), concurrent sandboxes then collide -- only the first gateway binds, the rest fail with EADDRINUSE -> no agent -> empty transcript -> reward 0. Since the agent's default max_concurrent is 4, the default config was broken under apptainer. Give each sandbox a unique gateway port (`openclaw config set gateway.port` + `--port`); the in-sandbox `openclaw agent` client reads gateway.port from the per-sandbox $HOME/.openclaw, so gateway and client agree. Harmless for opensandbox (each container has its own netns, so the port was never shared). Verified with 3 concurrent models (1 sandbox each): all produce real scores, zero EADDRINUSE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Failed tasks were returned as reward-0 rows in the main jsonl, so resume-after-walltime counted in-flight kills as done and never retried them. Classify failures like stirrup_agent: signal-killed sandbox execs are kill_shaped (not persisted, re-dispatched on resume), per-task timeouts are terminal sidecar rows, everything else goes to the sidecar with bounded retry. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
The sentinel-free-row regression test fails on the pre-fix agent (a failed task produced a row the dispatcher writes to the main jsonl); the rest pin kill_shaped/timeout/legitimate routing and that scored rollouts stay sentinel-free. Placeholder keys in the existing test fixture carry detect-secrets allowlist pragmas. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Drop the inline allowlist pragmas (test fixture lines stay as authored) and register the placeholder keys in the detect-secrets baseline instead; compress the added comments and remove an unused test helper. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
hkumar92
force-pushed
the
hshivkumar/add-pinchbench
branch
from
July 14, 2026 17:33
a2a7d31 to
76583bb
Compare
laszkiewiczp
added a commit
that referenced
this pull request
Aug 3, 2026
#2259) ## Summary PinchBench shipped as an agent server (`responses_api_agents/pinchbench`, #1810) but has no entry under `benchmarks/`, so it is neither discoverable via `gym list benchmarks` nor preparable via `gym eval prepare`. This adds the benchmark definition. - `config.yaml` chains `config_paths` to the agent config and narrows it to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2` shape, the existing precedent for a benchmark chaining to a `responses_api_agents/` config. - `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref `Dockerfile.benchmark` bakes into the per-task image, and writes all 147 manifest tasks. - `num_repeats: 3`, matching the n=3 parity validation in the agent README (reward is continuous, and temperature 1.0 + live web search make a single pass noisy). ## Notes for reviewers **Run-time interpolations.** The agent config interpolates eight knobs (`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not resolve on their own, so `gym eval prepare` would fail on a bare config. They are declared top-level with `oc.env:...,null` defaults. This does not break the documented `+sandbox_image=...` invocation: Hydra parses CLI args against an empty base *before* `config_paths` loads, and CLI merges last. A `null` still reaching the agent fails fast, since those fields are required `str` in `PinchBenchAgentConfig`. **Listing metadata on the base node.** `domain`/`description`/`value` are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml` rather than the benchmark config, because adding them to the inheriting node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so `ng_list_benchmarks` renders such a key fine and only the real prepare/run path trips the check. The `README.md` row is regenerated by the `update-readme-table` hook. **Drift guards.** Preparation fails loudly if the manifest is not 147 tasks, or if the regenerated rows do not byte-match the committed `responses_api_agents/pinchbench/data/example.jsonl` — so upstream prompt drift and divergence from `dataset_preprocess.py` cannot pass silently. ## Verification - `ng_prepare_benchmark "+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via both the live-clone path and `PINCHBENCH_SKILL_DIR` - Rows validated: all carry a non-empty `responses_create_params.input`, `task_id`s unique - `ng_list_benchmarks` lists `pinchbench` with domain `agent` and `num_repeats` 3 - Drift guard negative-tested (tampered prompt raises) - `pre-commit run` clean on all touched files - Generated JSONL is covered by the existing `benchmarks/.gitignore` ## Related Cherry-picked to `r0.5.0` in a companion PR (that release branch carries the agent; `r0.4.0` does not, as PinchBench landed on main after the cut). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
svcnvidia-nemo-ci
pushed a commit
that referenced
this pull request
Aug 3, 2026
#2259) ## Summary PinchBench shipped as an agent server (`responses_api_agents/pinchbench`, #1810) but has no entry under `benchmarks/`, so it is neither discoverable via `gym list benchmarks` nor preparable via `gym eval prepare`. This adds the benchmark definition. - `config.yaml` chains `config_paths` to the agent config and narrows it to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2` shape, the existing precedent for a benchmark chaining to a `responses_api_agents/` config. - `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref `Dockerfile.benchmark` bakes into the per-task image, and writes all 147 manifest tasks. - `num_repeats: 3`, matching the n=3 parity validation in the agent README (reward is continuous, and temperature 1.0 + live web search make a single pass noisy). ## Notes for reviewers **Run-time interpolations.** The agent config interpolates eight knobs (`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not resolve on their own, so `gym eval prepare` would fail on a bare config. They are declared top-level with `oc.env:...,null` defaults. This does not break the documented `+sandbox_image=...` invocation: Hydra parses CLI args against an empty base *before* `config_paths` loads, and CLI merges last. A `null` still reaching the agent fails fast, since those fields are required `str` in `PinchBenchAgentConfig`. **Listing metadata on the base node.** `domain`/`description`/`value` are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml` rather than the benchmark config, because adding them to the inheriting node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so `ng_list_benchmarks` renders such a key fine and only the real prepare/run path trips the check. The `README.md` row is regenerated by the `update-readme-table` hook. **Drift guards.** Preparation fails loudly if the manifest is not 147 tasks, or if the regenerated rows do not byte-match the committed `responses_api_agents/pinchbench/data/example.jsonl` — so upstream prompt drift and divergence from `dataset_preprocess.py` cannot pass silently. ## Verification - `ng_prepare_benchmark "+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via both the live-clone path and `PINCHBENCH_SKILL_DIR` - Rows validated: all carry a non-empty `responses_create_params.input`, `task_id`s unique - `ng_list_benchmarks` lists `pinchbench` with domain `agent` and `num_repeats` 3 - Drift guard negative-tested (tampered prompt raises) - `pre-commit run` clean on all touched files - Generated JSONL is covered by the existing `benchmarks/.gitignore` ## Related Cherry-picked to `r0.5.0` in a companion PR (that release branch carries the agent; `r0.4.0` does not, as PinchBench landed on main after the cut). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Aug 7, 2026
NVIDIA-NeMo#2259) ## Summary PinchBench shipped as an agent server (`responses_api_agents/pinchbench`, NVIDIA-NeMo#1810) but has no entry under `benchmarks/`, so it is neither discoverable via `gym list benchmarks` nor preparable via `gym eval prepare`. This adds the benchmark definition. - `config.yaml` chains `config_paths` to the agent config and narrows it to the benchmark dataset via `_inherit_from` — the `benchmarks/tau2` shape, the existing precedent for a benchmark chaining to a `responses_api_agents/` config. - `prepare.py` clones the PinchBench skill at `v2.0.0`, the same ref `Dockerfile.benchmark` bakes into the per-task image, and writes all 147 manifest tasks. - `num_repeats: 3`, matching the n=3 parity validation in the agent README (reward is continuous, and temperature 1.0 + live web search make a single pass noisy). ## Notes for reviewers **Run-time interpolations.** The agent config interpolates eight knobs (`${sandbox_image}`, `${model_base_url}`, judge/brave keys) that do not resolve on their own, so `gym eval prepare` would fail on a bare config. They are declared top-level with `oc.env:...,null` defaults. This does not break the documented `+sandbox_image=...` invocation: Hydra parses CLI args against an empty base *before* `config_paths` loads, and CLI merges last. A `null` still reaching the agent fails fast, since those fields are required `str` in `PinchBenchAgentConfig`. **Listing metadata on the base node.** `domain`/`description`/`value` are added to `responses_api_agents/pinchbench/configs/pinchbench.yaml` rather than the benchmark config, because adding them to the inheriting node raises `ConfigKeyError: Key 'domain' is not in struct`. Worth knowing: `discover_benchmarks()` uses the tolerant non-struct parse, so `ng_list_benchmarks` renders such a key fine and only the real prepare/run path trips the check. The `README.md` row is regenerated by the `update-readme-table` hook. **Drift guards.** Preparation fails loudly if the manifest is not 147 tasks, or if the regenerated rows do not byte-match the committed `responses_api_agents/pinchbench/data/example.jsonl` — so upstream prompt drift and divergence from `dataset_preprocess.py` cannot pass silently. ## Verification - `ng_prepare_benchmark "+config_paths=[benchmarks/pinchbench/config.yaml]"` — 147 rows, via both the live-clone path and `PINCHBENCH_SKILL_DIR` - Rows validated: all carry a non-empty `responses_create_params.input`, `task_id`s unique - `ng_list_benchmarks` lists `pinchbench` with domain `agent` and `num_repeats` 3 - Drift guard negative-tested (tampered prompt raises) - `pre-commit run` clean on all touched files - Generated JSONL is covered by the existing `benchmarks/.gitignore` ## Related Cherry-picked to `r0.5.0` in a companion PR (that release branch carries the agent; `r0.4.0` does not, as PinchBench landed on main after the cut). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds PinchBench — an LLM-as-OpenClaw-agent benchmark over 147 real-world tasks (calendar, email triage, CSV/log analysis, coding, research, writing), graded by deterministic checks, an LLM judge, or both — as an agent-server integration under
responses_api_agents/pinchbench/.How
swe_agents/harbor_agent): OpenClaw + PinchBench's own rollout/grading harness is the orchestration, so we wrap it rather than reimplement it.nemo_gym.sandboxAPI (apptainer / opensandbox) — each task gets its own~/.openclaw+ OpenClaw gateway, which is cliff-proof (a wedged task can't affect others; a gateway shared across all 147 tasks hitsWorkspaceVanishedError).Dockerfile.benchmarkclonesskill@v2.0.0at build and applies a small NVIDIA endpoint-integration patch (scripts/lib_agent.pyonly: custom OpenAI-compatible provider,JUDGE_BASE_URL/JUDGE_API_KEYrouting, brave/tavily web-search config, transcript-path resolution).reward+grading_breakdown+grading_notes+ the full OpenClaw transcript inraw_rollout.Validation
Vs vanilla standalone PinchBench on
Nemotron-3-Nano-30B-A3B(n=3,max_tokens=65536,temperature=1.0): Gym 0.583 ± 0.025 vs vanilla 0.564 ± 0.013 → parity. The apptainer-sandbox path was reproduced on a 10-task subset (~0.66); a full n=3 sandbox run is in progress.Notes
2026.6.5. Evaluation-only (OpenClaw exposes no token IDs/logprobs).🤖 Generated with Claude Code