Skip to content

PinchBench: route task failures through the dispatcher failure contract - #1983

Closed
laszkiewiczp wants to merge 11 commits into
mainfrom
plaszkiewicz/pinchbench-failure-routing
Closed

laszkiewiczp wants to merge 11 commits into
mainfrom
plaszkiewicz/pinchbench-failure-routing

Conversation

@laszkiewiczp

Copy link
Copy Markdown
Contributor

Problem

The PinchBench agent catches every task failure and returns it as a normal scored row (reward: 0, grading_notes: "run failed: ...") in the main rollouts jsonl. This breaks resume-after-walltime: tasks that were in flight when Slurm kills the job get permanently cached as reward-0 rows, and _load_from_cache counts them as done — they are never retried. On a cluster with a hard walltime cap this silently records a chunk of the dataset as zero-reward failures whenever a run needs more than one walltime window.

Fix

Adopt the failure-routing contract the dispatcher (nemo_gym/rollout_collection.py) already implements and stirrup_agent already uses:

  • sandbox execs that die by signal (rc<0 / 137 / 143 — walltime SIGTERM, preemption, OOM kill) raise SandboxKilledError → kill_shaped: no row anywhere, resume re-dispatches naturally
  • per-task timeouts → timeout_exceeded: one sidecar row, terminal
  • everything else → legitimate: sidecar row, retried up to NEMO_GYM_MAX_ROLLOUT_ATTEMPTS on resume

The main jsonl now only ever contains scored rollouts.

Validation

Full PinchBench run (1470 rollouts, 10 repeats) deliberately crossing a Slurm walltime restart at this commit:

  • at the kill: 1282 clean rows in main, all 176 in-flight tasks routed to the failures sidecar (previously they became poisoned reward-0 rows in main)
  • resume log: 1282 rows already done / 176 prior failure attempts / 0 terminal / 188 rows that still need to be run
  • final: 1470/1470 scored, zero run failed rows in main, mean reward consistent with single-window reference runs

Known refinement for a follow-up: at SIGTERM the apptainer exit code is not always signal-shaped, so kill-window failures can classify as legitimate instead of kill_shaped — functionally identical (sidecar + retry) but burns one of the retry attempts per crossing.

🤖 Generated with Claude Code

Haran Kumar Shiv Kumar and others added 5 commits June 29, 2026 08:14
PinchBench (https://github.com/pinchbench/skill) measures an LLM as the brain of an
OpenClaw agent across 147 real-world tasks (calendar, email, CSV/log analysis, coding,
research, writing), graded by deterministic checks, an LLM judge, or both.

Integrated at the agent-server level (like swe_agents / harbor_agent): OpenClaw plus
PinchBench's own rollout/grading harness is the orchestration, wrapped rather than
reimplemented. One sandbox per task (via nemo_gym.sandbox; apptainer / opensandbox) is
the isolation boundary -- each task gets its own ~/.openclaw + OpenClaw gateway, avoiding
the shared-workspace WorkspaceVanishedError cliff. The skill is not vendored: the image
clones skill@v2.0.0 at build and applies a small NVIDIA endpoint-integration patch
(scripts/lib_agent.py: custom OpenAI-compatible provider, JUDGE_BASE_URL/JUDGE_API_KEY
routing, brave/tavily web-search config, transcript-path resolution).

Validated vs vanilla standalone PinchBench on Nemotron-3-Nano-30B-A3B (n=3, max_tokens
65536, temperature 1.0): Gym 0.583 +/- 0.025 vs vanilla 0.564 +/- 0.013 -> parity.

Co-Authored-By: Anna Warno <awarno@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Address review on data/example.jsonl:
- Add data/example_rollouts.jsonl: 5 example rollouts (the 5 example tasks run
  through the agent — sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89).
- Embed each task's real prompt (the skill .md `## Prompt` section) in `input`
  instead of a placeholder; dataset_preprocess.py now extracts it.
- Clarify that `verifier_metadata.task_id` is the authoritative per-row selector:
  run_task.sh runs `benchmark.py --suite <task_id>`, and a subset is run by
  including only those rows. README + docstring updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Fix a concurrency bug: run_task.sh started the gateway on the fixed default port
(18789). Under a provider that shares the host network namespace (apptainer),
concurrent sandboxes then collide -- only the first gateway binds, the rest fail
with EADDRINUSE -> no agent -> empty transcript -> reward 0. Since the agent's
default max_concurrent is 4, the default config was broken under apptainer.

Give each sandbox a unique gateway port (`openclaw config set gateway.port` +
`--port`); the in-sandbox `openclaw agent` client reads gateway.port from the
per-sandbox $HOME/.openclaw, so gateway and client agree. Harmless for opensandbox
(each container has its own netns, so the port was never shared). Verified with 3
concurrent models (1 sandbox each): all produce real scores, zero EADDRINUSE.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Failed tasks were returned as reward-0 rows in the main jsonl, so
resume-after-walltime counted in-flight kills as done and never retried
them. Classify failures like stirrup_agent: signal-killed sandbox execs
are kill_shaped (not persisted, re-dispatched on resume), per-task
timeouts are terminal sidecar rows, everything else goes to the sidecar
with bounded retry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

laszkiewiczp and others added 6 commits July 10, 2026 09:06
The sentinel-free-row regression test fails on the pre-fix agent (a
failed task produced a row the dispatcher writes to the main jsonl);
the rest pin kill_shaped/timeout/legitimate routing and that scored
rollouts stay sentinel-free.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
detect-secrets flags the pre-existing fake keys at their new line
numbers; mark them with inline allowlist pragmas.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
On behalf of Anna Warno <awarno@nvidia.com>, I, plaszkiewicz <plaszkiewicz@nvidia.com>, hereby add my Signed-off-by to this commit: 319da4c

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
@laszkiewiczp

Copy link
Copy Markdown
Contributor Author

Superseded by #1984, which targets the PinchBench branch (awarno/pinchbench-efb-hsg) directly so the diff contains only the failure-routing fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants