Repository navigation
PinchBench: route task failures through the dispatcher failure contract - #1983
Closed
laszkiewiczp wants to merge 11 commits into
Closed
laszkiewiczp wants to merge 11 commits into
laszkiewiczp wants to merge 11 commits into
Conversation
PinchBench (https://github.com/pinchbench/skill) measures an LLM as the brain of an OpenClaw agent across 147 real-world tasks (calendar, email, CSV/log analysis, coding, research, writing), graded by deterministic checks, an LLM judge, or both. Integrated at the agent-server level (like swe_agents / harbor_agent): OpenClaw plus PinchBench's own rollout/grading harness is the orchestration, wrapped rather than reimplemented. One sandbox per task (via nemo_gym.sandbox; apptainer / opensandbox) is the isolation boundary -- each task gets its own ~/.openclaw + OpenClaw gateway, avoiding the shared-workspace WorkspaceVanishedError cliff. The skill is not vendored: the image clones skill@v2.0.0 at build and applies a small NVIDIA endpoint-integration patch (scripts/lib_agent.py: custom OpenAI-compatible provider, JUDGE_BASE_URL/JUDGE_API_KEY routing, brave/tavily web-search config, transcript-path resolution). Validated vs vanilla standalone PinchBench on Nemotron-3-Nano-30B-A3B (n=3, max_tokens 65536, temperature 1.0): Gym 0.583 +/- 0.025 vs vanilla 0.564 +/- 0.013 -> parity. Co-Authored-By: Anna Warno <awarno@nvidia.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Address review on data/example.jsonl: - Add data/example_rollouts.jsonl: 5 example rollouts (the 5 example tasks run through the agent — sanity/calendar/todo 1.0, daily_summary 0.9, email 0.89). - Embed each task's real prompt (the skill .md `## Prompt` section) in `input` instead of a placeholder; dataset_preprocess.py now extracts it. - Clarify that `verifier_metadata.task_id` is the authoritative per-row selector: run_task.sh runs `benchmark.py --suite <task_id>`, and a subset is run by including only those rows. README + docstring updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Fix a concurrency bug: run_task.sh started the gateway on the fixed default port (18789). Under a provider that shares the host network namespace (apptainer), concurrent sandboxes then collide -- only the first gateway binds, the rest fail with EADDRINUSE -> no agent -> empty transcript -> reward 0. Since the agent's default max_concurrent is 4, the default config was broken under apptainer. Give each sandbox a unique gateway port (`openclaw config set gateway.port` + `--port`); the in-sandbox `openclaw agent` client reads gateway.port from the per-sandbox $HOME/.openclaw, so gateway and client agree. Harmless for opensandbox (each container has its own netns, so the port was never shared). Verified with 3 concurrent models (1 sandbox each): all produce real scores, zero EADDRINUSE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
Failed tasks were returned as reward-0 rows in the main jsonl, so resume-after-walltime counted in-flight kills as done and never retried them. Classify failures like stirrup_agent: signal-killed sandbox execs are kill_shaped (not persisted, re-dispatched on resume), per-task timeouts are terminal sidecar rows, everything else goes to the sidecar with bounded retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
The sentinel-free-row regression test fails on the pre-fix agent (a failed task produced a row the dispatcher writes to the main jsonl); the rest pin kill_shaped/timeout/legitimate routing and that scored rollouts stay sentinel-free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
detect-secrets flags the pre-existing fake keys at their new line numbers; mark them with inline allowlist pragmas. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
…ch-failure-routing
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
On behalf of Anna Warno <awarno@nvidia.com>, I, plaszkiewicz <plaszkiewicz@nvidia.com>, hereby add my Signed-off-by to this commit: 319da4c Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Contributor
Author
|
Superseded by #1984, which targets the PinchBench branch (awarno/pinchbench-efb-hsg) directly so the diff contains only the failure-routing fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The PinchBench agent catches every task failure and returns it as a normal scored row (
reward: 0,grading_notes: "run failed: ...") in the main rollouts jsonl. This breaks resume-after-walltime: tasks that were in flight when Slurm kills the job get permanently cached as reward-0 rows, and_load_from_cachecounts them as done — they are never retried. On a cluster with a hard walltime cap this silently records a chunk of the dataset as zero-reward failures whenever a run needs more than one walltime window.Fix
Adopt the failure-routing contract the dispatcher (
nemo_gym/rollout_collection.py) already implements andstirrup_agentalready uses:SandboxKilledError→kill_shaped: no row anywhere, resume re-dispatches naturallytimeout_exceeded: one sidecar row, terminallegitimate: sidecar row, retried up toNEMO_GYM_MAX_ROLLOUT_ATTEMPTSon resumeThe main jsonl now only ever contains scored rollouts.
Validation
Full PinchBench run (1470 rollouts, 10 repeats) deliberately crossing a Slurm walltime restart at this commit:
1282 rows already done / 176 prior failure attempts / 0 terminal / 188 rows that still need to be runrun failedrows in main, mean reward consistent with single-window reference runsKnown refinement for a follow-up: at SIGTERM the apptainer exit code is not always signal-shaped, so kill-window failures can classify as
legitimateinstead ofkill_shaped— functionally identical (sidecar + retry) but burns one of the retry attempts per crossing.🤖 Generated with Claude Code