Skip to content

PinchBench: route task failures through the dispatcher failure contract - #1984

Closed
laszkiewiczp wants to merge 5 commits into
awarno/pinchbench-efb-hsgfrom
plaszkiewicz/pinchbench-failure-routing-v2
Closed

laszkiewiczp wants to merge 5 commits into
awarno/pinchbench-efb-hsgfrom
plaszkiewicz/pinchbench-failure-routing-v2

Conversation

@laszkiewiczp

Copy link
Copy Markdown
Contributor

Supersedes #1983 (same fix, retargeted at the PinchBench branch so only the fix is in the diff). Stacks on awarno/pinchbench-efb-hsg = #1810 + the HSG direct-exec commit.

Problem

The PinchBench agent catches every task failure and returns it as a normal scored row (reward: 0, grading_notes: "run failed: ...") in the main rollouts jsonl. This breaks resume-after-walltime: tasks in flight when Slurm kills the job get permanently cached as reward-0 rows, and _load_from_cache counts them as done — they are never retried. On a cluster with a hard walltime cap this silently records a chunk of the dataset as zero-reward failures whenever a run needs more than one walltime window.

Fix

Adopt the failure-routing contract the dispatcher (nemo_gym/rollout_collection.py) already implements and stirrup_agent already uses:

  • sandbox execs that die by signal (rc<0 / 137 / 143 — walltime SIGTERM, preemption, OOM kill) raise SandboxKilledError → kill_shaped: no row anywhere, resume re-dispatches naturally
  • per-task timeouts → timeout_exceeded: one sidecar row, terminal
  • everything else → legitimate: sidecar row, retried up to NEMO_GYM_MAX_ROLLOUT_ATTEMPTS on resume

The main jsonl now only ever contains scored rollouts. Tests cover the pre-fix regression (sentinel-free failure rows) and the routing classes; placeholder fixture keys carry detect-secrets allowlist pragmas.

Validation

Full PinchBench run (1470 rollouts, 10 repeats) deliberately crossing a Slurm walltime restart at this change:

  • at the kill: 1282 clean rows in main, all 176 in-flight tasks routed to the failures sidecar (previously they became poisoned reward-0 rows in main)
  • resume log: 1282 rows already done / 176 prior failure attempts / 0 terminal / 188 rows that still need to be run
  • final: 1470/1470 scored, zero run failed rows in main, mean reward consistent with single-window reference runs

🤖 Generated with Claude Code

Failed tasks were returned as reward-0 rows in the main jsonl, so
resume-after-walltime counted in-flight kills as done and never retried
them. Classify failures like stirrup_agent: signal-killed sandbox execs
are kill_shaped (not persisted, re-dispatched on resume), per-task
timeouts are terminal sidecar rows, everything else goes to the sidecar
with bounded retry.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
The sentinel-free-row regression test fails on the pre-fix agent (a
failed task produced a row the dispatcher writes to the main jsonl);
the rest pin kill_shaped/timeout/legitimate routing and that scored
rollouts stay sentinel-free. Placeholder keys in the existing test
fixture carry detect-secrets allowlist pragmas.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Drop the inline allowlist pragmas (test fixture lines stay as authored)
and register the placeholder keys in the detect-secrets baseline instead;
compress the added comments and remove an unused test helper.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
@laszkiewiczp
laszkiewiczp requested a review from a team as a code owner July 10, 2026 07:53
@laszkiewiczp

Copy link
Copy Markdown
Contributor Author

CI note: the Server suite shard-6 failure (tests/test_app.py::TestApp::test_sanity_query_input, tau2 response schema) is inherited from the branch base — the base predates #1954 which repointed the tau2-bench pins to stable branches. Unrelated to this diff (pinchbench files + secrets baseline + README only); it clears whenever the pinchbench stack rebases onto current main. All checks specific to this change pass: DCO, secrets-detector, Lint, Test.

The tau2-bench pin tracks a branch that moved schema (nl_assertions),
so the recorded request/response fixture on this base no longer matches
what the server produces; take the updated fixture from main's fix
commit f245781 without its runtime changes, which do not apply to this
base's tau2 layout.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
@laszkiewiczp

Copy link
Copy Markdown
Contributor Author

Follow-up: the shard-6 failure is now fixed in this branch rather than deferred — the tau2 recorded fixture from #1954 applied cleanly on its own (2ef18a53), while that commit's runtime changes don't apply to this base's tau2 layout and were skipped. Full CI is green.

@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 17, 2026
@AWarno
AWarno self-requested a review July 20, 2026 13:53
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants