Repository navigation
PinchBench: route task failures through the dispatcher failure contract - #1984
Closed
laszkiewiczp wants to merge 5 commits into
Closed
laszkiewiczp wants to merge 5 commits into
laszkiewiczp wants to merge 5 commits into
Conversation
Failed tasks were returned as reward-0 rows in the main jsonl, so resume-after-walltime counted in-flight kills as done and never retried them. Classify failures like stirrup_agent: signal-killed sandbox execs are kill_shaped (not persisted, re-dispatched on resume), per-task timeouts are terminal sidecar rows, everything else goes to the sidecar with bounded retry. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
The sentinel-free-row regression test fails on the pre-fix agent (a failed task produced a row the dispatcher writes to the main jsonl); the rest pin kill_shaped/timeout/legitimate routing and that scored rollouts stay sentinel-free. Placeholder keys in the existing test fixture carry detect-secrets allowlist pragmas. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Drop the inline allowlist pragmas (test fixture lines stay as authored) and register the placeholder keys in the detect-secrets baseline instead; compress the added comments and remove an unused test helper. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Contributor
Author
|
CI note: the Server suite shard-6 failure ( |
The tau2-bench pin tracks a branch that moved schema (nl_assertions), so the recorded request/response fixture on this base no longer matches what the server produces; take the updated fixture from main's fix commit f245781 without its runtime changes, which do not apply to this base's tau2 layout. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Contributor
Author
|
Follow-up: the shard-6 failure is now fixed in this branch rather than deferred — the tau2 recorded fixture from #1954 applied cleanly on its own ( |
AWarno
self-requested a review
July 20, 2026 13:53
AWarno
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1983 (same fix, retargeted at the PinchBench branch so only the fix is in the diff). Stacks on
awarno/pinchbench-efb-hsg= #1810 + the HSG direct-exec commit.Problem
The PinchBench agent catches every task failure and returns it as a normal scored row (
reward: 0,grading_notes: "run failed: ...") in the main rollouts jsonl. This breaks resume-after-walltime: tasks in flight when Slurm kills the job get permanently cached as reward-0 rows, and_load_from_cachecounts them as done — they are never retried. On a cluster with a hard walltime cap this silently records a chunk of the dataset as zero-reward failures whenever a run needs more than one walltime window.Fix
Adopt the failure-routing contract the dispatcher (
nemo_gym/rollout_collection.py) already implements andstirrup_agentalready uses:SandboxKilledError→kill_shaped: no row anywhere, resume re-dispatches naturallytimeout_exceeded: one sidecar row, terminallegitimate: sidecar row, retried up toNEMO_GYM_MAX_ROLLOUT_ATTEMPTSon resumeThe main jsonl now only ever contains scored rollouts. Tests cover the pre-fix regression (sentinel-free failure rows) and the routing classes; placeholder fixture keys carry detect-secrets allowlist pragmas.
Validation
Full PinchBench run (1470 rollouts, 10 repeats) deliberately crossing a Slurm walltime restart at this change:
1282 rows already done / 176 prior failure attempts / 0 terminal / 188 rows that still need to be runrun failedrows in main, mean reward consistent with single-window reference runs🤖 Generated with Claude Code