You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
harbor: a verifier that exits non-zero is recorded as a measured result #5642
Found while running SkillEvaluator's Tier 3 task folders on Gym. Gym cc8991e4 (main, 2026-11-04), Docker sandbox.
Summary. Gym ignores test.sh's exit code. A verifier that writes a reward and exits non-zero yields a measured, unmasked result. SkillEvaluator's verifier does this when a required LLM judge fails: it writes overall: 0.0, exits 1 and treats the trial as unscored. On Gym, judge outages therefore enter the scores as real results.
Observed.rollouts_run_summary.json: "status": "result", "mask_sample": false, "failure": null, verifier.return_code: 1, "reward": 0.0. Seen on a real SkillEvaluator task too: a judge timeout gave evaluation_status: failed in SkillEvaluator's sidecar and a measured Gym result.
Expected. An opt-in way to mask (or fail) a trial whose verifier exits non-zero, so an unmeasured trial is not scored.
Proposed fix. Opt-in verifier_exit_nonzero = "mask" in dataset.toml ([gym.defaults] / per task): in _score (environment_servers/harbor/verifier.py), after the timeout branch, mask with failure kind harbor:verifier_exit_nonzero and keep the parsed rewards as metrics. Terminal-Bench's default (score as measured) is unchanged. Related: #4982.
I'd like to pick this up after #5641, but there's one design point I'd rather settle before opening anything.
The proposal masks the sample on any non-zero test.sh exit (verifier_exit_nonzero = "mask" in [gym.defaults], exit_nonzero per task under [gym.tasks."<id>".verifier]). In shared mode test.sh runs where the agent worked, so an agent that breaks the grader (deletes a tool it calls, fills the disk) would turn what should be a 0 into a masked sample. That's a reward hack, and under masking it's invisible in the aggregate.
Two open PRs here draw the line narrower. #5675 (terminal_bench_2_1) leaves a row unscored only for verifier infrastructure errors, while a test.sh timeout and an absent or invalid reward.txt still score 0. #5655 (swe_rebench) treats exit 124 or 137 as a kill only when the output stops before the test output end marker, so a test command that itself exits 124 or 137 is still graded. Both treat "the evaluation didn't really run" as not a score, but only on a signal the agent can't easily produce.
So the question: would you rather
any non-zero exit masks, as proposed (simplest, matches the issue), or
the verifier signals "not measured" explicitly with a reserved exit code, say 75 (EX_TEMPFAIL), so SkillEvaluator's judge failure maps to exit 75 and every other non-zero exit stays a measured result?
With option 2 the setting could still default to off, and a dataset whose verifier can't be tampered with (separate mode) could opt into option 1. Either way, a timeout that left no reward would keep going through the timeout branch and score 0, and whatever the verifier wrote stays in verifier_rewards.
I have option 1 working locally with tests and can switch to option 2 quickly. Happy to build whichever you prefer.
Covered by main. Since #4049, a Harbor verifier that exits 75 without writing a reward is masked (harbor:verifier_unavailable), for separate-mode verifiers only; in shared mode the exit code is deliberately not trusted, because test.sh runs in the agent's sandbox. SkillEvaluator's verifier (shared mode; writes overall: 0.0, then exits 1) is therefore still scored on Gym; SkillEvaluator's own collector already drops those trials. If Gym's own numbers on these tasks need to exclude judge failures, the path is a separate verifier that exits 75 without a reward. #5799 is closed in favor of #4049.
Activity
zuraiz-anjum commentedon Nov 7, 2026
I'd like to pick this up after #5641, but there's one design point I'd rather settle before opening anything.
The proposal masks the sample on any non-zero
test.shexit (verifier_exit_nonzero = "mask"in[gym.defaults],exit_nonzeroper task under[gym.tasks."<id>".verifier]). In shared modetest.shruns where the agent worked, so an agent that breaks the grader (deletes a tool it calls, fills the disk) would turn what should be a 0 into a masked sample. That's a reward hack, and under masking it's invisible in the aggregate.Two open PRs here draw the line narrower. #5675 (terminal_bench_2_1) leaves a row unscored only for verifier infrastructure errors, while a
test.shtimeout and an absent or invalidreward.txtstill score 0. #5655 (swe_rebench) treats exit 124 or 137 as a kill only when the output stops before the test output end marker, so a test command that itself exits 124 or 137 is still graded. Both treat "the evaluation didn't really run" as not a score, but only on a signal the agent can't easily produce.So the question: would you rather
exit 75and every other non-zero exit stays a measured result?With option 2 the setting could still default to off, and a dataset whose verifier can't be tampered with (separate mode) could opt into option 1. Either way, a timeout that left no reward would keep going through the timeout branch and score 0, and whatever the verifier wrote stays in
verifier_rewards.I have option 1 working locally with tests and can switch to option 2 quickly. Happy to build whichever you prefer.
feat(harbor): let a dataset mask a trial whose verifier exits non-zero
linear-code commentedon Nov 18, 2026
from adasif: