Skip to content

harbor: a verifier that exits non-zero is recorded as a measured result #5642

Closed

Description

@qiaochuz-nv

Found while running SkillEvaluator's Tier 3 task folders on Gym. Gym cc8991e4 (main, 2026-11-04), Docker sandbox.

Summary. Gym ignores test.sh's exit code. A verifier that writes a reward and exits non-zero yields a measured, unmasked result. SkillEvaluator's verifier does this when a required LLM judge fails: it writes overall: 0.0, exits 1 and treats the trial as unscored. On Gym, judge outages therefore enter the scores as real results.

Repro (no model):

gym dataset init local/repro --root .
cat > repro/hello/tests/test.sh <<'EOF'
#!/bin/bash
mkdir -p /logs/verifier
echo '{"reward": 0.0}' > /logs/verifier/reward.json
echo "ERROR: required LLM judging failed" >&2
exit 1
EOF
gym dataset validate repro --sandbox docker

Observed. rollouts_run_summary.json: "status": "result", "mask_sample": false, "failure": null, verifier.return_code: 1, "reward": 0.0. Seen on a real SkillEvaluator task too: a judge timeout gave evaluation_status: failed in SkillEvaluator's sidecar and a measured Gym result.

Expected. An opt-in way to mask (or fail) a trial whose verifier exits non-zero, so an unmeasured trial is not scored.

Proposed fix. Opt-in verifier_exit_nonzero = "mask" in dataset.toml ([gym.defaults] / per task): in _score (environment_servers/harbor/verifier.py), after the timeout branch, mask with failure kind harbor:verifier_exit_nonzero and keep the parsed rewards as metrics. Terminal-Bench's default (score as measured) is unchanged. Related: #4982.

Activity

  1. zuraiz-anjum commented on Nov 7, 2026

    @zuraiz-anjum

    I'd like to pick this up after #5641, but there's one design point I'd rather settle before opening anything.

    The proposal masks the sample on any non-zero test.sh exit (verifier_exit_nonzero = "mask" in [gym.defaults], exit_nonzero per task under [gym.tasks."<id>".verifier]). In shared mode test.sh runs where the agent worked, so an agent that breaks the grader (deletes a tool it calls, fills the disk) would turn what should be a 0 into a masked sample. That's a reward hack, and under masking it's invisible in the aggregate.

    Two open PRs here draw the line narrower. #5675 (terminal_bench_2_1) leaves a row unscored only for verifier infrastructure errors, while a test.sh timeout and an absent or invalid reward.txt still score 0. #5655 (swe_rebench) treats exit 124 or 137 as a kill only when the output stops before the test output end marker, so a test command that itself exits 124 or 137 is still graded. Both treat "the evaluation didn't really run" as not a score, but only on a signal the agent can't easily produce.

    So the question: would you rather

    1. any non-zero exit masks, as proposed (simplest, matches the issue), or
    2. the verifier signals "not measured" explicitly with a reserved exit code, say 75 (EX_TEMPFAIL), so SkillEvaluator's judge failure maps to exit 75 and every other non-zero exit stays a measured result?

    With option 2 the setting could still default to off, and a dataset whose verifier can't be tampered with (separate mode) could opt into option 1. Either way, a timeout that left no reward would keep going through the timeout branch and score 0, and whatever the verifier wrote stays in verifier_rewards.

    I have option 1 working locally with tests and can switch to option 2 quickly. Happy to build whichever you prefer.

  2. added a commit that references this issue on Nov 11, 2026
    8c3b0a5
  3. linear-code commented on Nov 18, 2026

    @linear-code

    from adasif:

    Covered by main. Since #4049, a Harbor verifier that exits 75 without writing a reward is masked (harbor:verifier_unavailable), for separate-mode verifiers only; in shared mode the exit code is deliberately not trusted, because test.sh runs in the agent's sandbox. SkillEvaluator's verifier (shared mode; writes overall: 0.0, then exits 1) is therefore still scored on Gym; SkillEvaluator's own collector already drops those trials. If Gym's own numbers on these tasks need to exclude judge failures, the path is a separate verifier that exits 75 without a reward. #5799 is closed in favor of #4049.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions