Repository navigation
feat(agentic-vbench): add the Agentic-VBench Harbor benchmark entry - #4049
Conversation
a026267 to
f00da54
Compare
f00da54 to
d25c139
Compare
d25c139 to
7f124dd
Compare
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster> (cherry picked from commit f29bcc6) Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster> (cherry picked from commit 3a20fa1) Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster> (cherry picked from commit d268fb0) Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster> (cherry picked from commit 285e71c) Signed-off-by: Arushi Goel <arushig@nvidia.com>
…te Podman environment (cherry picked from commit 1655d98) Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: arushig <arushig@nsc-svg-slurm-1-login-01.nvidia.com> (cherry picked from commit f18f90e) Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com> (cherry picked from commit 21e99d5) Signed-off-by: Arushi Goel <arushig@nvidia.com>
The agent took the model endpoint and id from environment variables and OpenCode called vLLM directly, so a gym-native run's --checkpoint, served_model_name, seed and sampling overrides never reached it. The agent now references policy_model (model_server) and resolves the per-rollout proxy URL with base_url_for_run(get_server_url(...)), as opencode_sandboxed_agent does; the proxy rewrites the model name and applies policy_model.responses_api_models.vllm_model.sampling_overrides to OpenCode's chat completions. model_base_url/model_id stay as an escape hatch for launchers that serve the model themselves. Also: derive the episode root from the run's output_jsonl_fpath, emit equal_family_reward (reward * 25 / family size) and per-family reward keys, and compute mean/equal_family_reward (the leaderboard's equal mean of the four family means) in compute_metrics so Gym's aggregate carries the headline metric next to the task-weighted mean/reward. An episode directory a killed job left without gym_result.json is archived and the task rerun instead of aborting the resumed run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com> (cherry picked from commit 86e3284) Signed-off-by: Arushi Goel <arushig@nvidia.com>
A gym-native driver has only the Gym checkout, its venv and the cluster's cache mount. The benchmark now needs nothing else on the node: - harbor==0.6.6 is installed into the agent venv (requirements.txt), so harbor_python defaults to the agent's own interpreter; - prepare.py and the agent fetch PhiloLabs/agentic-vbench at the pinned revision into <HF_HOME>/../agentic_vbench with a shallow fetch by SHA (AGENTIC_VBENCH_ROOT / AGENTIC_VBENCH_REPO_URL still override); - a static Docker client with the compose plugin is downloaded into the cache when `docker` is not on PATH (Harbor's DockerEnvironment shells out to `docker compose`); - the remote rootless Podman backend is discovered through backend_dir/client.env (DOCKER_CERT_PATH rebound to that directory), the agent waits for it, probes `docker info` once at start, and removes an episode's containers when its runner fails or is cancelled; - the hub judge key falls back to the driver's INFERENCE_API_KEY, so no credentials file is required; - harbor_runner takes --backend, --judge-protocol, --max-turns, --verifier-timeout-multiplier and --model-timeout-ms from the agent configuration (environment variables remain as fallbacks) and sets OpenCode's provider timeout, which must cover a whole completion because Gym's proxy replays the stream once it is complete. config.yaml exposes every knob as an agentic_vbench_* selector for the launcher's run overrides. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com> (cherry picked from commit 2ced983) Signed-off-by: Arushi Goel <arushig@nvidia.com>
…nt Hub 4xx errors The Hub judge wrapper rejected the whole verification after repeated failures of any kind. For a 4xx that persists across retries the error is about this request (observed: Gemini refusing an empty audio extract), and the original judge, using the native SDK, catches that exception and scores the rubric item as a judge error while the rest of the verification continues. Raise such errors to the judge the same way; keep rejecting on transport (5xx, 408, 429, connection) and authentication failures so infrastructure problems never turn into silent zero scores. Signed-off-by: Arushi Goel <arushig@nvidia.com>
The primary OpenCode agent can delegate to subagents through its task tool, and the built-in subagents are available by default, so every run so far was 'default'. Add opencode_subagents to the agent config: 'off' removes the task tool and disables the built-in subagents (primary agent only); 'on' keeps them and registers a media-analyst subagent whose description makes delegation an explicit option. Task prompt and the primary agent's system prompt are unchanged in every mode; the mode is recorded so shards with different policies are never merged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Retry Hub judge calls that fail with 5xx/408/429 or transport errors up to 8 times with a 60 s-capped backoff (~3 min per item) instead of 5 quick attempts; request-scoped 4xx keep the short budget. Bedrock-backed Claude returned intermittent 503s for ~40 min and rejected otherwise complete Repurpose verifications. Scoring is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Agent setup downloads Node and OpenCode inside the task container; during registry slowdowns the 360 s default expires before the agent starts and the trial is lost as infrastructure. Setup time is not part of the agent budget, so scores are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
A task root carrying custom_tasks.json is treated as a set of generated tasks in the benchmark layout (tasks/agentic_vbench_<family>/<task>): ensure_checkout returns it as-is and inventory skips the pinned-revision, clean-tree and 100-task completeness checks. benchmark_revision becomes custom:<marker sha>. The pinned benchmark path is unchanged. Used for Curiosity SDG trace generation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor reports task.toml's name as <namespace>/<task>. Compare only the task component so generated tasks (e.g. curiosity-sdg/<task>) verify like the benchmark's agentic-vbench/<task>. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor ends each trial with 'docker compose down --rmi all', so every trial rebuilds its task image. keep_task_images=true (env AGENTIC_VBENCH_KEEP_TASK_IMAGES via the benchmark config) sets environment.delete to false so repeated trials of a task (trace generation) reuse one build and parallel shards no longer race on image deletion. Benchmark default unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
…materials The repair verifiers read /workspace/materials, which the task's setup step copies from the image's /baked into the agent's sandbox; a fresh verifier sandbox has only the declared artifacts, so 11 of the 18 oracle runs scored 0 with "missing source.mp4". stage_materials now declares /workspace/materials beside /workspace/output (a few MB per task). The judged tasks read their clip from /baked and are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor keeps reward.json as written and the benchmark reads `reward`; the server refused the whole file when any key was not a number, so the Agentic-VBench assembly and sequencing judges, which write a nested `details` block beside the score, were measured 0 (harbor:invalid_reward) on correct oracle solutions. With a numeric `reward`, non-numeric keys are now left out as metrics; without one every key is still a component and must be a number. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. The exit-75 rework follows the design closely:
- the marker file is gone;
_scoreclassifies from the exit record;- a written reward and the verifier timeout both win;
- every other missing reward stays a measured 0.
The Hub judge's new transport handling and the socket-server tests are solid, and every fix in this round has a test that fails when the fix is reverted.
What we reviewed: head f7b60fbfbc, the changes since 7de55d2f6b, against the benchmark-review checklist (Harbor format). We also read the pinned upstream tasks at 410a75fe3d.
Earlier threads: 24 were open.
- Resolved: 17 that this head fixes, each with a reply.
- Still open, with a reply: 7.
- the two evidence threads;
- the shared-mode part of the design thread;
- two Hub-judge gaps: a truncated error body and the soft deadline;
- the size-rejection thread;
- the separate-verification thread.
Overall: HOLD.
- One blocker: in the separate sandbox, 11 of the 18 repair verifiers can't find their inputs, so they score 0 on every rollout.
- The agent's output can still steer the separate verifier into exit 75, through its size and its length.
Every point is inline.
Merge state: no conflicts with main. CI has not run on this head yet.
Verified on a Linux workstation:
- the two benchmark test files: 28 passed, 1 skipped;
environment_servers/harbor/tests: 725 passed;- 150 repeated runs of the socket tests, including under load, with no flake;
- each claimed fix, reverted, fails its test;
- a prepare of the 100 tasks.
Not verified: any Docker or real-model run, and the Hub's request-size limit.
| "original_sha256": _file_digest(verifier), | ||
| "staged_sha256": _file_digest(target), | ||
| } | ||
| (staged / "task.toml").write_text(_declare_output_artifact(text)) |
There was a problem hiding this comment.
benchmarks/agentic_vbench/staging.py:306
Blocker: in the separate verifier, 11 of the 18 repair tasks score 0 on every rollout, the oracle included.
- Their
test.shpasses/workspace/materials/corrupted.mp4orsource.mp4to the judge (example). - The image creates that folder empty (Dockerfile). Only the step's
workdir/setup.shfills it, from/baked/input. - The server runs
setup.shin the agent's sandbox (environment_servers/harbor/app.py:1656), never in the fresh verifier sandbox (environment_servers/harbor/verifier.py:679). - So the judge writes 0 itself (
judge.py:325), and the row counts as a measured 0, not a masked one.
This follows from our own claim last round that the verifier needs only /workspace/output. Sorry about that.
Affected tasks:
exp-color-shot-{gobelins,v3-s1,visit-korea}exp-content-cut-{mkbhd,wsj}exp-deblur-{gaussian-mkbhd,motion-f1}exp-disfluency-interview-{3,4}exp-glitch-dup-longexp-sr-4x-shot
Could stage_materials rewrite the judge arguments from /workspace/materials/ to /baked/input/ in the staged test.sh?
- That is the image's own copy, the one
setup.shcopies from. It needs no extra copy, an honest agent's inputs are byte-identical, and the judge then compares against inputs the agent never touched. - Recording the rewrite in
materials.jsonand the README table would keep it traceable. - A staging test could assert that no step declaring an artifact still names
/workspace/materials.
The oracle validate would confirm the fix.
There was a problem hiding this comment.
Confirmed and fixed in 601f7b9: stage_materials now declares /workspace/materials beside /workspace/output as the step's artifacts (a few MB per task), so the fresh verifier sandbox receives what the agent's setup.sh copied from /baked/input; the judged tasks read their clip from /baked and are unchanged. The README says the folder is filled by setup.sh, not the image. Oracle: repair went from 7/18 at 1.0 (the 11 zeros were exactly missing source.mp4 / corrupted.mp4 at /workspace/materials) to 18/18 at 1.0.
| ) | ||
| if rewards is None: | ||
| exit_record = parse_exit_record(logs_dir) | ||
| if exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE: |
There was a problem hiding this comment.
environment_servers/harbor/verifier.py:1216
Note (the open part of the design thread): exit 75 is honored in shared mode too.
- The comment above and the docs say the code can be trusted only when
test.shruns separately, and they warn about shared mode._scoreitself doesn't know the mode. - This entry's 46 assembly and sequencing tasks run
set -euo pipefail; python3 /tests/judge.pyin the agent's sandbox. An agent that leaves apython3shim exiting 75 turns its own 0 into a masked row. - Tasks in the other
environments/harbor/*entries verify in shared mode unless theirtask.tomlasks otherwise, so the same applies there. - This widens an existing gap rather than opening a new one: a root agent can already force a masked
verifier_errorby removingtimeout(environment_servers/harbor/verifier.py:88).
We'd still lean toward enforcing it. _score would take a keyword separate: bool: True from _verify_separately, False from _run_shared_verifier, the two calls multi-step turns also use. In shared mode the row would stay a measured 0, as on main, with the ignored 75 noted in the reason. The cost is that a real outage in shared mode keeps scoring 0, which is today's behaviour. Your call; either way, please pin the choice with tests.
Tests that would pin it:
- a shared-mode case and a separate-mode case in
test_exit_75_without_a_reward_is_masked_not_scored; - a provider timeout with a
75exit record. Today, letting 75 beat the timeout still passes all 725 server tests.
There was a problem hiding this comment.
Enforced in 27dc4c1 as proposed: _score(..., separate: bool), passed True by _verify_separately and False by _run_shared_verifier, i.e. the path that ran this turn's test.sh, so a multi-step task with mixed turns gets each one right. Shared mode: no reward plus 75 falls through to the ordinary missing/invalid-reward result (measured 0) with "test.sh exited 75 in the agent's sandbox, where the code is not trusted" in the reason and verifier_return_code: 75 kept for counting. Separate mode: masked as before. The test covers separate (masked), shared (measured), valid-reward and timeout precedence and the ordinary nonzero exits; the docs and the troubleshooting row say the code is honored on the separate path only.
| | extras | ||
| ) | ||
| if rewards is None: | ||
| exit_record = parse_exit_record(logs_dir) |
There was a problem hiding this comment.
environment_servers/harbor/verifier.py:1215
Note: rewards is None also covers an invalid written file, so exit 75 plus a malformed reward.txt is masked. The docs say "a written reward always wins", and the outcome table scores an invalid file 0 (harbor:invalid_reward).
Masking seems the right call here: only the verifier writes that file, and it then reports it could not grade. Could the docs say "a valid written reward wins", with a test to pin it? No test covers either behaviour yet.
There was a problem hiding this comment.
Agreed on masking. The docs now say a valid written reward wins and that a malformed file beside the code is the verifier's own and leaves the run masked (task-folder-verifiers.mdx, the outcome table's first two rows carry the exception). test_exit_75_without_a_reward_is_masked_only_on_the_separate_path pins malformed reward.txt plus 75 as masked (separate), valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as harbor:invalid_reward, and the shared-mode case (27dc4c1).
| # A verifier timeout wins over the code, and in shared mode the agent's sandbox runs test.sh, so the code is | ||
| # trustworthy only for a task verified separately. | ||
| VERIFIER_UNAVAILABLE_EXIT_CODE = 75 | ||
| VERIFIER_UNAVAILABLE_KIND = "harbor:verifier_unavailable" |
There was a problem hiding this comment.
environment_servers/harbor/verifier.py:100
Question on the kind. Recovery keeps a masked row of a namespaced kind as the final result (nemo_gym/rollouts/rollout_recovery.py:456-478).
After one Hub outage, gym eval resubmit --recover stops with "masked … the run is not complete" (nemo_gym/orchestration/recovery.py:1590-1596). Only a fresh run can measure those rows.
That matches the other harbor:* masked kinds, so it isn't a regression. It does make "masked" mean "the run ends FAILED" until the bounded server-side re-run lands.
The registered verifier_error would give the operator a way out. A Harbor row keeps the agent's answer, so it would not be retried automatically, but --recover --rerun-masked-without-artifact --rerun-reason … could rerun it. That gate is better than an automatic retry while some 75s still depend on the agent's output. You would keep "exited 75" in the reason and verifier_return_code: 75 for counting.
Either option works: switch the kind, or have the troubleshooting row say such a run ends FAILED and needs a fresh run.
There was a problem hiding this comment.
Kept the namespaced kind and documented the consequence (27dc4c1): the troubleshooting row and the verifier page now say that gym eval resubmit --recover keeps the row masked and stops with "the run is not complete", and that a fresh run measures it once the service is back. I preferred this to switching to verifier_error, whose row above means "bash or timeout missing"; one kind with two meanings would make the per-kind counts harder to read, and the bounded server-side re-run is the follow-up that gives the operator a real way out.
|
|
||
| def _size_rejection(code: int, detail: str) -> bool: | ||
| """A request the Hub refuses for its size (inline media): infrastructure, not a judge error of this item.""" | ||
| return code == 413 or (code == 400 and any(w in detail.lower() for w in ("too large", "exceeds", "payload size"))) |
There was a problem hiding this comment.
benchmarks/agentic_vbench/hub_judge.py:65-67
Risk: the agent decides the request size, so its output can turn a failure into a masked row.
HubGeminiinlines the agent's wholerepurpose.mp4as base64 (benchmarks/agentic_vbench/hub_judge.py:172-190), where upstream uploads it through the Files API.- An overlong or high-bitrate output gets a 413, then exit 75 and a masked row. Say, the untrimmed 59-358 MB source.
- The bare
"exceeds"also matches Gemini's token-limit 400 ("input token count … exceeds the maximum"). That one is driven by output length, and upstream treats it as an item error.
Our suggestion last round to treat 413s as "could not grade" missed that the agent picks the size. Sorry for steering you there.
A fix in three parts:
- Bound only the video, since inlining is this shim's own deviation. Above a byte budget, send an ffmpeg proxy (at most 720p, keep the audio, bitrate = budget / duration); below it, send the bytes unchanged. Record the settings in the provenance.
- Then drop the 400-text match. A persistent size rejection on the audio or Claude items (upstream's own request shapes) becomes the item's
request_error, as upstream. - In the smoke, send the largest legitimate request: a full 250-frame pillar, a long mp3, and a proxy at the budget.
There was a problem hiding this comment.
Done in 53da51e, in the three parts you list. (1) Only the video is bounded: HubGemini.upload sends a clip over VIDEO_INLINE_BUDGET_BYTES (14 MiB, below Gemini's 20 MB inline request limit after base64) as an ffmpeg proxy, scale to at most 720p high, audio kept at 96 kbps, video bitrate = 0.92 x budget / duration, made once per file; below the budget the bytes go unchanged. The settings are recorded in every result as judge_media_transforms (original and proxy bytes, duration, max height, bitrates) beside judge_video_inline_budget_bytes. (2) The 400-text match and the 413 special case are gone: both are this request's error, raised to the judge as upstream. The claims in staging.py, the overlay, both READMEs and the PR body now say what the agent cannot do (run the verifier, read the key) and that the exit code is decided from the Hub's answers and the download's outcome, never from the output. (3) The proxy test uses a generated 2 s clip and is skipped where ffmpeg is absent (the CI container; the task images have it); the no-ffmpeg fallback has its own test. The largest-request smoke is owed with the full run.
| error = "unknown" | ||
| request_error = False | ||
| for attempt in range(ATTEMPTS): | ||
| if time.monotonic() - _STARTED > DEADLINE_SECONDS: |
There was a problem hiding this comment.
benchmarks/agentic_vbench/hub_judge.py:85-88
Risk: the deadline is soft, and it also counts time the agent's output causes.
- It is checked only before an attempt, and
urlopen(timeout=…)bounds each socket read, not the call. A trickling response held one call for 17 s against a 1 s deadline and a 0.5 s call timeout. _STARTEDis set at import, aftertest.sh'sfind /and its copies.- After the deadline, the original judge keeps doing local work (Whisper per audio item, frame extraction).
The result goes both ways. A slow but healthy Hub can still reach the 1800 s kill, a measured 0. A very long output can push local time past 1500 s and get masked.
A possible shape:
- Have the staged
test.shexport its start time. - Run a
threading.Timerthat callsos._exit(75)(sincesys.exitin a thread doesn't end the process) at the budget minus a margin foraggregate.py. - Exit 75 only when time spent in Hub calls accounts for the delay; otherwise let the run reach the verifier timeout.
- Pass
timeout=min(CALL_TIMEOUT_SECONDS, remaining)to each call.
Neither a deadline passing mid-retry nor a trickling response has a test yet.
There was a problem hiding this comment.
Reworked in 53da51e along the lines you describe, with one difference. Instead of a wall-clock timer, the budget is now an allowance of time spent waiting on the Hub (HUB_WAIT_SECONDS, 1200 s): _call runs the request on a thread and the caller stops waiting at min(CALL_TIMEOUT_SECONDS, remaining) (a result after the bound is dropped), the backoff is bounded by what is left and counts as waiting, and nothing else counts, so the import-time _STARTED is gone and local work (Whisper, frame extraction, the proxy encode) only ever runs the verifier into its own timeout, a measured 0. Exit 75 therefore happens only when Hub waiting accounts for the delay, which is what the timer-plus-check would have decided; the staged test.sh needs no start export. Tests: a stalled response with 0.4 s left ends in about 0.4 s, not the 2 s socket timeout; expiry during the backoff after a 503 makes exactly one attempt; 1.2 s of local work before a healthy call records nothing; a spent allowance makes no call.
| }, | ||
| {"anthropic-version": "2023-06-01"}, | ||
| ), | ||
| "Gemini": ( |
There was a problem hiding this comment.
benchmarks/agentic_vbench/hub_judge.py:212-226
Note: the probes don't send what the judge sends.
- Real Gemini calls carry
thinkingConfigandtemperature. Real Claude calls carry asystemblock withcache_control. - A route that accepts the "ping" but answers 400 to every real request passes preflight. Every item then scores 0 as a judge error, the case the preflight exists to catch.
Could the probes carry those fields? Two more suggestions:
- Run the preflight lazily, on the first Hub call. Then an output the original judge rejects without calling the Hub (under 10 KB, under 2 s, unreadable) still scores 0 during an outage instead of being masked.
- In the smoke, show both probes and one real item passing on each protocol.
There was a problem hiding this comment.
Done in 53da51e. The probes are built by probes() in the judge's shapes: Claude with a system prompt and a cache_control block on the user content, Gemini with gemini_generation_config(0.0, 16), the same builder every real call uses (thinking level and temperature included). The preflight is lazy: post() runs it before its first real call, once, so an output the judge rejects locally scores 0 during an outage instead of being masked, and main() no longer probes up front. Tests: the probe payloads' fields, the probes going out through the first real post() and not again, and a judge that makes no Hub call running without probes. The smoke showing both probes and a real item on each protocol is owed with the full run; the Hub key cannot do it today (budget exhausted, see the dataset.toml thread).
| sandbox or a reward that could not be copied out, set `mask_sample`: the rollout stays in `rollouts.jsonl` but is | ||
| left out of the score. | ||
|
|
||
| A verifier that cannot grade, because a judge service or an asset download it depends on is unavailable, exits |
There was a problem hiding this comment.
fern/versions/latest/pages/data/task-folders/task-folder-verifiers.mdx:55
Note: the page now contradicts itself.
fern/versions/latest/pages/data/task-folders/task-folder-verifiers.mdx:34-35still says "The exit code oftest.shdoes not affect the score."- The outcome table at
fern/versions/latest/pages/data/task-folders/task-folder-verifiers.mdx:45-49has no exit-75 row.
The table's caption says "These rows count in the score", so the simplest fix is to qualify L34-35 ("…except exit 75 without a reward, below"). Alternatively, add a row marked as masked.
It would also help to say that separate mode is necessary but not sufficient: nothing derived from the agent's artifact (its size, its duration) may decide the exit code.
There was a problem hiding this comment.
Fixed in 27dc4c1: the opening statement now says the exit code never changes a valid reward file's result and that, without one, every exit is a measured 0 except exit 75 from a separately run verifier; the table's first two rows carry that exception; "a valid written reward wins"; and the paragraph adds that separate mode is necessary, not sufficient, with the artifact's size, duration and content named as things that may not decide the code.
| # to null, which the server's `secrets` table rejects, so the server does not start without a key and no repurpose | ||
| # task can be scored 0 for want of one. The judged (repurpose) and repair tasks declare `/workspace/output` as their | ||
| # step's artifact; `separate_verifier_for_artifacts` grades them in a fresh sandbox of the task image, so the agent's | ||
| # sandbox never runs the verifier that holds the key, and cannot make it exit 75 ("could not grade"). Assembly and |
There was a problem hiding this comment.
environments/harbor/agentic_vbench/harbor_sandbox.yaml:11-14
Note (docs): a few claims need adjusting once the points above are settled.
- "Cannot make it exit 75" overstates it.
- The same wording is in
benchmarks/agentic_vbench/staging.py:16-19,benchmarks/agentic_vbench/README.md:26and the PR body. - The agent cannot run the verifier process or read the key, but the exit code still depends on the output it hands over (
hub_judge.py:67,:85).
- The same wording is in
environments/harbor/agentic_vbench/README.md:51-52says the image provides/workspace/materials, but each step'ssetup.shfills it from/baked/…. The blocker onstaging.py:306rests on the same assumption.benchmarks/agentic_vbench/README.md:25says the other 64 tasks need no credentials, but with this overlay the server does not start without a key. A line on running them without Hub access would help.
There was a problem hiding this comment.
All three adjusted (2478175, 53da51e). The "cannot make it exit 75" wording in staging.py, the overlay, benchmarks/agentic_vbench/README.md and the PR body now says what the agent cannot do (run the verifier process, read the key) and that the verifier derives the code from the Hub's answers and the download's outcome, never from the output, which with the size and allowance changes is now true of the code. The README says /workspace/materials is filled by each step's setup.sh from /baked in the agent's sandbox, which is why the repair verifiers receive it as an artifact (601f7b9). The 64 unjudged tasks have a keyless recipe: AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing plus the new harbor_sandbox_no_judge.yaml, the same overlay without the secret.
| "handler,expected", | ||
| [ | ||
| # A reset instead of the EOF is possible when the peer closes first; both are transport failures. | ||
| (_drop_before_status_line, ("RemoteDisconnected", "ConnectionResetError")), |
There was a problem hiding this comment.
tests/unit_tests/benchmarks/test_agentic_vbench_staging.py:510
Tests: every fix in this round has a test that fails when the fix is reverted. Nice work; 150 runs under load showed no flake.
A few branches stay green when reverted:
FAILURESkeyed by path only: two payloads on one route, where only one later succeeds.- The 400 "payload size" branch of
_size_rejection. - The deadline passing mid-retry after a 4xx: the
request_error = Falsereset atbenchmarks/agentic_vbench/hub_judge.py:87. - The continuation-line loop in
_guard_materials_fetch(benchmarks/agentic_vbench/staging.py:347-349), which no pinned task needs.
Here, truncated-body yielded IncompleteRead every time, so the looser expected tuple could narrow to it.
There was a problem hiding this comment.
Thanks. Each of the four now has a test that fails when the branch is reverted (53da51e): two payloads on one route with one later superseded (test_hub_post_keys_failures_by_payload_and_lets_a_final_answer_supersede_them); the _size_rejection 400 branch is gone with the function, and the same test pins a 413 and a token-limit 400 as item errors; the mid-retry expiry (test_hub_allowance_counts_waiting_on_the_hub_not_local_work, one attempt then the allowance message); and the continuation-line loop (test_materials_fetch_guard_follows_a_multi_line_curl_command). The truncated-body tuple is narrowed to IncompleteRead.
|
/ok to test f7b60fb |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
The Hub shim still lets a transport failure become a measured zero and lets output-size rejections select masking. The generic exit-75 branch also trusts shared verifiers.
I inspected all 16 changed files and traced the affected scoring and recovery paths. Valid parsed rewards still take precedence, and verifier timeouts are checked before exit 75.
This was a static review. I did not run tests, Docker builds or model evaluations. The pinned external tasks are not mounted, so I could not independently confirm the reported missing-input failure in 11 Repair tasks. The entry README confirms that oracle and model-run validation are still missing.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| """Short, single-line provider status message; empty when the body is not a JSON error.""" | ||
| try: | ||
| body = json.loads(exc.read().decode(errors="replace")) | ||
| except (ValueError, OSError): |
There was a problem hiding this comment.
BLOCKER: a truncated HTTP error body bypasses outage handling. In provider_error_message, exc.read() can raise http.client.IncompleteRead, which this handler does not catch. The call runs inside post()'s exception handler, so its exception bypasses the retries and FAILURES. During preflight it also escapes the RuntimeError handler, leaving an ordinary missing reward rather than exit 75.
Could this handler include http.client.HTTPException?
| except (ValueError, OSError): | |
| except (ValueError, OSError, http.client.HTTPException): |
This keeps the HTTP status available for classification even when its diagnostic body is unusable. Preserve ordinary request-scoped 400 handling, short single-line diagnostic messages, and the existing handling of truncated successful responses.
Proposed acceptance: add a socket response with status 503, Content-Length: 400, and a short body. After retries, post() should raise RuntimeError and record one failure; through preflight, main() should exit 75. Keep the existing assertions that persistent ordinary 400s leave FAILURES empty and truncated 200 responses are recorded as transport failures. I have not executed these checks.
There was a problem hiding this comment.
Done in 53da51e: provider_error_message catches http.client.HTTPException too, keeping the status for classification. Acceptance as proposed: the socket test's truncated-503-body case (503, Content-Length: 400, short body) makes post() raise RuntimeError after the retries and record one failure; test_hub_preflight_exits_75_on_a_truncated_error_body runs preflight() through the real post() against it and gets exit 75. Persistent ordinary 400s still leave FAILURES empty and the truncated-200 case is still a recorded transport failure.
|
|
||
| def _size_rejection(code: int, detail: str) -> bool: | ||
| """A request the Hub refuses for its size (inline media): infrastructure, not a judge error of this item.""" | ||
| return code == 413 or (code == 400 and any(w in detail.lower() for w in ("too large", "exceeds", "payload size"))) |
There was a problem hiding this comment.
BLOCKER: answer-size errors can remove failed work from scoring. HubGemini inlines the entire agent-produced file. Every 413, and a 400 containing the bare word exceeds, enters FAILURES. For example, {"error":{"message":"input token count exceeds the maximum"}} makes main() exit 75, masking the whole verification. Separate execution protects the verifier process, but does not make the answer's size infrastructure-owned.
Could post() keep answer-dependent token/payload rejections request-scoped, while the Gemini media path provides a validated transport for supported legitimate clips? Removing the classification alone is incomplete if legitimate clips still exceed this shim's inline limit. The invariant is that oversized or overlong answers cannot select masking, while supported answers remain gradable. The mounted source does not establish a supported upload API or a safe evidence-preserving proxy budget, so I cannot supply a complete transport patch. Record any media transformation in the protocol provenance and correct the claims that the agent cannot influence exit 75.
Proposed acceptance: persistent output-size 413 and token-limit 400 responses must not select exit 75 solely because of the answer. Exercise the largest supported video under both protocols and show grading succeeds. Preserve accepted media bytes and rubric evidence, plus masking for authentication failures, 408/429, 5xx and dropped connections. Provider limits and these checks were not verified here.
There was a problem hiding this comment.
Done in 53da51e. Persistent 413s and token-limit 400s are request-scoped: raised to the judge as the item's error, never in FAILURES, so they cannot select exit 75; the test pins both. The Gemini media path bounds the inline deviation: a clip over 14 MiB goes as an ffmpeg proxy (at most 720p, audio kept, bitrate from the budget and duration), recorded in every result as judge_media_transforms; supported clips under the budget go unchanged. Without ffmpeg the file goes as it is and the route decides, as an item error. The claims that the agent cannot influence exit 75 are reworded everywhere to what the agent cannot do and what decides the code.
| ) | ||
| if rewards is None: | ||
| exit_record = parse_exit_record(logs_dir) | ||
| if exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE: |
There was a problem hiding this comment.
RISK: exit 75 is trusted in the agent's shared sandbox. Both _run_shared_verifier and the separate path call this classifier. A shared task whose trusted script invokes an agent-modifiable executable can therefore return 75 without a reward and become masked instead of the measured zero it received on base. The new test exercises this shared-mode behavior. The documentation warns about it, but the classifier does not enforce the stated infrastructure-only boundary.
Could _score take a required keyword separate: bool, passed as True by _verify_separately and False by _run_shared_verifier, and use this guard?
if separate and exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE:Use the execution path, not the task's overall mode: multi-step tasks can mix shared and separate turns. Shared 75 should fall through to the ordinary missing/invalid-reward result. This closes this new masking path; it does not claim to make shared verification a security boundary.
Proposed acceptance: no reward plus 75 is measured in shared mode and masked in separate mode, including artifact-forced separate turns. An explicitly shared turn beside a separate turn must stay measured. Retain valid reward precedence, timeout precedence even with a 75 exit record, and measured zero for ordinary nonzero exits. These are proposed checks, not executed results.
There was a problem hiding this comment.
Done as proposed in 27dc4c1: _score(..., separate: bool) from the execution path (_verify_separately True, _run_shared_verifier False), and the guard if unavailable and separate:. Shared 75 falls through to the ordinary missing/invalid-reward result with the ignored code in the reason and verifier_return_code kept. The test covers no reward plus 75 measured in shared mode and masked in separate mode, valid-reward precedence, the timeout record winning (existing test), and measured zeros for ordinary nonzero exits.
| request_error = False | ||
| break | ||
| try: | ||
| with urllib.request.urlopen(request, timeout=CALL_TIMEOUT_SECONDS) as response: |
There was a problem hiding this comment.
RISK: the deadline does not bound a request already in progress. post() checks the deadline only before starting an attempt. An attempt started with one second left still receives the full 120-second timeout, and a response completed after the deadline is accepted without another check. The configured deadline therefore does not guarantee the promised finishing margin before the verifier's 1800-second kill.
Could request supervision enforce the remaining allowance across opening and consuming the response, and bound retry sleeps by that allowance? Passing min(CALL_TIMEOUT_SECONDS, remaining) is necessary but does not by itself provide whole-operation supervision. Keep this bound scoped to Hub work so local media processing does not become an infrastructure failure. The cancellation mechanism needs an implementation decision, so there is no safe one-line replacement to suggest.
Proposed acceptance: start a controlled request before the deadline and delay completion beyond it; verify that the configured bound ends the Hub operation rather than accepting a late success. Also cover expiry during backoff. Preserve successful requests within the allowance, recovery after transient failures, and no new request when the Hub allowance is already exhausted. I did not execute these cases.
There was a problem hiding this comment.
Done in 53da51e: _call runs each request on a thread and the caller waits at most min(CALL_TIMEOUT_SECONDS, remaining) for the whole operation (open plus body); a late result is dropped, the time waited counts against the allowance, and the retry sleep is bounded by what is left. Tests: a stalled server with 0.4 s left ends in a TimeoutError in about 0.4 s (not the 2 s socket timeout) with one failure recorded and the allowance spent; expiry during the backoff makes one attempt; a healthy call within the allowance succeeds; a spent allowance makes no request.
| error = "unknown" | ||
| request_error = False | ||
| for attempt in range(ATTEMPTS): | ||
| if time.monotonic() - _STARTED > DEADLINE_SECONDS: |
There was a problem hiding this comment.
RISK: local answer processing can be classified as a Hub outage. The clock starts at module import, not when Hub waiting starts. Any local work before a later post() consumes it, including the shim's file reading and base64 encoding. Once that time exceeds the allowance, even a healthy Hub is never contacted; the request is recorded as unavailable and the verification is masked.
A minimal proposed reproduction is a one-second judge allowance, more than one second of local preparation after import, then an immediately successful Hub endpoint. Could the infrastructure allowance track time spent waiting on Hub operations separately from artifact-driven local work? Local work alone must not select exit 75; it should stay subject to the task's ordinary verifier timeout. Keep the hard in-flight Hub bound requested separately.
Proposed acceptance: local preparation alone must not populate FAILURES or mask an otherwise healthy request; genuinely exhausted Hub waiting must still report unavailable. Update the existing expired-deadline test to set up exhausted Hub allowance explicitly rather than equating all time since _STARTED with an outage. Preserve the server's measured-zero verifier-timeout outcome. These checks were not run.
There was a problem hiding this comment.
Done in 53da51e: the allowance (HUB_WAIT_SECONDS) accumulates only time spent in _call and in the backoff between attempts, so local work before or between calls (file reading, base64, the proxy encode, the judge's own Whisper and frame extraction) never consumes it; the import-time clock is gone. Your reproduction is the test: 1.0 s allowance, 1.2 s of local waiting after import, then a healthy endpoint, which succeeds with FAILURES empty; the old expired-deadline test now sets the Hub waiting itself to the allowance and gets "allowance ... is spent" with no request made. Local work still ends in the server's verifier timeout, a measured 0.
|
|
||
| ## Scoring and validation status | ||
|
|
||
| The reward is the task's own `reward.json`, 0..1. The leaderboard's equal-family mean is `reward x 25 / family size` |
There was a problem hiding this comment.
RISK: the headline metric recipe mixes row normalization with family averaging. Multiplying by 25 / family size is correct when averaging all 100 task rows, not when averaging four family means. Following the latter reading gives (25/18 + 25/18 + 25/28 + 25/36) / 4 ≈ 1.0913 when every reward is 1. The measurement contract repeats the same recipe.
Could both locations say:
The leaderboard score is
(mean_repair + mean_assembly + mean_sequencing + mean_repurpose) / 4, using the unscaled family means. Equivalently, over the complete 100-task set, averagereward_i * 25 / size_of_family_iacross the 100 task rows.
Keep the distinction from the server's task-weighted metric and the restriction against reporting a subset as the full benchmark score. Do not change per-task rewards or silently impute masked rows as zero.
Proposed acceptance: a complete synthetic set with every reward 1 yields 1.0. Only Repair scoring 1 yields 0.25; only Sequencing scoring 1 also yields 0.25. Both formulas must agree. These are proposed arithmetic checks, not executed tests.
There was a problem hiding this comment.
Fixed in both places (2478175), in your words: the leaderboard score is (mean_repair + mean_assembly + mean_sequencing + mean_repurpose) / 4 with the unscaled family means, equivalently the mean of reward_i x 25 / size_of_family_i over the complete 100 rows; a subset is not the benchmark score and a masked row is not a 0. The two formulas agree on your three checks (all ones gives 1.0; Repair alone or Sequencing alone at 1 gives 0.25).
| ), | ||
| "Gemini": ( | ||
| f"/v1beta/models/{os.environ['AVB_JUDGE_GEMINI_MODEL']}:generateContent", | ||
| {"contents": [{"role": "user", "parts": [{"text": "ping"}]}], "generationConfig": {"maxOutputTokens": 1}}, |
There was a problem hiding this comment.
NOTE, suggestion: probe the Gemini controls used by real calls. Every generate_content() request carries thinkingConfig, and configured requests carry temperature; this probe carries neither. It therefore does not validate those shared protocol settings. A route that accepts the ping but rejects the configured thinking level would pass preflight and then turn that protocol incompatibility into item errors.
Could preflight reuse the real generation-config construction, with an appropriate small output budget? Keep evidence-dependent errors request-scoped rather than promoting every real-request 400 to an outage. I have not established that either configured route currently rejects these fields, so this is a coverage improvement, not a confirmed provider failure.
Proposed acceptance: a fake endpoint that accepts plain requests but rejects the configured thinking field should fail preflight with exit 75. An accepting endpoint should permit judging, and an invalid-media 400 after a successful probe should remain an item error. Preserve the configured thinking level and temperature in real requests.
There was a problem hiding this comment.
Done in 53da51e: the Gemini probe's generationConfig comes from gemini_generation_config(0.0, 16), the builder every real call uses, so it carries the configured thinking level and temperature with a small output budget; the Claude probe carries system and a cache_control block. A route that accepts a bare ping but refuses those fields now fails preflight with exit 75, and an invalid-media 400 after a passing probe stays an item error (request-scoped 4xx handling unchanged). Test: the probe payloads' fields and the probes going out through the first real call.
| sandbox or a reward that could not be copied out, set `mask_sample`: the rollout stays in `rollouts.jsonl` but is | ||
| left out of the score. | ||
|
|
||
| A verifier that cannot grade, because a judge service or an asset download it depends on is unavailable, exits |
There was a problem hiding this comment.
NOTE: align the reward-file contract with the new branch. The page still says the exit code does not affect the score, and its outcome table says every missing or malformed reward is measured zero. The new paragraph introduces an exception but says any written reward wins. In _score, parsing a malformed file yields rewards is None, so a non-timed-out exit 75 masks it too.
Could the opening statement say that exit codes do not change a parsed reward file's result, qualify the missing/malformed rows with the exit-75 exception, and change “A written reward” to “A valid written reward”? Keep the distinction that parsed finite components whose sum overflows still produce harbor:invalid_reward; that path does not enter the new branch. Apply the same wording to the troubleshooting row.
Proposed acceptance: pin malformed reward.txt plus 75 as masked in separate mode, valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as measured invalid reward, and timeout precedence over 75. Inspect the revised text against those outcomes. These tests were not executed.
There was a problem hiding this comment.
Aligned in 27dc4c1: the opening statement says the exit code never changes a parsed, valid reward file's result and names the one exception; the first two table rows carry it; "a valid written reward"; the overflow-sum path stays harbor:invalid_reward (unchanged code, outside the new branch). The troubleshooting row has the same wording. The test pins malformed reward.txt plus 75 as masked in separate mode, valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as measured invalid, and timeout precedence over 75 (existing test).
| | `harbor:missing_reward` | `test.sh` wrote neither `reward.txt` nor `reward.json`. A Python verifier that raised lands here. | Yes, as 0 | | ||
| | `harbor:invalid_reward` | The reward file could not be read as a finite number. | Yes, as 0 | | ||
| | `harbor:verifier_timeout` | `test.sh` ran past its timeout without a reward. | Yes, as 0 | | ||
| | `harbor:verifier_unavailable` | `test.sh` exited 75 (`EX_TEMPFAIL`) and wrote no reward: a judge service or asset download it depends on was unavailable. Trustworthy only when the task verifies separately. | No, masked | |
There was a problem hiding this comment.
NOTE, suggestion: document what recovery does with this masked kind. This new outcome is namespaced and unregistered. _masked_evaluation_side_kind excludes it from recoverable infrastructure rows, so default recovery keeps it masked. The recovery planner then stops with “the run is not complete.” “No, masked” does not tell an operator that retrying after the service recovers will leave the row unchanged.
Could this row or the adjacent fix paragraph state that --recover does not rerun this outcome and explain the supported way to remeasure it? If no selective rerun is supported for this kind, say that a fresh run is needed. This can be a documentation change; it need not introduce automatic regeneration or a server-wide retry mechanism here.
Static acceptance: the documented recovery action should match the registration guard and planner stop above. Preserve masking rather than zero-imputation, and keep the explicit statement that the server does not rerun a masked verification.
There was a problem hiding this comment.
Done in 27dc4c1: the row now says that gym eval resubmit --recover keeps the row masked and stops with "the run is not complete", and that a fresh run is needed to measure it once the service is back; the verifier page says the same. Masking rather than zero-imputation and the statement that the server does not rerun a masked verification are kept.
| Both commands take the deployment overlay: it carries the judge key, the episode cap, separate verification of the | ||
| judged and repair tasks and the harness's `image_input`, which the tasks need for the model to see the frames it | ||
| extracts. The key is required to start the server (below), so Repurpose can only be scored with access to the NVIDIA | ||
| Inference Hub; the other 64 tasks need no credentials. |
There was a problem hiding this comment.
NOTE, suggestion: give the other 64 tasks a keyless run recipe. The README says they need no credentials, but both documented commands use an overlay whose secret resolves to null without a Hub key. The server's secrets: dict[str, str] rejects that configuration even after preparation selects only Repair, Assembly and Sequencing.
Could you document AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing preparation together with an overlay that omits the judge secret? Keep separate_verifier_for_artifacts: true and image_input: true; dropping the entire overlay would also drop those required behaviors. Leave the full-set overlay's missing-key refusal intact.
Proposed acceptance: with both key variables unset, the 64-task preparation and keyless overlay should start and reach verification. The full-set overlay should still reject a missing key, and Repair should still verify separately. This is a documentation usability suggestion; I did not run either configuration.
There was a problem hiding this comment.
Done in 2478175: the entry README documents AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing preparation with the new harbor_sandbox_no_judge.yaml, the same overlay without the judge secret (separate_verifier_for_artifacts: true, the episode cap and image_input: true kept); the benchmark README's credentials cell points to it. The full-set overlay keeps its no-default secret, so it still refuses to start without a key. I have not run the keyless configuration end to end; the 64-task oracle numbers below come from the full-set overlay with a key.
`_score` takes `separate`: True from `_verify_separately`, False from `_run_shared_verifier` (the path, not the task's mode, since multi-step tasks mix the two). In the agent's sandbox a 75 without a reward is a measured 0 as on any other exit, with the ignored code in the reason; separately it is masked, a malformed reward beside it included (the verifier's own file). The docs say a valid written reward wins, qualify "the exit code does not affect the score", state that separate mode is necessary but not sufficient, and say what `gym eval resubmit --recover` does with the masked kind. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
hub_judge.py: the overall deadline becomes an allowance of time spent waiting on the Hub (calls in flight and backoff, 1200 s), so local work over the agent's output cannot look like an outage; each call is bounded as a whole on a thread and a late success is dropped; a 413 or a token-limit 400 is the item's error as upstream, never "could not grade", and a clip over the 14 MiB inline budget goes as an ffmpeg proxy (at most 720p, audio kept) recorded in the provenance; failures are keyed by route and payload and a final request error supersedes an earlier outage of the same request; a truncated error body is read for its status; the probes carry the judge's real request shapes and run before the first Hub call, so an output the judge rejects locally scores 0 during an outage. staging.py: `--retry-max-time 900` leaves the repair verifier room after the download. Tests cover each branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
…als provenance A `harbor_sandbox_no_judge.yaml` overlay and the `AGENTIC_VBENCH_HARBOR_FAMILIES` recipe for the 64 tasks that need no credentials; the leaderboard score as the mean of the four unscaled family means (equivalently `reward x 25 / family size` over the complete 100 rows); `/workspace/materials` is filled by each step's setup.sh from /baked, not by the image; the exit-75 claims say what the agent cannot do (run the verifier, read the key) and what decides the code. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
… README Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
|
Pushed
Oracle run of every Still owed and not yet posted: the full-set |
…et opencode_agent run Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
|
/ok to test f978bbd |
1 similar comment
|
/ok to test f978bbd |
|
/ok to test f978bbd |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43983812460 |
…y landed as NVIDIA-NeMo#4049) Main's squash of NVIDIA-NeMo#4049 matches this branch's pre-training entry, so the add/add conflicts resolve to this branch's files; the entry README takes main's recorded full-set run and injected-outage smoke in place of the open-items paragraph. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arushi Goel <arushig@nvidia.com>
Since the Agentic-VBench benchmark entry (NVIDIA-NeMo#4049), main's reward parser skips non-numeric keys (a `details` block, a `reason`) only beside a numeric `reward`; without one every key must be a number. The parser did not know the dataset's `reward_key`, so after the merge a file such as `{"overall": 0.5, "accuracy": 1.0, "details": {...}}` under `reward_key = "overall"` was refused whole: a measured 0 with `harbor:invalid_reward`. The same file with a `reward` key keeps its score. `parse_reward_file` now takes the score key, and `_collect_reward` passes the turn's `reward_key` on both verification paths: - A numeric value under that key makes the other keys metrics, as `reward` does, so non-numeric keys beside it are left out. - A file without the key holds no valid reward. A separately run test.sh that exits 75 is then masked as `harbor:verifier_unavailable`, and a run past its deadline is `harbor:verifier_timeout`, as for any file without a valid reward. Before, such a file parsed as valid and `_score` measured it as `harbor:invalid_reward`. `_score` no longer needs its own missing-key branch. Tests: a `details` block beside the key keeps the score, a file without the key is masked on exit 75 and a timeout past the deadline, and a file whose other keys sum to the score (`{"overall": 0.5}` or with a 0.0 penalty) names no `reward_components`. That last test fails if `reward_components` stops checking the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
What
Agentic-VBench, the 100 pinned video-editing tasks of PhiloLabs/agentic-vbench (Repair 18, Assembly 18, Sequencing 28, Repurpose 36; revision
410a75fe3d, Apache-2.0), as a task-folder benchmark on theharborEnvironment Server.environments/harbor/agentic_vbench/: the committed entry (dataset.toml, README,harbor_sandbox.yamldeployment overlay: judge key as a server secret,separate_verifier_for_artifacts, the harness'simage_input, episode cap). The tasks are written into it by the prepare script, not pulled from a Harbor hub.benchmarks/agentic_vbench/:prepare_harbor_env.py(writes the 100 tasks),checkout.py(pinned fetch and inventory),staging.py(per-task adjustments),hub_judge.py(the repurpose rubric judge with both providers on the NVIDIA Inference Hub).harborEnvironment Server change (reaches everyenvironments/harbor/*entry): a verifier that exits 75 (EX_TEMPFAIL) without writing a reward is masked withharbor:verifier_unavailableinstead of scoring 0, classified from the exit record the wrapper already writes. A written reward and a verifier timeout still win; every other exit without a reward stays a measured 0. Documented intask-folder-verifiers.mdxand the task-folder troubleshooting table.Per-task adjustments, recorded under the entry's
.records/<task>/with digests: the repurpose judges route through the Hub with a verifier-only${NVINFERENCE_API_KEY}template, probe each route (in the judge's own request shapes) before their first Hub call, and exit 75 on outages, unusable routes or a spent allowance of time waiting on the Hub; what the agent's output decides never selects that exit (a request the route refuses for its size or length is that item's error, as upstream, and a clip over the inline budget goes as a bounded ffmpeg proxy, recorded in the provenance). The pinned assembly judges (verified by digest) score a non-objectsolution.json0 instead of crashing; the repair verifiers retry the golden-asset download longer within a bound and exit 75 when it still fails. The judged and repair steps declare their artifacts (/workspace/output, plus/workspace/materialsfor repair, which the setup step fills in the agent's sandbox and the repair verifiers read), so with the overlay they verify in a fresh sandbox of the task image: the agent's sandbox never runs the process that holds the key, and the server honors exit 75 only on that path (a shared-mode 75 is measured).Why
The benchmark's own runtime is Harbor 0.6.6 with OpenCode in the task container. Running the same task folders through the
harborEnvironment Server makes the benchmark usable with any sandboxed session harness. The relay agent of earlier revisions of this PR is out.Exit 75 is the smallest signal that keeps judge-service or asset-download outages at verification time out of the score without an upstream Harbor change: plain Harbor sees a missing reward, Gym classifies the code. Without it a missing
reward.jsonis a measured 0.No issue is linked; the benchmark addition was agreed in review on this PR.
Validation
pytest tests/unit_tests/benchmarks/test_agentic_vbench_staging.py tests/unit_tests/benchmarks/test_agentic_vbench_prepare_harbor_env.py tests/unit_tests/data/tasks/harbor tests/unit_tests/test_public_api_facades.py tests/unit_tests/environment tests/unit_tests/tooling: the staging tests drivehub_judge.postagainst a local socket server (dropped before the status line, truncated body, stalled), a 5xx, a truncated 5xx body, a 413 and a token-limit 400 as item errors, the Hub allowance (local work excluded, a call in flight bounded, expiry during backoff), failures keyed by payload and superseded by a final answer, the probes' request shapes and lazy preflight, the video proxy,main()'s success path and its three exit-75 paths, and run the patched repair fetch against a failingcurlstub.environment_servers/harbor/testsin the server's own venv: 725 passed, including the exit-75 test (masked; reward wins; a generic nonzero exit is still a measured 0).python -m benchmarks.agentic_vbench.prepare_harbor_envon the pinned checkout: 100 task folders written, everytask.tomlloads withHarborTaskConfig, 36 judged tasks carry the verifier-only key template, 18 repair verifiers carry the exit-75 guard, 54 steps declare the output artifact, 18materials.jsonrecords.pre-commit run --all-files: clean.uv lock --check: clean. detect-secrets hook with main's baseline: clean (the pinned revision and judge digests carry inline allowlist pragmas; no baseline entries, no conflict with main).Rollout evidence
Oracle run on this head (
gym dataset validateof everysolution/, 100 tasks,--sandbox docker, the overlay, judge protocolnvinference-hybrid):/workspace/materialsblocker)harbor:verifier_timeout(measured 0); 25 judged by the Hub pair, rewards 0 to 0.72Equal-family mean 0.80. What the run found and what it did not:
harbor:invalid_reward: the pinned judges write{"reward": 1.0, "details": {...}}, and the server refused any non-numeric key. Fixed in 7dce1ec (non-numeric keys beside a numericrewardare metrics). Then the 11 repair zeros you predicted onstaging.py:306, fixed in 601f7b9.verifier_seconds248 to 1246 for the 25 that finished, 1834 to 2047 for the 11 killed at the 1800 s budget); the published runs gave the verifier 3x, which the entry cannot express yet (README). The 8 scored zeros have negative rubric penalties (rawdown to -164), i.e. the upstream reference edits fail those rubrics, not an infrastructure failure. No clip crossed the 14 MiB inline budget, so the proxy was not exercised.harbor:verifier_unavailable, masked, none scored 0, with the Hub's message infailure_reason; the 64 other tasks were unaffected. The injected smoke proper (a bad route on one task, a blocked download on one repair task) and the full-setopencode_agentrun withimage_inputare still owed and go here when done.DOCKER_HOST, where the task Dockerfiles do not build (apt cannot drop to its sandbox user in a rootless build). For this run the 100 images were built outside Gym with that one apt setting disabled in the base layer and the tasks pointed at them with[gym.tasks."<id>".environment] docker_imageoverrides in a run-only copy of the entry; the committed entry is unchanged. A Docker host that builds the task images as the entry says would let the next run be the committed path end to end; help finding one is welcome.Compatibility
New benchmark. The
harborserver change adds one masked outcome for an exit code no current verifier uses;verifiedstays false indataset.toml.🤖 Generated with Claude Code