Skip to content

feat(agentic-vbench): add the Agentic-VBench Harbor benchmark entry - #4049

Merged
yfw merged 40 commits into
NVIDIA-NeMo:mainfrom
goelarushi:agentic-vbench
Nov 17, 2026
Merged

yfw merged 40 commits into
NVIDIA-NeMo:mainfrom
goelarushi:agentic-vbench

Conversation

@goelarushi

@goelarushi goelarushi commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

What

Agentic-VBench, the 100 pinned video-editing tasks of PhiloLabs/agentic-vbench (Repair 18, Assembly 18, Sequencing 28, Repurpose 36; revision 410a75fe3d, Apache-2.0), as a task-folder benchmark on the harbor Environment Server.

  • environments/harbor/agentic_vbench/: the committed entry (dataset.toml, README, harbor_sandbox.yaml deployment overlay: judge key as a server secret, separate_verifier_for_artifacts, the harness's image_input, episode cap). The tasks are written into it by the prepare script, not pulled from a Harbor hub.
  • benchmarks/agentic_vbench/: prepare_harbor_env.py (writes the 100 tasks), checkout.py (pinned fetch and inventory), staging.py (per-task adjustments), hub_judge.py (the repurpose rubric judge with both providers on the NVIDIA Inference Hub).
  • harbor Environment Server change (reaches every environments/harbor/* entry): a verifier that exits 75 (EX_TEMPFAIL) without writing a reward is masked with harbor:verifier_unavailable instead of scoring 0, classified from the exit record the wrapper already writes. A written reward and a verifier timeout still win; every other exit without a reward stays a measured 0. Documented in task-folder-verifiers.mdx and the task-folder troubleshooting table.

Per-task adjustments, recorded under the entry's .records/<task>/ with digests: the repurpose judges route through the Hub with a verifier-only ${NVINFERENCE_API_KEY} template, probe each route (in the judge's own request shapes) before their first Hub call, and exit 75 on outages, unusable routes or a spent allowance of time waiting on the Hub; what the agent's output decides never selects that exit (a request the route refuses for its size or length is that item's error, as upstream, and a clip over the inline budget goes as a bounded ffmpeg proxy, recorded in the provenance). The pinned assembly judges (verified by digest) score a non-object solution.json 0 instead of crashing; the repair verifiers retry the golden-asset download longer within a bound and exit 75 when it still fails. The judged and repair steps declare their artifacts (/workspace/output, plus /workspace/materials for repair, which the setup step fills in the agent's sandbox and the repair verifiers read), so with the overlay they verify in a fresh sandbox of the task image: the agent's sandbox never runs the process that holds the key, and the server honors exit 75 only on that path (a shared-mode 75 is measured).

Why

The benchmark's own runtime is Harbor 0.6.6 with OpenCode in the task container. Running the same task folders through the harbor Environment Server makes the benchmark usable with any sandboxed session harness. The relay agent of earlier revisions of this PR is out.

Exit 75 is the smallest signal that keeps judge-service or asset-download outages at verification time out of the score without an upstream Harbor change: plain Harbor sees a missing reward, Gym classifies the code. Without it a missing reward.json is a measured 0.

No issue is linked; the benchmark addition was agreed in review on this PR.

Validation

  • pytest tests/unit_tests/benchmarks/test_agentic_vbench_staging.py tests/unit_tests/benchmarks/test_agentic_vbench_prepare_harbor_env.py tests/unit_tests/data/tasks/harbor tests/unit_tests/test_public_api_facades.py tests/unit_tests/environment tests/unit_tests/tooling: the staging tests drive hub_judge.post against a local socket server (dropped before the status line, truncated body, stalled), a 5xx, a truncated 5xx body, a 413 and a token-limit 400 as item errors, the Hub allowance (local work excluded, a call in flight bounded, expiry during backoff), failures keyed by payload and superseded by a final answer, the probes' request shapes and lazy preflight, the video proxy, main()'s success path and its three exit-75 paths, and run the patched repair fetch against a failing curl stub.
  • environment_servers/harbor/tests in the server's own venv: 725 passed, including the exit-75 test (masked; reward wins; a generic nonzero exit is still a measured 0).
  • python -m benchmarks.agentic_vbench.prepare_harbor_env on the pinned checkout: 100 task folders written, every task.toml loads with HarborTaskConfig, 36 judged tasks carry the verifier-only key template, 18 repair verifiers carry the exit-75 guard, 54 steps declare the output artifact, 18 materials.json records.
  • pre-commit run --all-files: clean. uv lock --check: clean. detect-secrets hook with main's baseline: clean (the pinned revision and judge digests carry inline allowlist pragmas; no baseline entries, no conflict with main).

Rollout evidence

Oracle run on this head (gym dataset validate of every solution/, 100 tasks, --sandbox docker, the overlay, judge protocol nvinference-hybrid):

Family Scored Mean Masked Notes
Assembly 18/18 1.000 0
Sequencing 28/28 1.000 0
Repair 18/18 1.000 0 7/18 before 601f7b9 (the /workspace/materials blocker)
Repurpose 36/36 0.202 0 11 harbor:verifier_timeout (measured 0); 25 judged by the Hub pair, rewards 0 to 0.72

Equal-family mean 0.80. What the run found and what it did not:

  • The first pass scored every assembly and sequencing solution 0 as harbor:invalid_reward: the pinned judges write {"reward": 1.0, "details": {...}}, and the server refused any non-numeric key. Fixed in 7dce1ec (non-numeric keys beside a numeric reward are metrics). Then the 11 repair zeros you predicted on staging.py:306, fixed in 601f7b9.
  • Repurpose: no judge-item errors, no HTTP 4xx/5xx and no exit 75 in the 36 verifier outputs; the probes passed on both routes. The 11 timeouts are the judge's own Whisper transcription of long clips in the fresh verifier sandbox (verifier_seconds 248 to 1246 for the 25 that finished, 1834 to 2047 for the 11 killed at the 1800 s budget); the published runs gave the verifier 3x, which the entry cannot express yet (README). The 8 scored zeros have negative rubric penalties (raw down to -164), i.e. the upstream reference edits fail those rubrics, not an infrastructure failure. No clip crossed the 14 MiB inline budget, so the proxy was not exercised.
  • Outage behaviour, observed rather than injected: on 11-15 the Hub key was over budget (HTTP 422 on the Claude probe) and all 36 repurpose rows came back harbor:verifier_unavailable, masked, none scored 0, with the Hub's message in failure_reason; the 64 other tasks were unaffected. The injected smoke proper (a bad route on one task, a blocked download on one repair task) and the full-set opencode_agent run with image_input are still owed and go here when done.
  • Host: the only Docker endpoint I can reach is a rootless Podman daemon over DOCKER_HOST, where the task Dockerfiles do not build (apt cannot drop to its sandbox user in a rootless build). For this run the 100 images were built outside Gym with that one apt setting disabled in the base layer and the tasks pointed at them with [gym.tasks."<id>".environment] docker_image overrides in a run-only copy of the entry; the committed entry is unchanged. A Docker host that builds the task images as the entry says would let the next run be the committed path end to end; help finding one is welcome.

Compatibility

New benchmark. The harbor server change adds one masked outcome for an exit code no current verifier uses; verified stays false in dataset.toml.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 5, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 6, 2026
@goelarushi goelarushi changed the title Agentic vbench feat(agentic-vbench): add the Agentic-VBench environment, agent and benchmark Oct 22, 2026
@goelarushi
goelarushi requested a review from a team as a code owner November 12, 2026 10:04
Arushi Goel and others added 16 commits November 12, 2026 07:20
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster>
(cherry picked from commit f29bcc6)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster>
(cherry picked from commit 3a20fa1)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster>
(cherry picked from commit d268fb0)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@oci-nrt-cs-001-login-02.cm.cluster>
(cherry picked from commit 285e71c)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
…te Podman environment

(cherry picked from commit 1655d98)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: arushig <arushig@nsc-svg-slurm-1-login-01.nvidia.com>
(cherry picked from commit f18f90e)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
(cherry picked from commit 21e99d5)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
The agent took the model endpoint and id from environment variables and
OpenCode called vLLM directly, so a gym-native run's --checkpoint,
served_model_name, seed and sampling overrides never reached it. The
agent now references policy_model (model_server) and resolves the
per-rollout proxy URL with base_url_for_run(get_server_url(...)), as
opencode_sandboxed_agent does; the proxy rewrites the model name and
applies policy_model.responses_api_models.vllm_model.sampling_overrides
to OpenCode's chat completions. model_base_url/model_id stay as an
escape hatch for launchers that serve the model themselves.

Also: derive the episode root from the run's output_jsonl_fpath, emit
equal_family_reward (reward * 25 / family size) and per-family reward
keys, and compute mean/equal_family_reward (the leaderboard's equal mean
of the four family means) in compute_metrics so Gym's aggregate carries
the headline metric next to the task-weighted mean/reward. An episode
directory a killed job left without gym_result.json is archived and the
task rerun instead of aborting the resumed run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
(cherry picked from commit 86e3284)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
A gym-native driver has only the Gym checkout, its venv and the cluster's
cache mount. The benchmark now needs nothing else on the node:

- harbor==0.6.6 is installed into the agent venv (requirements.txt), so
  harbor_python defaults to the agent's own interpreter;
- prepare.py and the agent fetch PhiloLabs/agentic-vbench at the pinned
  revision into <HF_HOME>/../agentic_vbench with a shallow fetch by SHA
  (AGENTIC_VBENCH_ROOT / AGENTIC_VBENCH_REPO_URL still override);
- a static Docker client with the compose plugin is downloaded into the
  cache when `docker` is not on PATH (Harbor's DockerEnvironment shells
  out to `docker compose`);
- the remote rootless Podman backend is discovered through
  backend_dir/client.env (DOCKER_CERT_PATH rebound to that directory), the
  agent waits for it, probes `docker info` once at start, and removes an
  episode's containers when its runner fails or is cancelled;
- the hub judge key falls back to the driver's INFERENCE_API_KEY, so no
  credentials file is required;
- harbor_runner takes --backend, --judge-protocol, --max-turns,
  --verifier-timeout-multiplier and --model-timeout-ms from the agent
  configuration (environment variables remain as fallbacks) and sets
  OpenCode's provider timeout, which must cover a whole completion
  because Gym's proxy replays the stream once it is complete.

config.yaml exposes every knob as an agentic_vbench_* selector for the
launcher's run overrides.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
(cherry picked from commit 2ced983)
Signed-off-by: Arushi Goel <arushig@nvidia.com>
…nt Hub 4xx errors

The Hub judge wrapper rejected the whole verification after repeated failures of any
kind. For a 4xx that persists across retries the error is about this request (observed:
Gemini refusing an empty audio extract), and the original judge, using the native SDK,
catches that exception and scores the rubric item as a judge error while the rest of
the verification continues. Raise such errors to the judge the same way; keep rejecting
on transport (5xx, 408, 429, connection) and authentication failures so infrastructure
problems never turn into silent zero scores.

Signed-off-by: Arushi Goel <arushig@nvidia.com>
The primary OpenCode agent can delegate to subagents through its task tool, and the
built-in subagents are available by default, so every run so far was 'default'. Add
opencode_subagents to the agent config: 'off' removes the task tool and disables the
built-in subagents (primary agent only); 'on' keeps them and registers a media-analyst
subagent whose description makes delegation an explicit option. Task prompt and the
primary agent's system prompt are unchanged in every mode; the mode is recorded so
shards with different policies are never merged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Retry Hub judge calls that fail with 5xx/408/429 or transport errors up to 8 times with a
60 s-capped backoff (~3 min per item) instead of 5 quick attempts; request-scoped 4xx keep
the short budget. Bedrock-backed Claude returned intermittent 503s for ~40 min and rejected
otherwise complete Repurpose verifications. Scoring is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Agent setup downloads Node and OpenCode inside the task container; during registry
slowdowns the 360 s default expires before the agent starts and the trial is lost as
infrastructure. Setup time is not part of the agent budget, so scores are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
A task root carrying custom_tasks.json is treated as a set of generated tasks in
the benchmark layout (tasks/agentic_vbench_<family>/<task>): ensure_checkout
returns it as-is and inventory skips the pinned-revision, clean-tree and
100-task completeness checks. benchmark_revision becomes custom:<marker sha>.
The pinned benchmark path is unchanged. Used for Curiosity SDG trace generation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor reports task.toml's name as <namespace>/<task>. Compare only the task
component so generated tasks (e.g. curiosity-sdg/<task>) verify like the
benchmark's agentic-vbench/<task>.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor ends each trial with 'docker compose down --rmi all', so every trial
rebuilds its task image. keep_task_images=true (env
AGENTIC_VBENCH_KEEP_TASK_IMAGES via the benchmark config) sets environment.delete
to false so repeated trials of a task (trace generation) reuse one build and
parallel shards no longer race on image deletion. Benchmark default unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
@goelarushi goelarushi changed the title feat(agentic-vbench): add the Agentic-VBench environment, agent and benchmark feat(agentic-vbench): add the Agentic-VBench Harbor benchmark entry Nov 15, 2026
@goelarushi
goelarushi marked this pull request as draft November 15, 2026 09:41
@goelarushi
goelarushi marked this pull request as ready for review November 15, 2026 10:35
goelarushi and others added 2 commits November 15, 2026 05:35
…materials

The repair verifiers read /workspace/materials, which the task's setup step copies
from the image's /baked into the agent's sandbox; a fresh verifier sandbox has only
the declared artifacts, so 11 of the 18 oracle runs scored 0 with "missing source.mp4".
stage_materials now declares /workspace/materials beside /workspace/output (a few MB
per task). The judged tasks read their clip from /baked and are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
Harbor keeps reward.json as written and the benchmark reads `reward`; the server
refused the whole file when any key was not a number, so the Agentic-VBench assembly
and sequencing judges, which write a nested `details` block beside the score, were
measured 0 (harbor:invalid_reward) on correct oracle solutions. With a numeric
`reward`, non-numeric keys are now left out as metrics; without one every key is still
a component and must be a number.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. The exit-75 rework follows the design closely:

  • the marker file is gone;
  • _score classifies from the exit record;
  • a written reward and the verifier timeout both win;
  • every other missing reward stays a measured 0.

The Hub judge's new transport handling and the socket-server tests are solid, and every fix in this round has a test that fails when the fix is reverted.

What we reviewed: head f7b60fbfbc, the changes since 7de55d2f6b, against the benchmark-review checklist (Harbor format). We also read the pinned upstream tasks at 410a75fe3d.

Earlier threads: 24 were open.

  • Resolved: 17 that this head fixes, each with a reply.
  • Still open, with a reply: 7.
    • the two evidence threads;
    • the shared-mode part of the design thread;
    • two Hub-judge gaps: a truncated error body and the soft deadline;
    • the size-rejection thread;
    • the separate-verification thread.

Overall: HOLD.

  • One blocker: in the separate sandbox, 11 of the 18 repair verifiers can't find their inputs, so they score 0 on every rollout.
  • The agent's output can still steer the separate verifier into exit 75, through its size and its length.

Every point is inline.

Merge state: no conflicts with main. CI has not run on this head yet.

Verified on a Linux workstation:

  • the two benchmark test files: 28 passed, 1 skipped;
  • environment_servers/harbor/tests: 725 passed;
  • 150 repeated runs of the socket tests, including under load, with no flake;
  • each claimed fix, reverted, fails its test;
  • a prepare of the 100 tasks.

Not verified: any Docker or real-model run, and the Hub's request-size limit.

Comment thread benchmarks/agentic_vbench/staging.py Outdated
"original_sha256": _file_digest(verifier),
"staged_sha256": _file_digest(target),
}
(staged / "task.toml").write_text(_declare_output_artifact(text))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarks/agentic_vbench/staging.py:306

Blocker: in the separate verifier, 11 of the 18 repair tasks score 0 on every rollout, the oracle included.

This follows from our own claim last round that the verifier needs only /workspace/output. Sorry about that.

Affected tasks:

  • exp-color-shot-{gobelins,v3-s1,visit-korea}
  • exp-content-cut-{mkbhd,wsj}
  • exp-deblur-{gaussian-mkbhd,motion-f1}
  • exp-disfluency-interview-{3,4}
  • exp-glitch-dup-long
  • exp-sr-4x-shot

Could stage_materials rewrite the judge arguments from /workspace/materials/ to /baked/input/ in the staged test.sh?

  • That is the image's own copy, the one setup.sh copies from. It needs no extra copy, an honest agent's inputs are byte-identical, and the judge then compares against inputs the agent never touched.
  • Recording the rewrite in materials.json and the README table would keep it traceable.
  • A staging test could assert that no step declaring an artifact still names /workspace/materials.

The oracle validate would confirm the fix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 601f7b9: stage_materials now declares /workspace/materials beside /workspace/output as the step's artifacts (a few MB per task), so the fresh verifier sandbox receives what the agent's setup.sh copied from /baked/input; the judged tasks read their clip from /baked and are unchanged. The README says the folder is filled by setup.sh, not the image. Oracle: repair went from 7/18 at 1.0 (the 11 zeros were exactly missing source.mp4 / corrupted.mp4 at /workspace/materials) to 18/18 at 1.0.

Comment thread environment_servers/harbor/verifier.py Outdated
)
if rewards is None:
exit_record = parse_exit_record(logs_dir)
if exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

environment_servers/harbor/verifier.py:1216

Note (the open part of the design thread): exit 75 is honored in shared mode too.

  • The comment above and the docs say the code can be trusted only when test.sh runs separately, and they warn about shared mode. _score itself doesn't know the mode.
  • This entry's 46 assembly and sequencing tasks run set -euo pipefail; python3 /tests/judge.py in the agent's sandbox. An agent that leaves a python3 shim exiting 75 turns its own 0 into a masked row.
  • Tasks in the other environments/harbor/* entries verify in shared mode unless their task.toml asks otherwise, so the same applies there.
  • This widens an existing gap rather than opening a new one: a root agent can already force a masked verifier_error by removing timeout (environment_servers/harbor/verifier.py:88).

We'd still lean toward enforcing it. _score would take a keyword separate: bool: True from _verify_separately, False from _run_shared_verifier, the two calls multi-step turns also use. In shared mode the row would stay a measured 0, as on main, with the ignored 75 noted in the reason. The cost is that a real outage in shared mode keeps scoring 0, which is today's behaviour. Your call; either way, please pin the choice with tests.

Tests that would pin it:

  • a shared-mode case and a separate-mode case in test_exit_75_without_a_reward_is_masked_not_scored;
  • a provider timeout with a 75 exit record. Today, letting 75 beat the timeout still passes all 725 server tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enforced in 27dc4c1 as proposed: _score(..., separate: bool), passed True by _verify_separately and False by _run_shared_verifier, i.e. the path that ran this turn's test.sh, so a multi-step task with mixed turns gets each one right. Shared mode: no reward plus 75 falls through to the ordinary missing/invalid-reward result (measured 0) with "test.sh exited 75 in the agent's sandbox, where the code is not trusted" in the reason and verifier_return_code: 75 kept for counting. Separate mode: masked as before. The test covers separate (masked), shared (measured), valid-reward and timeout precedence and the ordinary nonzero exits; the docs and the troubleshooting row say the code is honored on the separate path only.

| extras
)
if rewards is None:
exit_record = parse_exit_record(logs_dir)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

environment_servers/harbor/verifier.py:1215

Note: rewards is None also covers an invalid written file, so exit 75 plus a malformed reward.txt is masked. The docs say "a written reward always wins", and the outcome table scores an invalid file 0 (harbor:invalid_reward).

Masking seems the right call here: only the verifier writes that file, and it then reports it could not grade. Could the docs say "a valid written reward wins", with a test to pin it? No test covers either behaviour yet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on masking. The docs now say a valid written reward wins and that a malformed file beside the code is the verifier's own and leaves the run masked (task-folder-verifiers.mdx, the outcome table's first two rows carry the exception). test_exit_75_without_a_reward_is_masked_only_on_the_separate_path pins malformed reward.txt plus 75 as masked (separate), valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as harbor:invalid_reward, and the shared-mode case (27dc4c1).

# A verifier timeout wins over the code, and in shared mode the agent's sandbox runs test.sh, so the code is
# trustworthy only for a task verified separately.
VERIFIER_UNAVAILABLE_EXIT_CODE = 75
VERIFIER_UNAVAILABLE_KIND = "harbor:verifier_unavailable"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

environment_servers/harbor/verifier.py:100

Question on the kind. Recovery keeps a masked row of a namespaced kind as the final result (nemo_gym/rollouts/rollout_recovery.py:456-478).

After one Hub outage, gym eval resubmit --recover stops with "masked … the run is not complete" (nemo_gym/orchestration/recovery.py:1590-1596). Only a fresh run can measure those rows.

That matches the other harbor:* masked kinds, so it isn't a regression. It does make "masked" mean "the run ends FAILED" until the bounded server-side re-run lands.

The registered verifier_error would give the operator a way out. A Harbor row keeps the agent's answer, so it would not be retried automatically, but --recover --rerun-masked-without-artifact --rerun-reason … could rerun it. That gate is better than an automatic retry while some 75s still depend on the agent's output. You would keep "exited 75" in the reason and verifier_return_code: 75 for counting.

Either option works: switch the kind, or have the troubleshooting row say such a run ends FAILED and needs a fresh run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the namespaced kind and documented the consequence (27dc4c1): the troubleshooting row and the verifier page now say that gym eval resubmit --recover keeps the row masked and stops with "the run is not complete", and that a fresh run measures it once the service is back. I preferred this to switching to verifier_error, whose row above means "bash or timeout missing"; one kind with two meanings would make the per-kind counts harder to read, and the bounded server-side re-run is the follow-up that gives the operator a real way out.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated

def _size_rejection(code: int, detail: str) -> bool:
"""A request the Hub refuses for its size (inline media): infrastructure, not a judge error of this item."""
return code == 413 or (code == 400 and any(w in detail.lower() for w in ("too large", "exceeds", "payload size")))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarks/agentic_vbench/hub_judge.py:65-67

Risk: the agent decides the request size, so its output can turn a failure into a masked row.

  • HubGemini inlines the agent's whole repurpose.mp4 as base64 (benchmarks/agentic_vbench/hub_judge.py:172-190), where upstream uploads it through the Files API.
  • An overlong or high-bitrate output gets a 413, then exit 75 and a masked row. Say, the untrimmed 59-358 MB source.
  • The bare "exceeds" also matches Gemini's token-limit 400 ("input token count … exceeds the maximum"). That one is driven by output length, and upstream treats it as an item error.

Our suggestion last round to treat 413s as "could not grade" missed that the agent picks the size. Sorry for steering you there.

A fix in three parts:

  • Bound only the video, since inlining is this shim's own deviation. Above a byte budget, send an ffmpeg proxy (at most 720p, keep the audio, bitrate = budget / duration); below it, send the bytes unchanged. Record the settings in the provenance.
  • Then drop the 400-text match. A persistent size rejection on the audio or Claude items (upstream's own request shapes) becomes the item's request_error, as upstream.
  • In the smoke, send the largest legitimate request: a full 250-frame pillar, a long mp3, and a proxy at the budget.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e, in the three parts you list. (1) Only the video is bounded: HubGemini.upload sends a clip over VIDEO_INLINE_BUDGET_BYTES (14 MiB, below Gemini's 20 MB inline request limit after base64) as an ffmpeg proxy, scale to at most 720p high, audio kept at 96 kbps, video bitrate = 0.92 x budget / duration, made once per file; below the budget the bytes go unchanged. The settings are recorded in every result as judge_media_transforms (original and proxy bytes, duration, max height, bitrates) beside judge_video_inline_budget_bytes. (2) The 400-text match and the 413 special case are gone: both are this request's error, raised to the judge as upstream. The claims in staging.py, the overlay, both READMEs and the PR body now say what the agent cannot do (run the verifier, read the key) and that the exit code is decided from the Hub's answers and the download's outcome, never from the output. (3) The proxy test uses a generated 2 s clip and is skipped where ffmpeg is absent (the CI container; the task images have it); the no-ffmpeg fallback has its own test. The largest-request smoke is owed with the full run.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated
error = "unknown"
request_error = False
for attempt in range(ATTEMPTS):
if time.monotonic() - _STARTED > DEADLINE_SECONDS:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarks/agentic_vbench/hub_judge.py:85-88

Risk: the deadline is soft, and it also counts time the agent's output causes.

  • It is checked only before an attempt, and urlopen(timeout=…) bounds each socket read, not the call. A trickling response held one call for 17 s against a 1 s deadline and a 0.5 s call timeout.
  • _STARTED is set at import, after test.sh's find / and its copies.
  • After the deadline, the original judge keeps doing local work (Whisper per audio item, frame extraction).

The result goes both ways. A slow but healthy Hub can still reach the 1800 s kill, a measured 0. A very long output can push local time past 1500 s and get masked.

A possible shape:

  • Have the staged test.sh export its start time.
  • Run a threading.Timer that calls os._exit(75) (since sys.exit in a thread doesn't end the process) at the budget minus a margin for aggregate.py.
  • Exit 75 only when time spent in Hub calls accounts for the delay; otherwise let the run reach the verifier timeout.
  • Pass timeout=min(CALL_TIMEOUT_SECONDS, remaining) to each call.

Neither a deadline passing mid-retry nor a trickling response has a test yet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworked in 53da51e along the lines you describe, with one difference. Instead of a wall-clock timer, the budget is now an allowance of time spent waiting on the Hub (HUB_WAIT_SECONDS, 1200 s): _call runs the request on a thread and the caller stops waiting at min(CALL_TIMEOUT_SECONDS, remaining) (a result after the bound is dropped), the backoff is bounded by what is left and counts as waiting, and nothing else counts, so the import-time _STARTED is gone and local work (Whisper, frame extraction, the proxy encode) only ever runs the verifier into its own timeout, a measured 0. Exit 75 therefore happens only when Hub waiting accounts for the delay, which is what the timer-plus-check would have decided; the staged test.sh needs no start export. Tests: a stalled response with 0.4 s left ends in about 0.4 s, not the 2 s socket timeout; expiry during the backoff after a 503 makes exactly one attempt; 1.2 s of local work before a healthy call records nothing; a spent allowance makes no call.

},
{"anthropic-version": "2023-06-01"},
),
"Gemini": (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarks/agentic_vbench/hub_judge.py:212-226

Note: the probes don't send what the judge sends.

  • Real Gemini calls carry thinkingConfig and temperature. Real Claude calls carry a system block with cache_control.
  • A route that accepts the "ping" but answers 400 to every real request passes preflight. Every item then scores 0 as a judge error, the case the preflight exists to catch.

Could the probes carry those fields? Two more suggestions:

  • Run the preflight lazily, on the first Hub call. Then an output the original judge rejects without calling the Hub (under 10 KB, under 2 s, unreadable) still scores 0 during an outage instead of being masked.
  • In the smoke, show both probes and one real item passing on each protocol.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e. The probes are built by probes() in the judge's shapes: Claude with a system prompt and a cache_control block on the user content, Gemini with gemini_generation_config(0.0, 16), the same builder every real call uses (thinking level and temperature included). The preflight is lazy: post() runs it before its first real call, once, so an output the judge rejects locally scores 0 during an outage instead of being masked, and main() no longer probes up front. Tests: the probe payloads' fields, the probes going out through the first real post() and not again, and a judge that makes no Hub call running without probes. The smoke showing both probes and a real item on each protocol is owed with the full run; the Hub key cannot do it today (budget exhausted, see the dataset.toml thread).

sandbox or a reward that could not be copied out, set `mask_sample`: the rollout stays in `rollouts.jsonl` but is
left out of the score.

A verifier that cannot grade, because a judge service or an asset download it depends on is unavailable, exits

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fern/versions/latest/pages/data/task-folders/task-folder-verifiers.mdx:55

Note: the page now contradicts itself.

The table's caption says "These rows count in the score", so the simplest fix is to qualify L34-35 ("…except exit 75 without a reward, below"). Alternatively, add a row marked as masked.

It would also help to say that separate mode is necessary but not sufficient: nothing derived from the agent's artifact (its size, its duration) may decide the exit code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 27dc4c1: the opening statement now says the exit code never changes a valid reward file's result and that, without one, every exit is a measured 0 except exit 75 from a separately run verifier; the table's first two rows carry that exception; "a valid written reward wins"; and the paragraph adds that separate mode is necessary, not sufficient, with the artifact's size, duration and content named as things that may not decide the code.

# to null, which the server's `secrets` table rejects, so the server does not start without a key and no repurpose
# task can be scored 0 for want of one. The judged (repurpose) and repair tasks declare `/workspace/output` as their
# step's artifact; `separate_verifier_for_artifacts` grades them in a fresh sandbox of the task image, so the agent's
# sandbox never runs the verifier that holds the key, and cannot make it exit 75 ("could not grade"). Assembly and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

environments/harbor/agentic_vbench/harbor_sandbox.yaml:11-14

Note (docs): a few claims need adjusting once the points above are settled.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three adjusted (2478175, 53da51e). The "cannot make it exit 75" wording in staging.py, the overlay, benchmarks/agentic_vbench/README.md and the PR body now says what the agent cannot do (run the verifier process, read the key) and that the verifier derives the code from the Hub's answers and the download's outcome, never from the output, which with the size and allowance changes is now true of the code. The README says /workspace/materials is filled by each step's setup.sh from /baked in the agent's sandbox, which is why the repair verifiers receive it as an artifact (601f7b9). The 64 unjudged tasks have a keyless recipe: AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing plus the new harbor_sandbox_no_judge.yaml, the same overlay without the secret.

"handler,expected",
[
# A reset instead of the EOF is possible when the peer closes first; both are transport failures.
(_drop_before_status_line, ("RemoteDisconnected", "ConnectionResetError")),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/unit_tests/benchmarks/test_agentic_vbench_staging.py:510

Tests: every fix in this round has a test that fails when the fix is reverted. Nice work; 150 runs under load showed no flake.

A few branches stay green when reverted:

Here, truncated-body yielded IncompleteRead every time, so the looser expected tuple could narrow to it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Each of the four now has a test that fails when the branch is reverted (53da51e): two payloads on one route with one later superseded (test_hub_post_keys_failures_by_payload_and_lets_a_final_answer_supersede_them); the _size_rejection 400 branch is gone with the function, and the same test pins a 413 and a token-limit 400 as item errors; the mid-retry expiry (test_hub_allowance_counts_waiting_on_the_hub_not_local_work, one attempt then the allowance message); and the continuation-line loop (test_materials_fetch_guard_follows_a_multi_line_curl_command). The truncated-body tuple is narrowed to IncompleteRead.

@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test f7b60fb

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

The Hub shim still lets a transport failure become a measured zero and lets output-size rejections select masking. The generic exit-75 branch also trusts shared verifiers.

I inspected all 16 changed files and traced the affected scoring and recovery paths. Valid parsed rewards still take precedence, and verifier timeouts are checked before exit 75.

This was a static review. I did not run tests, Docker builds or model evaluations. The pinned external tasks are not mounted, so I could not independently confirm the reported missing-input failure in 11 Repair tasks. The entry README confirms that oracle and model-run validation are still missing.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated
"""Short, single-line provider status message; empty when the body is not a JSON error."""
try:
body = json.loads(exc.read().decode(errors="replace"))
except (ValueError, OSError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER: a truncated HTTP error body bypasses outage handling. In provider_error_message, exc.read() can raise http.client.IncompleteRead, which this handler does not catch. The call runs inside post()'s exception handler, so its exception bypasses the retries and FAILURES. During preflight it also escapes the RuntimeError handler, leaving an ordinary missing reward rather than exit 75.

Could this handler include http.client.HTTPException?

Suggested change
except (ValueError, OSError):
except (ValueError, OSError, http.client.HTTPException):

This keeps the HTTP status available for classification even when its diagnostic body is unusable. Preserve ordinary request-scoped 400 handling, short single-line diagnostic messages, and the existing handling of truncated successful responses.

Proposed acceptance: add a socket response with status 503, Content-Length: 400, and a short body. After retries, post() should raise RuntimeError and record one failure; through preflight, main() should exit 75. Keep the existing assertions that persistent ordinary 400s leave FAILURES empty and truncated 200 responses are recorded as transport failures. I have not executed these checks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e: provider_error_message catches http.client.HTTPException too, keeping the status for classification. Acceptance as proposed: the socket test's truncated-503-body case (503, Content-Length: 400, short body) makes post() raise RuntimeError after the retries and record one failure; test_hub_preflight_exits_75_on_a_truncated_error_body runs preflight() through the real post() against it and gets exit 75. Persistent ordinary 400s still leave FAILURES empty and the truncated-200 case is still a recorded transport failure.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated

def _size_rejection(code: int, detail: str) -> bool:
"""A request the Hub refuses for its size (inline media): infrastructure, not a judge error of this item."""
return code == 413 or (code == 400 and any(w in detail.lower() for w in ("too large", "exceeds", "payload size")))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER: answer-size errors can remove failed work from scoring. HubGemini inlines the entire agent-produced file. Every 413, and a 400 containing the bare word exceeds, enters FAILURES. For example, {"error":{"message":"input token count exceeds the maximum"}} makes main() exit 75, masking the whole verification. Separate execution protects the verifier process, but does not make the answer's size infrastructure-owned.

Could post() keep answer-dependent token/payload rejections request-scoped, while the Gemini media path provides a validated transport for supported legitimate clips? Removing the classification alone is incomplete if legitimate clips still exceed this shim's inline limit. The invariant is that oversized or overlong answers cannot select masking, while supported answers remain gradable. The mounted source does not establish a supported upload API or a safe evidence-preserving proxy budget, so I cannot supply a complete transport patch. Record any media transformation in the protocol provenance and correct the claims that the agent cannot influence exit 75.

Proposed acceptance: persistent output-size 413 and token-limit 400 responses must not select exit 75 solely because of the answer. Exercise the largest supported video under both protocols and show grading succeeds. Preserve accepted media bytes and rubric evidence, plus masking for authentication failures, 408/429, 5xx and dropped connections. Provider limits and these checks were not verified here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e. Persistent 413s and token-limit 400s are request-scoped: raised to the judge as the item's error, never in FAILURES, so they cannot select exit 75; the test pins both. The Gemini media path bounds the inline deviation: a clip over 14 MiB goes as an ffmpeg proxy (at most 720p, audio kept, bitrate from the budget and duration), recorded in every result as judge_media_transforms; supported clips under the budget go unchanged. Without ffmpeg the file goes as it is and the route decides, as an item error. The claims that the agent cannot influence exit 75 are reworded everywhere to what the agent cannot do and what decides the code.

Comment thread environment_servers/harbor/verifier.py Outdated
)
if rewards is None:
exit_record = parse_exit_record(logs_dir)
if exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: exit 75 is trusted in the agent's shared sandbox. Both _run_shared_verifier and the separate path call this classifier. A shared task whose trusted script invokes an agent-modifiable executable can therefore return 75 without a reward and become masked instead of the measured zero it received on base. The new test exercises this shared-mode behavior. The documentation warns about it, but the classifier does not enforce the stated infrastructure-only boundary.

Could _score take a required keyword separate: bool, passed as True by _verify_separately and False by _run_shared_verifier, and use this guard?

            if separate and exit_record is not None and exit_record[0] == VERIFIER_UNAVAILABLE_EXIT_CODE:

Use the execution path, not the task's overall mode: multi-step tasks can mix shared and separate turns. Shared 75 should fall through to the ordinary missing/invalid-reward result. This closes this new masking path; it does not claim to make shared verification a security boundary.

Proposed acceptance: no reward plus 75 is measured in shared mode and masked in separate mode, including artifact-forced separate turns. An explicitly shared turn beside a separate turn must stay measured. Retain valid reward precedence, timeout precedence even with a 75 exit record, and measured zero for ordinary nonzero exits. These are proposed checks, not executed results.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as proposed in 27dc4c1: _score(..., separate: bool) from the execution path (_verify_separately True, _run_shared_verifier False), and the guard if unavailable and separate:. Shared 75 falls through to the ordinary missing/invalid-reward result with the ignored code in the reason and verifier_return_code kept. The test covers no reward plus 75 measured in shared mode and masked in separate mode, valid-reward precedence, the timeout record winning (existing test), and measured zeros for ordinary nonzero exits.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated
request_error = False
break
try:
with urllib.request.urlopen(request, timeout=CALL_TIMEOUT_SECONDS) as response:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: the deadline does not bound a request already in progress. post() checks the deadline only before starting an attempt. An attempt started with one second left still receives the full 120-second timeout, and a response completed after the deadline is accepted without another check. The configured deadline therefore does not guarantee the promised finishing margin before the verifier's 1800-second kill.

Could request supervision enforce the remaining allowance across opening and consuming the response, and bound retry sleeps by that allowance? Passing min(CALL_TIMEOUT_SECONDS, remaining) is necessary but does not by itself provide whole-operation supervision. Keep this bound scoped to Hub work so local media processing does not become an infrastructure failure. The cancellation mechanism needs an implementation decision, so there is no safe one-line replacement to suggest.

Proposed acceptance: start a controlled request before the deadline and delay completion beyond it; verify that the configured bound ends the Hub operation rather than accepting a late success. Also cover expiry during backoff. Preserve successful requests within the allowance, recovery after transient failures, and no new request when the Hub allowance is already exhausted. I did not execute these cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e: _call runs each request on a thread and the caller waits at most min(CALL_TIMEOUT_SECONDS, remaining) for the whole operation (open plus body); a late result is dropped, the time waited counts against the allowance, and the retry sleep is bounded by what is left. Tests: a stalled server with 0.4 s left ends in a TimeoutError in about 0.4 s (not the 2 s socket timeout) with one failure recorded and the allowance spent; expiry during the backoff makes one attempt; a healthy call within the allowance succeeds; a spent allowance makes no request.

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated
error = "unknown"
request_error = False
for attempt in range(ATTEMPTS):
if time.monotonic() - _STARTED > DEADLINE_SECONDS:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: local answer processing can be classified as a Hub outage. The clock starts at module import, not when Hub waiting starts. Any local work before a later post() consumes it, including the shim's file reading and base64 encoding. Once that time exceeds the allowance, even a healthy Hub is never contacted; the request is recorded as unavailable and the verification is masked.

A minimal proposed reproduction is a one-second judge allowance, more than one second of local preparation after import, then an immediately successful Hub endpoint. Could the infrastructure allowance track time spent waiting on Hub operations separately from artifact-driven local work? Local work alone must not select exit 75; it should stay subject to the task's ordinary verifier timeout. Keep the hard in-flight Hub bound requested separately.

Proposed acceptance: local preparation alone must not populate FAILURES or mask an otherwise healthy request; genuinely exhausted Hub waiting must still report unavailable. Update the existing expired-deadline test to set up exhausted Hub allowance explicitly rather than equating all time since _STARTED with an outage. Preserve the server's measured-zero verifier-timeout outcome. These checks were not run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e: the allowance (HUB_WAIT_SECONDS) accumulates only time spent in _call and in the backoff between attempts, so local work before or between calls (file reading, base64, the proxy encode, the judge's own Whisper and frame extraction) never consumes it; the import-time clock is gone. Your reproduction is the test: 1.0 s allowance, 1.2 s of local waiting after import, then a healthy endpoint, which succeeds with FAILURES empty; the old expired-deadline test now sets the Hub waiting itself to the allowance and gets "allowance ... is spent" with no request made. Local work still ends in the server's verifier timeout, a measured 0.


## Scoring and validation status

The reward is the task's own `reward.json`, 0..1. The leaderboard's equal-family mean is `reward x 25 / family size`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK: the headline metric recipe mixes row normalization with family averaging. Multiplying by 25 / family size is correct when averaging all 100 task rows, not when averaging four family means. Following the latter reading gives (25/18 + 25/18 + 25/28 + 25/36) / 4 ≈ 1.0913 when every reward is 1. The measurement contract repeats the same recipe.

Could both locations say:

The leaderboard score is (mean_repair + mean_assembly + mean_sequencing + mean_repurpose) / 4, using the unscaled family means. Equivalently, over the complete 100-task set, average reward_i * 25 / size_of_family_i across the 100 task rows.

Keep the distinction from the server's task-weighted metric and the restriction against reporting a subset as the full benchmark score. Do not change per-task rewards or silently impute masked rows as zero.

Proposed acceptance: a complete synthetic set with every reward 1 yields 1.0. Only Repair scoring 1 yields 0.25; only Sequencing scoring 1 also yields 0.25. Both formulas must agree. These are proposed arithmetic checks, not executed tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in both places (2478175), in your words: the leaderboard score is (mean_repair + mean_assembly + mean_sequencing + mean_repurpose) / 4 with the unscaled family means, equivalently the mean of reward_i x 25 / size_of_family_i over the complete 100 rows; a subset is not the benchmark score and a masked row is not a 0. The two formulas agree on your three checks (all ones gives 1.0; Repair alone or Sequencing alone at 1 gives 0.25).

Comment thread benchmarks/agentic_vbench/hub_judge.py Outdated
),
"Gemini": (
f"/v1beta/models/{os.environ['AVB_JUDGE_GEMINI_MODEL']}:generateContent",
{"contents": [{"role": "user", "parts": [{"text": "ping"}]}], "generationConfig": {"maxOutputTokens": 1}},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE, suggestion: probe the Gemini controls used by real calls. Every generate_content() request carries thinkingConfig, and configured requests carry temperature; this probe carries neither. It therefore does not validate those shared protocol settings. A route that accepts the ping but rejects the configured thinking level would pass preflight and then turn that protocol incompatibility into item errors.

Could preflight reuse the real generation-config construction, with an appropriate small output budget? Keep evidence-dependent errors request-scoped rather than promoting every real-request 400 to an outage. I have not established that either configured route currently rejects these fields, so this is a coverage improvement, not a confirmed provider failure.

Proposed acceptance: a fake endpoint that accepts plain requests but rejects the configured thinking field should fail preflight with exit 75. An accepting endpoint should permit judging, and an invalid-media 400 after a successful probe should remain an item error. Preserve the configured thinking level and temperature in real requests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 53da51e: the Gemini probe's generationConfig comes from gemini_generation_config(0.0, 16), the builder every real call uses, so it carries the configured thinking level and temperature with a small output budget; the Claude probe carries system and a cache_control block. A route that accepts a bare ping but refuses those fields now fails preflight with exit 75, and an invalid-media 400 after a passing probe stays an item error (request-scoped 4xx handling unchanged). Test: the probe payloads' fields and the probes going out through the first real call.

sandbox or a reward that could not be copied out, set `mask_sample`: the rollout stays in `rollouts.jsonl` but is
left out of the score.

A verifier that cannot grade, because a judge service or an asset download it depends on is unavailable, exits

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: align the reward-file contract with the new branch. The page still says the exit code does not affect the score, and its outcome table says every missing or malformed reward is measured zero. The new paragraph introduces an exception but says any written reward wins. In _score, parsing a malformed file yields rewards is None, so a non-timed-out exit 75 masks it too.

Could the opening statement say that exit codes do not change a parsed reward file's result, qualify the missing/malformed rows with the exit-75 exception, and change “A written reward” to “A valid written reward”? Keep the distinction that parsed finite components whose sum overflows still produce harbor:invalid_reward; that path does not enter the new branch. Apply the same wording to the troubleshooting row.

Proposed acceptance: pin malformed reward.txt plus 75 as masked in separate mode, valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as measured invalid reward, and timeout precedence over 75. Inspect the revised text against those outcomes. These tests were not executed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aligned in 27dc4c1: the opening statement says the exit code never changes a parsed, valid reward file's result and names the one exception; the first two table rows carry it; "a valid written reward"; the overflow-sum path stays harbor:invalid_reward (unchanged code, outside the new branch). The troubleshooting row has the same wording. The test pins malformed reward.txt plus 75 as masked in separate mode, valid reward 1 plus 75 as measured 1, malformed reward plus an ordinary exit as measured invalid, and timeout precedence over 75 (existing test).

| `harbor:missing_reward` | `test.sh` wrote neither `reward.txt` nor `reward.json`. A Python verifier that raised lands here. | Yes, as 0 |
| `harbor:invalid_reward` | The reward file could not be read as a finite number. | Yes, as 0 |
| `harbor:verifier_timeout` | `test.sh` ran past its timeout without a reward. | Yes, as 0 |
| `harbor:verifier_unavailable` | `test.sh` exited 75 (`EX_TEMPFAIL`) and wrote no reward: a judge service or asset download it depends on was unavailable. Trustworthy only when the task verifies separately. | No, masked |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE, suggestion: document what recovery does with this masked kind. This new outcome is namespaced and unregistered. _masked_evaluation_side_kind excludes it from recoverable infrastructure rows, so default recovery keeps it masked. The recovery planner then stops with “the run is not complete.” “No, masked” does not tell an operator that retrying after the service recovers will leave the row unchanged.

Could this row or the adjacent fix paragraph state that --recover does not rerun this outcome and explain the supported way to remeasure it? If no selective rerun is supported for this kind, say that a fresh run is needed. This can be a documentation change; it need not introduce automatic regeneration or a server-wide retry mechanism here.

Static acceptance: the documented recovery action should match the registration guard and planner stop above. Preserve masking rather than zero-imputation, and keep the explicit statement that the server does not rerun a masked verification.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 27dc4c1: the row now says that gym eval resubmit --recover keeps the row masked and stops with "the run is not complete", and that a fresh run is needed to measure it once the service is back; the verifier page says the same. Masking rather than zero-imputation and the statement that the server does not rerun a masked verification are kept.

Both commands take the deployment overlay: it carries the judge key, the episode cap, separate verification of the
judged and repair tasks and the harness's `image_input`, which the tasks need for the model to see the frames it
extracts. The key is required to start the server (below), so Repurpose can only be scored with access to the NVIDIA
Inference Hub; the other 64 tasks need no credentials.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE, suggestion: give the other 64 tasks a keyless run recipe. The README says they need no credentials, but both documented commands use an overlay whose secret resolves to null without a Hub key. The server's secrets: dict[str, str] rejects that configuration even after preparation selects only Repair, Assembly and Sequencing.

Could you document AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing preparation together with an overlay that omits the judge secret? Keep separate_verifier_for_artifacts: true and image_input: true; dropping the entire overlay would also drop those required behaviors. Leave the full-set overlay's missing-key refusal intact.

Proposed acceptance: with both key variables unset, the 64-task preparation and keyless overlay should start and reach verification. The full-set overlay should still reject a missing key, and Repair should still verify separately. This is a documentation usability suggestion; I did not run either configuration.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2478175: the entry README documents AGENTIC_VBENCH_HARBOR_FAMILIES=repair,assembly,sequencing preparation with the new harbor_sandbox_no_judge.yaml, the same overlay without the judge secret (separate_verifier_for_artifacts: true, the episode cap and image_input: true kept); the benchmark README's credentials cell points to it. The full-set overlay keeps its no-default secret, so it still refuses to start without a key. I have not run the keyless configuration end to end; the 64-task oracle numbers below come from the full-set overlay with a key.

goelarushi and others added 4 commits November 16, 2026 00:40
`_score` takes `separate`: True from `_verify_separately`, False from
`_run_shared_verifier` (the path, not the task's mode, since multi-step tasks mix
the two). In the agent's sandbox a 75 without a reward is a measured 0 as on any
other exit, with the ignored code in the reason; separately it is masked, a
malformed reward beside it included (the verifier's own file). The docs say a
valid written reward wins, qualify "the exit code does not affect the score",
state that separate mode is necessary but not sufficient, and say what
`gym eval resubmit --recover` does with the masked kind.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
hub_judge.py: the overall deadline becomes an allowance of time spent waiting on
the Hub (calls in flight and backoff, 1200 s), so local work over the agent's
output cannot look like an outage; each call is bounded as a whole on a thread
and a late success is dropped; a 413 or a token-limit 400 is the item's error as
upstream, never "could not grade", and a clip over the 14 MiB inline budget goes
as an ffmpeg proxy (at most 720p, audio kept) recorded in the provenance;
failures are keyed by route and payload and a final request error supersedes an
earlier outage of the same request; a truncated error body is read for its
status; the probes carry the judge's real request shapes and run before the first
Hub call, so an output the judge rejects locally scores 0 during an outage.
staging.py: `--retry-max-time 900` leaves the repair verifier room after the
download. Tests cover each branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
…als provenance

A `harbor_sandbox_no_judge.yaml` overlay and the `AGENTIC_VBENCH_HARBOR_FAMILIES`
recipe for the 64 tasks that need no credentials; the leaderboard score as the
mean of the four unscaled family means (equivalently `reward x 25 / family size`
over the complete 100 rows); `/workspace/materials` is filled by each step's
setup.sh from /baked, not by the image; the exit-75 claims say what the agent
cannot do (run the verifier, read the key) and what decides the code.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
… README

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
@goelarushi

Copy link
Copy Markdown
Contributor Author

Pushed 5d81ede98 (six commits on top of f7b60fbfb), one per review item, and replied in every thread with the commit that addresses it:

  • 601f7b920 the separately verified repair tasks receive /workspace/materials (the staging.py:306 blocker);
  • 7dce1ecc2 the server keeps a reward.json score beside non-numeric metric keys (found by the oracle run: the pinned assembly and sequencing judges write a details object beside reward, and every correct solution was measured 0);
  • 27dc4c177 exit 75 is honored only on the separate verifier path (_score(separate=)), docs aligned (valid reward wins, recovery behaviour);
  • 53da51e96 hub_judge.py: allowance of time waiting on the Hub instead of a wall-clock deadline, each call bounded as a whole, 413 and token-limit 400 as item errors with a bounded ffmpeg proxy for the inlined clip, failures keyed by payload and superseded by a final answer, truncated error bodies, probes in the judge's request shapes before the first Hub call; curl --retry-max-time 900;
  • 247817547 keyless overlay and recipe for the 64 unjudged tasks, the leaderboard formula, the materials and exit-75 claims;
  • 5d81ede98 the oracle results in the entry README.

Oracle run of every solution/ on this head (details and per-row reasons in the dataset.toml thread): Assembly 18/18, Sequencing 28/28 and Repair 18/18 at 1.0; Repurpose 36/36 scored at 0.202 with no masked rows, 11 at the 1800 s verifier budget inside the judge's own transcription, 8 zeros on rubric penalties of the reference edits.

Still owed and not yet posted: the full-set opencode_agent run with image_input on, and the injected-outage smoke. The PR body's evidence section says the same. CI needs the /ok to test on 5d81ede980988741e0b198bfe4e5ae619c719ac4.

…et opencode_agent run

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
@goelarushi

Copy link
Copy Markdown
Contributor Author

/ok to test f978bbd

1 similar comment
@goelarushi

Copy link
Copy Markdown
Contributor Author

/ok to test f978bbd

@yfw

yfw commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

/ok to test f978bbd

@yfw
yfw added this pull request to the merge queue Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43983812460

Merged via the queue into NVIDIA-NeMo:main with commit 9adce85 Nov 17, 2026
67 checks passed
goelarushi added a commit to goelarushi/Gym that referenced this pull request Nov 17, 2026
…y landed as NVIDIA-NeMo#4049)

Main's squash of NVIDIA-NeMo#4049 matches this branch's pre-training entry, so the add/add
conflicts resolve to this branch's files; the entry README takes main's recorded
full-set run and injected-outage smoke in place of the open-items paragraph.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arushi Goel <arushig@nvidia.com>
adil-a added a commit to zuraiz-anjum/Gym that referenced this pull request Nov 18, 2026
Since the Agentic-VBench benchmark entry (NVIDIA-NeMo#4049), main's reward parser
skips non-numeric keys (a `details` block, a `reason`) only beside a
numeric `reward`; without one every key must be a number. The parser did
not know the dataset's `reward_key`, so after the merge a file such as
`{"overall": 0.5, "accuracy": 1.0, "details": {...}}` under `reward_key
= "overall"` was refused whole: a measured 0 with
`harbor:invalid_reward`. The same file with a `reward` key keeps its
score.

`parse_reward_file` now takes the score key, and `_collect_reward` passes
the turn's `reward_key` on both verification paths:
- A numeric value under that key makes the other keys metrics, as
  `reward` does, so non-numeric keys beside it are left out.
- A file without the key holds no valid reward. A separately run
  test.sh that exits 75 is then masked as `harbor:verifier_unavailable`,
  and a run past its deadline is `harbor:verifier_timeout`, as for any
  file without a valid reward. Before, such a file parsed as valid and
  `_score` measured it as `harbor:invalid_reward`. `_score` no longer
  needs its own missing-key branch.

Tests: a `details` block beside the key keeps the score, a file without
the key is masked on exit 75 and a timeout past the deadline, and a file
whose other keys sum to the score (`{"overall": 0.5}` or with a 0.0
penalty) names no `reward_components`. That last test fails if
`reward_components` stops checking the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants