Skip to content
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
a955e80
feat(agentic-vbench): add pinned Harbor evaluation environment
Oct 1, 2026
7266080
docs(agentic-vbench): resolve runtime configuration before evaluation
Oct 1, 2026
ed9d1cc
fix(agentic-vbench): shorten private runtime paths for Podman sockets
Oct 1, 2026
0523057
feat(agentic-vbench): support disjoint task and family selections
Oct 1, 2026
8c9b3bd
feat(agentic-vbench): upstream Harbor 0.6.6 runner with rootless/remo…
Oct 2, 2026
e3b4647
fix(agentic-vbench): stabilize native Factory rollouts and Hub judging
Oct 4, 2026
7ddaa8c
fix(agentic-vbench): update runner and judge handling
goelarushi Oct 5, 2026
881557c
feat(agentic-vbench): route OpenCode through Gym's policy model server
goelarushi Oct 4, 2026
4ce2ecc
feat(agentic-vbench): self-contained inputs for gym-native drivers
goelarushi Oct 4, 2026
bad0b65
fix(agentic-vbench): mirror the judge's per-item handling of persiste…
goelarushi Oct 5, 2026
b1d129f
feat(agentic-vbench): OpenCode subagent policy knob (default/off/on)
goelarushi Oct 9, 2026
e4a7fef
fix(agentic-vbench): judge outlasts provider outages
goelarushi Oct 9, 2026
cfd7d12
fix(agentic-vbench): give agent setup a 3x timeout allowance
goelarushi Oct 10, 2026
395b628
feat(agentic-vbench): custom task roots for generated tasks
goelarushi Oct 15, 2026
7db9fee
fix(agentic-vbench): accept custom task namespaces in Harbor results
goelarushi Oct 15, 2026
05981fe
feat(agentic-vbench): keep_task_images knob for repeated trials
goelarushi Oct 15, 2026
865ea67
fix(agentic-vbench): nvinference-hybrid judge routes reachable by the…
goelarushi Oct 19, 2026
1038d29
test(agentic-vbench): expect the current nvinference-hybrid judge routes
goelarushi Oct 21, 2026
75138b9
feat(agentic-vbench): benchmark manifest, README row and resumable ep…
goelarushi Oct 21, 2026
88eb4bc
fix(agentic-vbench): give a cold task image build up to an hour
goelarushi Oct 21, 2026
7db9bbd
fix(agentic-vbench): literal dataset path in the benchmark config
goelarushi Oct 21, 2026
d4a3f45
fix(agentic-vbench): re-enable pathspec for Harbor in the agent venv
goelarushi Oct 21, 2026
df03625
fix(agentic-vbench): follow main's package layout, component locks an…
goelarushi Nov 12, 2026
df24830
feat(agentic-vbench): bring the agent to the version used for NeMo-RL…
goelarushi Nov 12, 2026
79f950f
feat(agentic-vbench): run the tasks through the harbor Environment Se…
goelarushi Nov 12, 2026
db188ce
feat(agentic-vbench): let subagent training expose the SFT traces' an…
goelarushi Nov 12, 2026
7de55d2
refactor(agentic-vbench): land the Harbor entry only and mask verifie…
goelarushi Nov 13, 2026
db675e9
fix(agentic-vbench): report "could not grade" with exit 75 and verify…
goelarushi Nov 14, 2026
78f8183
Merge remote-tracking branch 'upstream/main' into agentic-vbench
goelarushi Nov 14, 2026
0bd3d89
test(agentic-vbench): keep the stalled-Hub socket server independent …
goelarushi Nov 14, 2026
eca8361
test(agentic-vbench): make the truncated-body Hub case deterministic
goelarushi Nov 14, 2026
393e070
test(agentic-vbench): drain the client in the truncated-body Hub case…
goelarushi Nov 14, 2026
f7b60fb
test(agentic-vbench): give every socket-server case an orderly EOF
goelarushi Nov 14, 2026
601f7b9
fix(agentic-vbench): give the separately verified repair tasks their …
goelarushi Nov 15, 2026
7dce1ec
fix(harbor): keep a reward.json score beside non-numeric metric keys
goelarushi Nov 15, 2026
27dc4c1
fix(harbor): honor a verifier's exit 75 only on the separate path
goelarushi Nov 16, 2026
53da51e
fix(agentic-vbench): keep the agent's output out of the judge's exit 75
goelarushi Nov 16, 2026
2478175
docs(agentic-vbench): keyless run recipe, leaderboard formula, materi…
goelarushi Nov 16, 2026
5d81ede
docs(agentic-vbench): record the oracle validate results in the entry…
goelarushi Nov 16, 2026
f978bbd
docs(agentic-vbench): record the injected-outage smoke and the full-s…
goelarushi Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions benchmarks/agentic_vbench/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data/
45 changes: 45 additions & 0 deletions benchmarks/agentic_vbench/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Agentic-VBench

Evaluate video-editing agents on the 100 pinned [Agentic-VBench](https://github.com/PhiloLabs/agentic-vbench)
tasks: Repair (18), Assembly (18), Sequencing (28) and Repurpose (36). The tasks are Harbor task folders and run
through Gym's generic [`harbor` environment server](../../environment_servers/harbor/README.md) with any sandboxed
session harness; the committed entry is [`environments/harbor/agentic_vbench/`](../../environments/harbor/agentic_vbench/README.md).
This folder holds the code that prepares it.

| File | Role |
| --- | --- |
| `prepare_harbor_env.py` | Writes the 100 pinned tasks into the entry (`python -m benchmarks.agentic_vbench.prepare_harbor_env`) |
| `checkout.py` | Fetches the pinned upstream revision into the Gym cache root and lists the four families' tasks |
| `staging.py` | The per-task adjustments: Hub judge routing, assembly judge guard, bounded golden-asset retries, exit 75 for verifier-side outages, `/workspace/output` as the judged and repair steps' artifact |
| `hub_judge.py` | Drop-in for the repurpose tasks' `judge.py`: the unchanged rubric judge with both providers served by the NVIDIA Inference Hub |

## Measurement contract

| | |
|---|---|
| Source | `PhiloLabs/agentic-vbench` @ `410a75fe3dae37de4344fc9e5317da089505330a` (Apache-2.0); the four leaderboard families, the "understanding" family is excluded as upstream does |
| Task contract | The task's own `instruction.md` is the prompt; the model works in the task's own image with the harness's tools and must inspect extracted frames itself (no media is injected into the prompt). Reference solutions stay under `solution/`, which the agent's sandbox does not see |
| Harness | Any session harness the `harbor` server can run in a sandbox; `opencode_agent` with `image_input` (set by the entry's overlay) is the one under validation (the published numbers come from OpenCode 1.14.39 inside the task container) |
| Budget | Each task declares 1800 s for the agent and 1800 s for the verifier, used as published (`[gym.defaults] timeout_multiplier` would scale both, so the published 3x verifier-only allowance is not expressible yet) |
| Reward | The task's `reward.json`, 0..1, higher is better; the leaderboard number is the equal-family mean (`reward x 25 / family size` averaged over the four families), computed from the rows by task family |
| Judge | Repurpose tasks carry an LLM rubric judged over the NVIDIA Inference Hub on `NVINFERENCE_API_KEY` (or `INFERENCE_API_KEY`); without Hub access the 36 Repurpose tasks cannot be scored, the other 64 need no credentials. `AGENTIC_VBENCH_JUDGE_PROTOCOL=nvinference-upstream` (default) uses the upstream pair, Claude Opus 4.7 and Gemini 3.1 Pro; `nvinference-hybrid` uses Claude Opus 4.8 and Gemini 3.5 Flash with bounded thinking. Hub-judged scores are protocol-equivalent, not byte-identical, to the upstream SDK judges; the protocol is recorded per task in the entry's `.records/`. The Gemini judge inlines the output clip as base64; a request the Hub refuses for its size is reported as unavailable, not scored |
| Infrastructure failures | A judge route the key cannot use, a Hub outage, a missing judge key, the judge's own deadline, or a failed golden-asset download make the verifier exit 75 without a reward; the server masks the rollout (`harbor:verifier_unavailable`) instead of scoring 0. The judged and repair tasks verify in a separate sandbox so the agent cannot produce that exit itself |
| Determinism | Stochastic (sampling, container timing, judge) |

## Reference results

| Model | Avg | Repurpose | Sequencing | Repair | Assembly |
|---|---:|---:|---:|---:|---:|
| Super 3.5 GA (vLLM 0.26, TP4, temperature 0.6, top-p 0.95) | 12.2 ± 0.4 | 16.5 | 5.2 | 15.7 | 11.3 |
| Kimi K3 (NVIDIA Inference Hub) | 36.3 ± 1.8 | 39.2 | 40.8 | 25.6 | 39.5 |
| Qwen3.8-Flash-Next (vLLM, temperature 1.0, top-p 0.95) | 37.5 ± 2.5 | 46.4 | 30.0 | 17.9 | 55.8 |

Avg is the unweighted mean of the four family means in percent over three complete rollouts (request seeds 201,
202 and 203); ± is the population standard deviation across the seeds. These numbers were measured in 2026-10 with
the benchmark's own runtime (Harbor 0.6.6 with OpenCode 1.14.39 in the task container) driven from NVIDIA Eval
Factory. Two conventions differ from this entry's defaults: the reference runs judged with the `nvinference-hybrid`
pair, where the entry defaults to `nvinference-upstream` (select hybrid with `AGENTIC_VBENCH_JUDGE_PROTOCOL` to
compare), and they scored failed rollouts 0, where the entry masks a verifier that could not grade. They are the
reference the entry is compared against, not results of the entry: the entry has not been baselined with
`gym eval run` yet (no oracle validate output and no full-set run are recorded), so `verified` stays false in
`dataset.toml`. Do not report a subset or a single-seed result as the benchmark score.
2 changes: 2 additions & 0 deletions benchmarks/agentic_vbench/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
119 changes: 119 additions & 0 deletions benchmarks/agentic_vbench/checkout.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""The pinned Agentic-VBench checkout: fetch it once and list its 100 benchmark tasks."""

import fcntl
import hashlib
import os
import shutil
import subprocess
import tempfile
from collections import Counter
from contextlib import contextmanager
from pathlib import Path
from typing import Iterator


REPO_URL = "https://github.com/PhiloLabs/agentic-vbench.git"
REVISION = "410a75fe3dae37de4344fc9e5317da089505330a" # pragma: allowlist secret
# The four leaderboard families and their task counts; the checkout's separate "understanding" area is not part
# of the benchmark.
BENCHMARK_FAMILIES = {"repair": 18, "assembly": 18, "sequencing": 28, "repurpose": 36}


def cache_root() -> Path:
override = os.environ.get("AGENTIC_VBENCH_CACHE_ROOT")
if override:
return Path(override).expanduser()
hf_home = Path(os.environ.get("HF_HOME", "~/.cache/huggingface")).expanduser()
return hf_home.parent / "agentic_vbench"


def default_checkout_root() -> Path:
return cache_root() / f"agentic-vbench-{REVISION[:12]}"


@contextmanager
def _locked(path: Path) -> Iterator[None]:
"""Serialize concurrent fetches where the filesystem supports flock.

Some network mounts refuse flock; then the staging-directory rename below is the only guard, and a
concurrent loser simply redoes the work.
"""
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w") as handle:
locked = True
try:
fcntl.flock(handle, fcntl.LOCK_EX)
except OSError:
locked = False
try:
yield
finally:
if locked:
fcntl.flock(handle, fcntl.LOCK_UN)


def _git(*args: str, cwd: Path) -> None:
subprocess.run(["git", "-C", str(cwd), *args], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)


def ensure_checkout(root: Path | None = None, repo_url: str | None = None) -> Path:
"""Materialize the pinned revision at ``root`` (the cache root by default); reuse a matching checkout."""
root = (root or default_checkout_root()).expanduser()
repo_url = repo_url or os.environ.get("AGENTIC_VBENCH_REPO_URL") or REPO_URL
with _locked(root.parent / f"{root.name}.lock"):
if (root / ".git").exists():
head = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
if head != REVISION:
raise ValueError(f"Existing checkout {root} is at {head}, expected {REVISION}")
return root
if root.exists() and any(root.iterdir()):
raise ValueError(f"Checkout target is not empty: {root}")
staging = Path(tempfile.mkdtemp(prefix=f".{root.name}.", dir=root.parent))
try:
_git("init", "--quiet", cwd=staging)
_git("remote", "add", "origin", repo_url, cwd=staging)
try:
_git("fetch", "--quiet", "--depth", "1", "origin", REVISION, cwd=staging)
except subprocess.CalledProcessError:
# Some servers refuse fetching an unadvertised commit; take the full history.
_git("fetch", "--quiet", "origin", cwd=staging)
_git("checkout", "--quiet", "--detach", REVISION, cwd=staging)
if (root / ".git").exists():
shutil.rmtree(staging) # an unlocked concurrent caller won
else:
staging.replace(root)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
raise
return root


def inventory(root: Path) -> dict[str, dict]:
"""The 100 pinned tasks by task id: family, prompt digest and revision; raises on any drift from the pin."""
root = root.resolve(strict=True)
revision = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
if revision != REVISION:
raise ValueError(f"Expected Agentic-VBench {REVISION}, found {revision}")
changed = subprocess.check_output(["git", "-C", str(root), "status", "--porcelain", "--", "tasks"], text=True)
if changed.strip():
raise ValueError("Benchmark tasks differ from the pinned checkout")
tasks: dict[str, dict] = {}
family_dirs = (root / "tasks" / f"agentic_vbench_{family}" for family in BENCHMARK_FAMILIES)
for path in sorted(task for family_dir in family_dirs for task in family_dir.glob("*/task.toml")):
family = path.parent.parent.name.removeprefix("agentic_vbench_")
task_id = path.parent.name
if task_id in tasks:
raise ValueError(f"Duplicate task id across families: {path}")
prompt = (path.parent / "steps/solve/instruction.md").read_text()
tasks[task_id] = {
"task_id": task_id,
"family": family,
"prompt_sha256": hashlib.sha256(prompt.encode()).hexdigest(),
"benchmark_revision": revision,
}
counts = Counter(task["family"] for task in tasks.values())
if counts != BENCHMARK_FAMILIES:
raise ValueError(f"Expected the complete 100-task inventory, got {dict(counts)}")
return tasks
Loading
Loading