Skip to content

feat(launcher): add fast benchmark profiles and named resumable runs - #3602

Draft
christian-muertz wants to merge 1 commit into
codex/benchmark-launcherfrom
codex/benchmark-fast-runs
Draft

christian-muertz wants to merge 1 commit into
codex/benchmark-launcherfrom
codex/benchmark-fast-runs

Conversation

@christian-muertz

Copy link
Copy Markdown

Adds Brian Fitzgerald's fast benchmark profiles and named resumable runs on top of #3600.

Named runs share a results directory and queue successive Slurm jobs with a singleton dependency, allowing each job to resume cached rollouts. Fast Lightning profiles cover Terminal-Bench 2.1 (Terminus and OpenCode), SWE-bench Verified/Multilingual, and IFBench. They share capped generation settings, a pinned vLLM nightly, and routing of failed agent runs to the failures sidecar. Includes the development-checkout cache-directory and cleanup-script fixes.

The four-task smoke profile requires a separately prepared benchmarks/terminal_bench_2_1/data/smoke4_distinct.jsonl; no dataset or local configuration is included in this PR. Fast settings change evaluation budgets, so results should be identified as using these profiles.

Validation: repository-wide pre-commit checks, Bash syntax, YAML base-config references, and 16 mocked named/unnamed launcher invocations passed. Secret scanning passed. No live evaluation jobs were submitted; this ports the existing changes without retuning the benchmark behavior.

Stack: #3599 → #3600 → this PR. Review against codex/benchmark-launcher.

@copy-pr-bot

copy-pr-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@christian-muertz
christian-muertz force-pushed the codex/benchmark-fast-runs branch from e791870 to 466dc5b Compare September 22, 2026 09:39
@yaoyu-33 yaoyu-33 added area:orchestration Job submission, deployment, services, and lifecycle management complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 22, 2026
@christian-muertz
christian-muertz force-pushed the codex/benchmark-fast-runs branch from 466dc5b to 5d3398c Compare September 22, 2026 12:10
@christian-muertz
christian-muertz force-pushed the codex/benchmark-fast-runs branch 2 times, most recently from bdca7eb to 9d0edd4 Compare September 22, 2026 14:38
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 23, 2026
@christian-muertz
christian-muertz force-pushed the codex/benchmark-fast-runs branch 2 times, most recently from e2a5190 to 81f39be Compare September 23, 2026 10:04
Add shared fast serving settings, benchmark overrides, named output directories,
and serialized restart submissions. Preserve the supplied implementation while
applying it on top of the public launcher branch.

Co-authored-by: Brian Fitzgerald <bfitzgerald@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christian Muertz <cmrtz@nvidia.com>
@christian-muertz
christian-muertz force-pushed the codex/benchmark-fast-runs branch from 81f39be to f851606 Compare September 23, 2026 10:51
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 01:55
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:orchestration Job submission, deployment, services, and lifecycle management complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants