Skip to content

docs(skills): add-benchmark health checks and session rules - #4627

Merged
wprazuch merged 1 commit into
mainfrom
wprazuch/add-benchmark-skill-checks
Oct 23, 2026
Merged

wprazuch merged 1 commit into
mainfrom
wprazuch/add-benchmark-skill-checks

Conversation

@wprazuch

Copy link
Copy Markdown
Contributor

What

Updates the add-benchmark skill (.agents/skills/add-benchmark/SKILL.md).

  • Says the Add a benchmark page wins if the two disagree.
  • Step 3: an environment failure returns mask_sample, failure_kind and failure_reason, not a 0.0 reward.
  • Step 3: rules for a server that keeps session state (num_workers: 1, close_resources_session(), a per-session lock, free on failed seed, VERIFIER_FIXTURE, REVERIFY_MODE).
  • Step 7 is now "Exit checks": rollout health, gym eval compare, saved config. It replaces the old Qwen baseline step.
  • A short route for adding a harness instead of a benchmark.

Steps 1 to 6 are unchanged.

Validation

  • pre-commit run --all-files passes.
  • I checked each Gym name in the text against the source: ReverifyMode, close_resources_session, VERIFIER_FIXTURE, num_workers on agent servers.

Rollout evidence

N/A. Docs-only change, no code or config.

No issue: small docs change.

@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation complexity:low Localized change in one scope with a small, straightforward review surface docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer labels Oct 15, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 16, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 22, 2026
@wprazuch
wprazuch enabled auto-merge October 23, 2026 06:52
Step 7 is now health checks instead of a model baseline. Adds the
mask_sample rule, session-state rules for resources servers and a
short route for adding a harness.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch
wprazuch force-pushed the wprazuch/add-benchmark-skill-checks branch from f86d771 to ef73925 Compare October 23, 2026 06:54
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 23, 2026 06:54
@wprazuch
wprazuch added this pull request to the merge queue Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39914491040

Merged via the queue into main with commit 6360a71 Oct 23, 2026
35 of 36 checks passed
@wprazuch
wprazuch deleted the wprazuch/add-benchmark-skill-checks branch October 23, 2026 06:56

This branch was previously deployed

1 inactive deployment
public — ef739255 Deployed Oct 23, 2026 by copy-pr-bot[bot] via release / finalize / notify #6745
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation complexity:low Localized change in one scope with a small, straightforward review surface docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants