Skip to content

docs: rewrite add-benchmark skill for catalog overlays - #2908

Draft
sephmard wants to merge 2 commits into
docs/benchmark-onboarding-2545from
docs/add-benchmark-skill
Draft

sephmard wants to merge 2 commits into
docs/benchmark-onboarding-2545from
docs/add-benchmark-skill

Conversation

@sephmard

@sephmard sephmard commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite .agents/skills/add-benchmark so agents copy the closest catalog overlay instead of scaffolding a resources server
  • Point Codex at the same skill ( .codex/skills/add-benchmark symlink) and drop references/patterns.md
  • Keep Agent Skills / debugging pointers aligned: new verify() is New Environment, not this skill

Stacked on #2546.

Test plan

  • pytest tests/unit_tests/test_fern_docs_links.py::TestFernDocsLinks::test_add_benchmark_skill_is_an_overlay_only_playbook
  • Ask an assistant to add a math overlay (copy GSM8K) and confirm it does not run gym env init --resources-server
  • Ask an assistant to add an MCQ overlay (copy GPQA) and confirm it does not paste GSM8K YAML or rewrite the HuggingFace id

@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

@sephmard
sephmard force-pushed the docs/add-benchmark-skill branch from aedefdc to 08cc675 Compare September 1, 2026 02:00
Teach copy-closest-overlay rather than scaffolding a scorer, and keep Codex on the same skill via symlink.

Signed-off-by: Seph Mard <smard@nvidia.com>
The overlay skill no longer mentions the internal registry.

Signed-off-by: Seph Mard <smard@nvidia.com>
@sephmard
sephmard force-pushed the docs/add-benchmark-skill branch from 08cc675 to c2c3fa7 Compare September 1, 2026 02:15
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers docs Documentation-only updates or documentation debt labels Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers docs Documentation-only updates or documentation debt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants