Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions fern/versions/latest/pages/evaluation/benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,12 @@ aime24_math_with_judge_simple_agent:
Use the CLI to browse what's available before running:

```bash
# List all benchmarks with domain and repeat count
# List all benchmarks with domain and description
gym list benchmarks

# Inspect one benchmark, including its agent harness
gym list benchmarks aime24

# Filter to a capability area
gym search benchmarks math
gym search benchmarks coding
Expand All @@ -66,7 +69,7 @@ For external benchmarks, reproduce the original reported numbers outside Gym bef

## Benchmark Categories

NeMo Gym includes 80+ benchmarks across six categories. Use `gym list benchmarks` to browse the full list with domains and repeat counts.
NeMo Gym includes 80+ benchmarks across six categories. Use `gym list benchmarks` to browse the full list with domains and descriptions.

**Math & Science**
AIME, IMO, formal proofs (Lean4), physics (critpt, ugphysics), chemistry (ether0, BunsenBench), FrontierScience, and multilingual math. Verification ranges from exact match and math-verify to LLM judges for open-ended proofs.
Expand Down
3 changes: 2 additions & 1 deletion fern/versions/latest/pages/evaluation/environment-list.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ NeMo Gym includes 100+ environments covering math, coding, reasoning, knowledge,
Use the CLI to discover what's available before running:

```bash
gym list benchmarks # all benchmarks with domain and repeat count
gym list benchmarks # all benchmarks with domain and description
gym list benchmarks aime24 # inspect one benchmark, including its agent harness
gym list environments # unified environment and benchmark catalog
gym search math # search environments and benchmarks by capability area
gym list benchmarks --json # machine-readable output for scripting or CI
Expand Down
33 changes: 19 additions & 14 deletions fern/versions/latest/pages/get-started/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,23 +98,28 @@ NeMo Gym ships with environments across many domains. You can use these existing
gym list benchmarks
```

The table below is trimmed to a few rows and a narrower **Description** column so it fits this page:

```text
Available benchmarks in NeMo Gym (76)
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark name ┃ Domain ┃ Agent name ┃ Num repeats ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ aime24 │ math │ aime24_math_with_judge_simple_ag… │ 32 │
│ aime25 │ math │ aime25_math_with_judge_simple_ag… │ 32 │
│ browsecomp │ agent │ browsecomp_tavily_search_simple_… │ 1 │
│ gpqa │ knowledge │ gpqa_mcqa_simple_agent │ 8 │
│ ifbench │ instruction_fol… │ ifbench_benchmark_simple_agent │ 5 │
| ... | ... | ... | ... |
│ tau2 │ agent │ tau2_benchmark_agent │ 8 │
│ xstest │ safety │ xstest_benchmark_simple_agent │ 4 │
└──────────────────┴────────────────────┴───────────────────────────────────┴─────────────┘
Available environments and benchmarks (113; manifests 0/292)
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Kind ┃ Status ┃ Lifecycle ┃ Domain ┃ Description ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ aime24 │ benchmark │ no-manifest │ │ math │ OpenMathReasoning math dataset with… │
│ gpqa │ benchmark │ no-manifest │ │ knowledge │ Multi-choice question answering pro… │
│ ifbench │ benchmark │ no-manifest │ │ instruction_following │ IFBench instruction following evalu… │
│ xstest │ benchmark │ no-manifest │ │ safety │ XSTest safety benchmark - exaggerat… │
│ ... │ ... │ ... │ │ ... │ ... │
└─────────┴───────────┴─────────────┴───────────┴───────────────────────┴──────────────────────────────────────┘
```

**Status** reports whether the entry has a manifest, and **Lifecycle** is populated only for manifest-backed entries. To see which agent harness and datasets a benchmark composes with, inspect it by name:

```bash
gym list benchmarks aime24
```

This lists benchmarks with pre-configured agents. For the full set of environments (including training environments), see the [Available Environments](https://github.com/NVIDIA-NeMo/Gym#-available-environments) table.
For the full set of environments (including training environments), see the [Available Environments](https://github.com/NVIDIA-NeMo/Gym#-available-environments) table.

Every CLI command supports `-h` or `--help` for detailed usage information:

Expand Down
5 changes: 3 additions & 2 deletions fern/versions/latest/pages/reference/cli-commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -142,12 +142,13 @@ All commands accept:

### `gym list benchmarks`

List the benchmarks available in NeMo Gym, with their domain, agent, and configured number of repeats.
List the benchmarks available in NeMo Gym, with their kind, manifest status, domain, and description.
The value in the **Name** column is the benchmark identifier and can be passed to `--benchmark`.
To see which agent harness a benchmark composes with, inspect it with `gym list benchmarks <name>`.

| Option | Description |
| --- | --- |
| `[<name>]` | Inspect a single benchmark (config path, dataset, prepare script) instead of listing all. |
| `[<name>]` | Inspect a single benchmark (config path, manifest status, agent, datasets) instead of listing all. |
| `--json` | Output the benchmark list as JSON. |

```bash
Expand Down
33 changes: 21 additions & 12 deletions nemo_gym/benchmarks.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,15 @@ def from_initial_config_dict(
if len(datasets) < 1:
return

# A config resolving to several benchmark datasets is an eval suite. Reporting the first would alias
# the suite to whichever benchmark happened to sort first, under the suite's own name.
if len(datasets) > 1:
print(f"Expected 1 benchmark dataset for config {path}, but found {len(datasets)}!")
print(
f"Warning: skipping '{path}': it resolves to {len(datasets)} benchmark datasets, "
"so it is an eval suite rather than a single benchmark.",
file=sys.stderr,
)
return

dataset = datasets[0]

Expand All @@ -115,30 +122,32 @@ def _benchmark_config_name(rel_config_path: Path) -> str:


def _is_benchmark_config(config_path: Path) -> bool:
"""True if the config declares a `type: benchmark` dataset anywhere in its structure.
"""True if the config declares exactly one `type: benchmark` dataset in its own structure.

A raw single-file parse (no `config_paths`/interpolation resolution), so it's format-agnostic and can't
fail on includes. An unparseable file is kept (returns True) so the resolve step surfaces a diagnostic.
fail on includes. Declaring several makes it an eval suite rather than a benchmark: there is no single
name, agent, or repeat count to catalog it under, so it is not a valid `--benchmark` argument. An
unparseable file is kept (returns True) so the resolve step surfaces a diagnostic.
"""

def declares(node: object) -> bool:
def count(node: object) -> int:
if isinstance(node, dict):
return node.get("type") == "benchmark" or any(declares(v) for v in node.values())
return (1 if node.get("type") == "benchmark" else 0) + sum(count(v) for v in node.values())
if isinstance(node, list):
return any(declares(v) for v in node)
return False
return sum(count(v) for v in node)
return 0

try:
return declares(yaml.safe_load(config_path.read_text(errors="ignore")))
return count(yaml.safe_load(config_path.read_text(errors="ignore"))) == 1
except yaml.YAMLError:
return True


def _benchmark_config_paths(benchmarks_dir: Path) -> List[Path]:
"""Sorted config paths under one dir that declare a benchmark, discovered by content.

A config is a benchmark iff it declares a `type: benchmark` dataset, regardless of filename, so we scan
every yaml. :func:`_is_benchmark_config` is a cheap prefilter (pay the resolve cost only on real
A config is a benchmark iff it declares exactly one `type: benchmark` dataset, regardless of filename, so
we scan every yaml. :func:`_is_benchmark_config` is a cheap prefilter (pay the resolve cost only on real
candidates) that also catches non-`config.yaml` names like tau2's `configs/*.yaml`. Empty if dir missing.
"""
if not benchmarks_dir.is_dir():
Expand All @@ -155,8 +164,8 @@ def _discover_benchmarks_in_dir(benchmarks_dir: Path) -> Dict[str, BenchmarkConf
# Listing has no runtime context, so tolerate unset runtime-only values.
maybe_bc = BenchmarkConfig.from_config_path(config_path, strict=False)
except Exception as e:
# Still unresolvable (e.g. a multi-benchmark suite) — skip with a warning rather than fail the
# whole listing, so it isn't silently invisible.
# Still unresolvable (e.g. missing runtime-only values) — skip with a warning rather than fail
# the whole listing, so it isn't silently invisible.
print(
f"Warning: skipping benchmark config '{config_path}': could not resolve it "
f"({type(e).__name__}: {str(e).splitlines()[0]}).",
Expand Down
19 changes: 12 additions & 7 deletions nemo_gym/global_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -361,10 +361,14 @@ def load_extra_config_paths(self, config_paths: List[str]) -> Tuple[List[str], L

if duplicate_config_paths:
duplicate_config_paths_str = "".join(f"- {p}\n" for p in duplicate_config_paths)
print(f"""Found configs that reference the same source config path. You may want to double check whether the configs you have need to use different configs for the same server.
# Diagnostics go to stderr so that `--json` output on stdout stays machine-readable.
print(
f"""Found configs that reference the same source config path. You may want to double check whether the configs you have need to use different configs for the same server.
In cases like these, you may want to consider using the `inherit_from` OmegaConf directive e.g. '++my_specific_server=${{inherit_from:generic_server}}' and then overriding config parameters in `my_specific_server`.
Duplicate config paths:
{duplicate_config_paths_str}""")
{duplicate_config_paths_str}""",
file=sys.stderr,
)

# Flatten the include tree so that every config merges after -- and therefore
# overrides -- the ones it pulled in, and each subtree stays contiguous in
Expand Down Expand Up @@ -725,14 +729,15 @@ def parse(self, parse_config: Optional[GlobalConfigDictParserConfig] = None) ->
almost_servers = self.detect_and_report_almost_servers(global_config_dict)

if almost_servers:
rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]")
rich.print("[yellow]Configuration Warnings: Almost-Servers Detected[/yellow]")
rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]")
# Diagnostics go to stderr so that `--json` output on stdout stays machine-readable.
rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]", file=sys.stderr)
rich.print("[yellow]Configuration Warnings: Almost-Servers Detected[/yellow]", file=sys.stderr)
rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]", file=sys.stderr)

for server_name, error in almost_servers:
rich.print(format_almost_server_warning(server_name, error))
rich.print(format_almost_server_warning(server_name, error), file=sys.stderr)

rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]\n")
rich.print("[yellow]═══════════════════════════════════════════════════[/yellow]\n", file=sys.stderr)

error_on_almost_servers = global_config_dict.get("error_on_almost_servers", True)
if error_on_almost_servers:
Expand Down
29 changes: 29 additions & 0 deletions tests/unit_tests/test_benchmarks.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,21 @@ def test_prefilter_matches_type_benchmark_across_yaml_formatting(self, tmp_path,
config_path.write_text(text)
assert _is_benchmark_config(config_path) is is_benchmark

def test_prefilter_rejects_a_multi_benchmark_suite(self, tmp_path) -> None:
# A config declaring several benchmark datasets is an eval suite: it has no single name, agent, or
# repeat count, so it is not a valid `--benchmark` argument and must not reach the catalog.
from nemo_gym.benchmarks import _benchmark_config_paths

(tmp_path / "single").mkdir()
(tmp_path / "single" / "config.yaml").write_text("x:\n datasets:\n - name: a\n type: benchmark\n")
(tmp_path / "suite").mkdir()
(tmp_path / "suite" / "config.yaml").write_text(
"x:\n datasets:\n - name: a\n type: benchmark\ny:\n datasets:\n - name: b\n type: benchmark\n"
)

found = {str(p.relative_to(tmp_path)) for p in _benchmark_config_paths(tmp_path)}
assert found == {"single/config.yaml"}

def test_prefilter_keeps_unparseable_yaml_as_candidate(self, tmp_path) -> None:
# A file we can't parse can't be classified, so it is kept as a candidate for the resolve step to
# diagnose rather than silently dropped.
Expand Down Expand Up @@ -210,6 +225,20 @@ def fake_from_config_path(path, *, strict=True):
err = capsys.readouterr().err
assert "Warning" in err and "bad" in err

def test_suite_resolving_to_many_benchmarks_is_skipped_not_aliased(self, tmp_path: Path, capsys) -> None:
# Resolving `config_paths` can surface several benchmark datasets even when the file itself declares
# one. Reporting the first would publish the suite under another benchmark's name, agent, and repeat
# count, so `--benchmark <suite>` would silently run that other benchmark instead.
from nemo_gym.benchmarks import BenchmarkConfig

# Chains two real benchmarks the way `benchmarks/nemotron_3.5_super/` chains its eval suite.
suite = tmp_path / "suite.yaml"
suite.write_text("config_paths:\n- benchmarks/gpqa/config.yaml\n- benchmarks/aime24/config.yaml\n")

assert BenchmarkConfig.from_config_path(suite, strict=False) is None
err = capsys.readouterr().err
assert "2 benchmark datasets" in err and "eval suite" in err

def test_every_repo_benchmark_appears_in_listing(self, capsys) -> None:
# Every config that declares a `type: benchmark` dataset must surface as its own listing entry —
# no silent drop from a name collision (the name-keyed dict is last-writer-wins) or a resolve
Expand Down
Loading
Loading