Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions benchmarks/indic/mbpp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Indic MBPP

Original MBPP from `ai4bharat/indic-mbpp`: 500 Python tasks in each of 14 Indic
languages. Includes an English baseline (`en`) from the same source records.
Reuses Gym's [English MBPP prompt](../../mbpp/prompts/default.yaml), `simple_agent`,
and the existing [SciCodePile assertion verifier](../../../resources_servers/scicodepile/README.md).
The `indic_mbpp_environment_server` uses Gym's `legacy_agent` relay to route
episodes to `indic_mbpp_simple_agent`, following the upstream benchmark layout.
The relay does not change the prompt or scoring contract.

## Configuration

Each prompt contains the translated description, task setup when present, and the
first original assertion as an example. Four spaces are replaced with a tab, as
in English MBPP preparation. Reference solutions and the other two assertions are
kept out of the prompt. Each response must provide complete Python code.

Reward is 1 only when all three original `test_list` assertions pass. The existing
verifier runs each response in a fresh subprocess with a 120-second timeout.
`challenge_test_list` is excluded from the standard score. Default repeats: 1.
Use `pass@1/accuracy` for a single attempt; repeated runs also report pass@k and
`pass@1[avg-of-k]/accuracy`. Inspect coverage and `harness_failure` alongside scores.
Saved verifier results retain a language-qualified `task_id`, such as
`Mbpp/hi/11`, so results can be grouped by language without joining the inputs.
The top-level input `language` and `uuid` are not returned by SciCodePile.
Older prepared files use `Mbpp/11`; regenerate them to obtain qualified IDs.
This metadata change does not change prompts, assertions, or rewards.

Gym's existing `mbpp` benchmark is MBPP+ (378 EvalPlus tasks), while this dataset
contains the 500 original MBPP test tasks. Their scores are different benchmarks:
only 224 task IDs overlap, and 115 of those have revised English descriptions.
Use `indic/mbpp` with `languages: [en]` for a comparison on the same 500 tasks,
with identical model and generation settings. The existing `mbpp` benchmark and
its EvalPlus base/plus metrics are unchanged.

## Prepare data

Install Gym and follow the shared verifier's execution requirements. Generated
code runs as a subprocess; use an isolated evaluation environment as described
in the verifier README.
The source repository requires approved Hugging Face access. Authenticate with
`hf auth login` (or supply `HF_TOKEN`) using an account that can read
`ai4bharat/indic-mbpp`; authentication alone does not grant dataset access.

```bash
gym eval prepare --benchmark indic/mbpp
```

Defaults to `as`, `bn`, `gu`, `hi`, `kn`, `ml`, `mr`, `ne`, `or`, `pa`, `sa`, `ta`,
`te`, and `ur` (7,000 tasks). For per-language results, prepare and run one language
at a time; use `en` for the matching baseline:

```bash
gym eval prepare --benchmark indic/mbpp \
'+prepare_script_args={languages:[hi]}' \
+use_cached_prepared_benchmarks=false
```

Use `task_ids` in `prepare_script_args` to select smoke-test tasks. Data is downloaded
from the dataset's default revision. Missing translations are rejected and generated
data remains local.
Preparation also writes an adjacent `.provenance.json` file recording the resolved
dataset commit, source parquet SHA-256, selected languages/task IDs, and row count.
The revision comes from the downloaded snapshot path, not a separate lookup of
the latest commit. Non-snapshot paths are recorded as `null`, with the checksum
still retained. No source revision is pinned. Keep this sidecar with the inputs
and results; cached prepared data does not automatically refresh upstream changes.

## Collect rollouts

```bash
gym eval run --benchmark indic/mbpp \
--model-type vllm_model \
--model MODEL_NAME \
--model-url http://HOST:PORT/v1 \
--model-api-key dummy \
--split benchmark \
--output results/indic_mbpp/rollouts.jsonl
```

The canonical config does not set sampling, thinking, a seed, or an output-token
limit. Specify these in the model/run configuration, and use the same settings
for each language and the English baseline. The verifier timeout is not a model
generation-token limit.

## Paired real-model smoke

The standalone [smoke config](smoke.yaml) selects one attempt each for two task
IDs in English, Hindi, and Kannada after preparing the inputs below. It uses
`google/gemma-4-31B-it`, thinking enabled, temperature 1.0, top-p 0.95, top-k 64,
seed 0, and at most 8,192 output tokens (including reasoning).
See the [inspected smoke results](smoke-results.md) for the source/model revisions,
server settings, outputs, and rewards. This small check is not a benchmark score.

```bash
python -m benchmarks.indic.mbpp.prepare \
--languages en hi kn --task-ids 11 367 \
--output-fpath benchmarks/indic/mbpp/data/review-smoke.jsonl

gym eval run --config benchmarks/indic/mbpp/smoke.yaml \
--agent indic_mbpp_simple_agent \
--model google/gemma-4-31B-it \
--model-url http://HOST:PORT/v1 --model-api-key dummy \
--split benchmark --concurrency 3 \
--output results/indic_mbpp_smoke/rollouts.jsonl
```

Start a compatible vLLM endpoint with the `gemma4` reasoning parser. Do not append
`--model-type vllm_model` to this smoke command: the config already includes the
adapter, and reloading its defaults afterward can erase the sampling overrides.
Sampling/seed and thinking are scoped to the smoke model adapter, not global
request metadata. The smoke has one repeat; use a separate run configuration
with independent seeds when estimating multi-attempt metrics.

The dataset declares Apache-2.0.
2 changes: 2 additions & 0 deletions benchmarks/indic/mbpp/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
29 changes: 29 additions & 0 deletions benchmarks/indic/mbpp/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
config_paths:
- resources_servers/scicodepile/configs/scicodepile.yaml

indic_mbpp_resources_server:
_inherit_from: scicodepile

indic_mbpp_simple_agent:
_inherit_from: scicodepile_simple_agent
responses_api_agents:
simple_agent:
resources_server:
name: indic_mbpp_resources_server
datasets:
- name: indic_mbpp
type: benchmark
jsonl_fpath: benchmarks/indic/mbpp/data/mbpp_benchmark.jsonl
prepare_script: benchmarks/indic/mbpp/prepare.py
prompt_config: benchmarks/mbpp/prompts/default.yaml
num_repeats: 1
license: Apache-2.0

indic_mbpp_environment_server:
_inherit_from: scicodepile_environment_server
environment_servers:
legacy_agent:
agent_server:
name: indic_mbpp_simple_agent
2 changes: 2 additions & 0 deletions benchmarks/indic/mbpp/data/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*
!.gitignore
31 changes: 31 additions & 0 deletions benchmarks/indic/mbpp/manifest.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
name: indic/mbpp
version: 0.1.0
experimental: true
kind: benchmark
integration_profile: custom-gym-verifier
domain: coding
description: Original MBPP in 14 Indic languages, with unchanged assertion tests and a matching English baseline.
modality: text
licensing: Apache-2.0
authors: [NVIDIA NeMo Gym contributors]
reward:
range: [0, 1]
higher_is_better: true
determinism: stochastic
resources_server: scicodepile
agent_server: simple_agent
model_server: policy_model
datasets:
- name: indic_mbpp
type: benchmark
jsonl_fpath: benchmarks/indic/mbpp/data/mbpp_benchmark.jsonl
prepare_script: benchmarks/indic/mbpp/prepare.py
prompt_config: benchmarks/mbpp/prompts/default.yaml
num_repeats: 1
standard_prompt_config: benchmarks/mbpp/prompts/default.yaml
canonical_split: benchmark
session_model: episode
state: none
lifecycle: active
178 changes: 178 additions & 0 deletions benchmarks/indic/mbpp/prepare.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Prepare original MBPP translations and a matching English baseline."""

import argparse
import ast
import hashlib
import json
import re
from collections.abc import Mapping, Sequence
from pathlib import Path
from textwrap import indent

from datasets import load_dataset
from huggingface_hub import get_token, hf_hub_download

from nemo_gym.global_config import HF_TOKEN_KEY_NAME, maybe_get_global_config_dict


BENCHMARK_DIR = Path(__file__).parent
OUTPUT_FPATH = BENCHMARK_DIR / "data" / "mbpp_benchmark.jsonl"
SOURCE_ID = "ai4bharat/indic-mbpp"
EXPECTED_ROWS = 500
LANGUAGE_NAMES = {
"as": "Assamese",
"bn": "Bengali",
"gu": "Gujarati",
"hi": "Hindi",
"kn": "Kannada",
"ml": "Malayalam",
"mr": "Marathi",
"ne": "Nepali",
"or": "Odia",
"pa": "Punjabi",
"sa": "Sanskrit",
"ta": "Tamil",
"te": "Telugu",
"ur": "Urdu",
}
DEFAULT_LANGUAGES = tuple(LANGUAGE_NAMES)
SourceRow = Mapping[str, str | int | list[str]]


def _verifier_metadata(record: SourceRow) -> dict:
task_id = record.get("task_id")
tests = record.get("test_list")
setup = record.get("test_setup_code")
code = record.get("code")
if type(task_id) is not int or task_id < 1:
raise ValueError("task_id must be a positive integer")
if not isinstance(tests, list) or len(tests) != 3 or any(not isinstance(t, str) or not t.strip() for t in tests):
raise ValueError(f"Expected three nonempty original tests for task {task_id}")
if not isinstance(setup, str) or not isinstance(code, str):
raise ValueError(f"Missing setup or reference code for task {task_id}")
functions = {node.name for node in ast.parse(code).body if isinstance(node, ast.FunctionDef)}
calls = {
node.func.id
for node in ast.walk(ast.parse(tests[0]))
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
}
entry_points = functions & calls
if len(entry_points) != 1:
raise ValueError(f"Cannot identify one tested function for task {task_id}")
# The setup can depend on classes supplied by the candidate, so run it inside check().
# Preserve a candidate function named check before the runner installs its callback.
test = (
'def check(candidate, _original_check=globals().get("check")):\n'
" if _original_check is not None:\n"
' globals()["check"] = _original_check\n' + indent(setup + "\n" + "\n".join(tests), " ") + "\n"
)
ast.parse(test)
return {"task_id": f"Mbpp/{task_id}", "entry_point": entry_points.pop(), "test": test}


def build_rows(
records: Sequence[SourceRow],
*,
languages: Sequence[str] = DEFAULT_LANGUAGES,
task_ids: Sequence[int] | None = None,
) -> list[dict]:
"""Build translated prompts and unchanged original MBPP assertions."""
if (
isinstance(languages, str)
or not languages
or any(code not in (*DEFAULT_LANGUAGES, "en") for code in languages)
):
raise ValueError(f"languages must be a nonempty sequence from {(*DEFAULT_LANGUAGES, 'en')}")
if len(set(languages)) != len(languages):
raise ValueError("languages must be unique")
if len(records) != EXPECTED_ROWS:
raise ValueError(f"Expected {EXPECTED_ROWS} MBPP tasks, got {len(records)}")
metadata = {}
for record in records:
meta = _verifier_metadata(record)
identity = record["task_id"]
if identity in metadata:
raise ValueError(f"Duplicate task_id: {identity}")
metadata[identity] = meta
if task_ids is not None:
if not task_ids or any(type(value) is not int for value in task_ids) or len(set(task_ids)) != len(task_ids):
raise ValueError("task_ids must contain unique integer IDs")
if set(task_ids) - metadata.keys():
raise ValueError(f"Unknown task_ids: {sorted(set(task_ids) - metadata.keys())}")
selected = set(metadata if task_ids is None else task_ids)
rows = []
for language in languages:
column = "text" if language == "en" else f"text_{LANGUAGE_NAMES[language]}_translation"
for record in records:
identity = record["task_id"]
text = record.get(column)
if not isinstance(text, str) or not text.strip():
raise ValueError(f"Missing problem text: {language}/{identity}")
if identity not in selected:
continue
setup = record["test_setup_code"]
example = (setup + "\n" if setup else "") + record["test_list"][0]
question = ('"""\n' + text + "\n" + example + '\n"""\n').replace(" ", "\t")
rows.append(
{
"question": question,
"verifier_metadata": {**metadata[identity], "task_id": f"Mbpp/{language}/{identity}"},
"language": language,
"uuid": f"{SOURCE_ID}/{language}/{identity}",
}
)
return rows


def prepare(
*,
languages: Sequence[str] = DEFAULT_LANGUAGES,
task_ids: Sequence[int] | None = None,
output_fpath: str | None = None,
) -> Path:
"""Download the latest translations and write tasks plus source provenance."""
config = maybe_get_global_config_dict()
token = config.get(HF_TOKEN_KEY_NAME) if config is not None else None
source = hf_hub_download(
repo_id=SOURCE_ID,
filename="test.parquet",
repo_type="dataset",
token=token or get_token(),
)
Comment thread
yuhezhang-ai marked this conversation as resolved.
Resolved
records = load_dataset("parquet", data_files={"test": source}, split="test").to_list()
rows = build_rows(records, languages=languages, task_ids=task_ids)
# Do not resolve this symlink: the snapshot path carries the downloaded commit,
# while its target is a content-addressed blob. No second lookup of mutable HEAD.
source_path = Path(source)
revision = source_path.parent.name
provenance = {
"repo_id": SOURCE_ID,
"filename": "test.parquet",
"resolved_revision": revision
if source_path.parent.parent.name == "snapshots" and re.fullmatch(r"[0-9a-f]{40}", revision)
else None,
"sha256": hashlib.sha256(source_path.read_bytes()).hexdigest(),
"languages": list(languages),
"task_ids": list(task_ids) if task_ids is not None else None,
"rows": len(rows),
}
output = Path(output_fpath) if output_fpath else OUTPUT_FPATH
output.parent.mkdir(parents=True, exist_ok=True)
with output.open("w", encoding="utf-8") as stream:
for row in rows:
stream.write(json.dumps(row, ensure_ascii=False) + "\n")
provenance_path = output.with_suffix(".provenance.json")
provenance_path.write_text(json.dumps(provenance, indent=2) + "\n", encoding="utf-8")
print(f"Wrote {len(rows)} problems to {output}")
print(f"Dataset revision: {provenance['resolved_revision'] or 'unknown'}; provenance: {provenance_path}")
return output


if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--languages", nargs="+", default=DEFAULT_LANGUAGES)
parser.add_argument("--task-ids", nargs="+", type=int)
parser.add_argument("--output-fpath")
prepare(**vars(parser.parse_args()))
Loading
Loading