Repository navigation
feat(indic mbpp): add Indic MBPP #3644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
3c2b8f8
feat(benchmarks): add Indic MBPP with original tests and English base…
anushaknvidia 07346cf
remove dataset version
0765a45
fix(benchmarks): address Indic MBPP reproducibility review
anushaknvidia 7bb48b6
fix(benchmarks): align Indic MBPP with main environment routing
anushaknvidia File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> | ||
| <!-- SPDX-License-Identifier: Apache-2.0 --> | ||
|
|
||
| # Indic MBPP | ||
|
|
||
| Original MBPP from `ai4bharat/indic-mbpp`: 500 Python tasks in each of 14 Indic | ||
| languages. Includes an English baseline (`en`) from the same source records. | ||
| Reuses Gym's [English MBPP prompt](../../mbpp/prompts/default.yaml), `simple_agent`, | ||
| and the existing [SciCodePile assertion verifier](../../../resources_servers/scicodepile/README.md). | ||
| The `indic_mbpp_environment_server` uses Gym's `legacy_agent` relay to route | ||
| episodes to `indic_mbpp_simple_agent`, following the upstream benchmark layout. | ||
| The relay does not change the prompt or scoring contract. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Each prompt contains the translated description, task setup when present, and the | ||
| first original assertion as an example. Four spaces are replaced with a tab, as | ||
| in English MBPP preparation. Reference solutions and the other two assertions are | ||
| kept out of the prompt. Each response must provide complete Python code. | ||
|
|
||
| Reward is 1 only when all three original `test_list` assertions pass. The existing | ||
| verifier runs each response in a fresh subprocess with a 120-second timeout. | ||
| `challenge_test_list` is excluded from the standard score. Default repeats: 1. | ||
| Use `pass@1/accuracy` for a single attempt; repeated runs also report pass@k and | ||
| `pass@1[avg-of-k]/accuracy`. Inspect coverage and `harness_failure` alongside scores. | ||
| Saved verifier results retain a language-qualified `task_id`, such as | ||
| `Mbpp/hi/11`, so results can be grouped by language without joining the inputs. | ||
| The top-level input `language` and `uuid` are not returned by SciCodePile. | ||
| Older prepared files use `Mbpp/11`; regenerate them to obtain qualified IDs. | ||
| This metadata change does not change prompts, assertions, or rewards. | ||
|
|
||
| Gym's existing `mbpp` benchmark is MBPP+ (378 EvalPlus tasks), while this dataset | ||
| contains the 500 original MBPP test tasks. Their scores are different benchmarks: | ||
| only 224 task IDs overlap, and 115 of those have revised English descriptions. | ||
| Use `indic/mbpp` with `languages: [en]` for a comparison on the same 500 tasks, | ||
| with identical model and generation settings. The existing `mbpp` benchmark and | ||
| its EvalPlus base/plus metrics are unchanged. | ||
|
|
||
| ## Prepare data | ||
|
|
||
| Install Gym and follow the shared verifier's execution requirements. Generated | ||
| code runs as a subprocess; use an isolated evaluation environment as described | ||
| in the verifier README. | ||
| The source repository requires approved Hugging Face access. Authenticate with | ||
| `hf auth login` (or supply `HF_TOKEN`) using an account that can read | ||
| `ai4bharat/indic-mbpp`; authentication alone does not grant dataset access. | ||
|
|
||
| ```bash | ||
| gym eval prepare --benchmark indic/mbpp | ||
| ``` | ||
|
|
||
| Defaults to `as`, `bn`, `gu`, `hi`, `kn`, `ml`, `mr`, `ne`, `or`, `pa`, `sa`, `ta`, | ||
| `te`, and `ur` (7,000 tasks). For per-language results, prepare and run one language | ||
| at a time; use `en` for the matching baseline: | ||
|
|
||
| ```bash | ||
| gym eval prepare --benchmark indic/mbpp \ | ||
| '+prepare_script_args={languages:[hi]}' \ | ||
| +use_cached_prepared_benchmarks=false | ||
| ``` | ||
|
|
||
| Use `task_ids` in `prepare_script_args` to select smoke-test tasks. Data is downloaded | ||
| from the dataset's default revision. Missing translations are rejected and generated | ||
| data remains local. | ||
| Preparation also writes an adjacent `.provenance.json` file recording the resolved | ||
| dataset commit, source parquet SHA-256, selected languages/task IDs, and row count. | ||
| The revision comes from the downloaded snapshot path, not a separate lookup of | ||
| the latest commit. Non-snapshot paths are recorded as `null`, with the checksum | ||
| still retained. No source revision is pinned. Keep this sidecar with the inputs | ||
| and results; cached prepared data does not automatically refresh upstream changes. | ||
|
|
||
| ## Collect rollouts | ||
|
|
||
| ```bash | ||
| gym eval run --benchmark indic/mbpp \ | ||
| --model-type vllm_model \ | ||
| --model MODEL_NAME \ | ||
| --model-url http://HOST:PORT/v1 \ | ||
| --model-api-key dummy \ | ||
| --split benchmark \ | ||
| --output results/indic_mbpp/rollouts.jsonl | ||
| ``` | ||
|
|
||
| The canonical config does not set sampling, thinking, a seed, or an output-token | ||
| limit. Specify these in the model/run configuration, and use the same settings | ||
| for each language and the English baseline. The verifier timeout is not a model | ||
| generation-token limit. | ||
|
|
||
| ## Paired real-model smoke | ||
|
|
||
| The standalone [smoke config](smoke.yaml) selects one attempt each for two task | ||
| IDs in English, Hindi, and Kannada after preparing the inputs below. It uses | ||
| `google/gemma-4-31B-it`, thinking enabled, temperature 1.0, top-p 0.95, top-k 64, | ||
| seed 0, and at most 8,192 output tokens (including reasoning). | ||
| See the [inspected smoke results](smoke-results.md) for the source/model revisions, | ||
| server settings, outputs, and rewards. This small check is not a benchmark score. | ||
|
|
||
| ```bash | ||
| python -m benchmarks.indic.mbpp.prepare \ | ||
| --languages en hi kn --task-ids 11 367 \ | ||
| --output-fpath benchmarks/indic/mbpp/data/review-smoke.jsonl | ||
|
|
||
| gym eval run --config benchmarks/indic/mbpp/smoke.yaml \ | ||
| --agent indic_mbpp_simple_agent \ | ||
| --model google/gemma-4-31B-it \ | ||
| --model-url http://HOST:PORT/v1 --model-api-key dummy \ | ||
| --split benchmark --concurrency 3 \ | ||
| --output results/indic_mbpp_smoke/rollouts.jsonl | ||
| ``` | ||
|
|
||
| Start a compatible vLLM endpoint with the `gemma4` reasoning parser. Do not append | ||
| `--model-type vllm_model` to this smoke command: the config already includes the | ||
| adapter, and reloading its defaults afterward can erase the sampling overrides. | ||
| Sampling/seed and thinking are scoped to the smoke model adapter, not global | ||
| request metadata. The smoke has one repeat; use a separate run configuration | ||
| with independent seeds when estimating multi-attempt metrics. | ||
|
|
||
| The dataset declares Apache-2.0. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| config_paths: | ||
| - resources_servers/scicodepile/configs/scicodepile.yaml | ||
|
|
||
| indic_mbpp_resources_server: | ||
| _inherit_from: scicodepile | ||
|
|
||
| indic_mbpp_simple_agent: | ||
| _inherit_from: scicodepile_simple_agent | ||
| responses_api_agents: | ||
| simple_agent: | ||
| resources_server: | ||
| name: indic_mbpp_resources_server | ||
| datasets: | ||
| - name: indic_mbpp | ||
| type: benchmark | ||
| jsonl_fpath: benchmarks/indic/mbpp/data/mbpp_benchmark.jsonl | ||
| prepare_script: benchmarks/indic/mbpp/prepare.py | ||
| prompt_config: benchmarks/mbpp/prompts/default.yaml | ||
| num_repeats: 1 | ||
| license: Apache-2.0 | ||
|
|
||
| indic_mbpp_environment_server: | ||
| _inherit_from: scicodepile_environment_server | ||
| environment_servers: | ||
| legacy_agent: | ||
| agent_server: | ||
| name: indic_mbpp_simple_agent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| * | ||
| !.gitignore |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| name: indic/mbpp | ||
| version: 0.1.0 | ||
| experimental: true | ||
| kind: benchmark | ||
| integration_profile: custom-gym-verifier | ||
| domain: coding | ||
| description: Original MBPP in 14 Indic languages, with unchanged assertion tests and a matching English baseline. | ||
| modality: text | ||
| licensing: Apache-2.0 | ||
| authors: [NVIDIA NeMo Gym contributors] | ||
| reward: | ||
| range: [0, 1] | ||
| higher_is_better: true | ||
| determinism: stochastic | ||
| resources_server: scicodepile | ||
| agent_server: simple_agent | ||
| model_server: policy_model | ||
| datasets: | ||
| - name: indic_mbpp | ||
| type: benchmark | ||
| jsonl_fpath: benchmarks/indic/mbpp/data/mbpp_benchmark.jsonl | ||
| prepare_script: benchmarks/indic/mbpp/prepare.py | ||
| prompt_config: benchmarks/mbpp/prompts/default.yaml | ||
| num_repeats: 1 | ||
| standard_prompt_config: benchmarks/mbpp/prompts/default.yaml | ||
| canonical_split: benchmark | ||
| session_model: episode | ||
| state: none | ||
| lifecycle: active |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| """Prepare original MBPP translations and a matching English baseline.""" | ||
|
|
||
| import argparse | ||
| import ast | ||
| import hashlib | ||
| import json | ||
| import re | ||
| from collections.abc import Mapping, Sequence | ||
| from pathlib import Path | ||
| from textwrap import indent | ||
|
|
||
| from datasets import load_dataset | ||
| from huggingface_hub import get_token, hf_hub_download | ||
|
|
||
| from nemo_gym.global_config import HF_TOKEN_KEY_NAME, maybe_get_global_config_dict | ||
|
|
||
|
|
||
| BENCHMARK_DIR = Path(__file__).parent | ||
| OUTPUT_FPATH = BENCHMARK_DIR / "data" / "mbpp_benchmark.jsonl" | ||
| SOURCE_ID = "ai4bharat/indic-mbpp" | ||
| EXPECTED_ROWS = 500 | ||
| LANGUAGE_NAMES = { | ||
| "as": "Assamese", | ||
| "bn": "Bengali", | ||
| "gu": "Gujarati", | ||
| "hi": "Hindi", | ||
| "kn": "Kannada", | ||
| "ml": "Malayalam", | ||
| "mr": "Marathi", | ||
| "ne": "Nepali", | ||
| "or": "Odia", | ||
| "pa": "Punjabi", | ||
| "sa": "Sanskrit", | ||
| "ta": "Tamil", | ||
| "te": "Telugu", | ||
| "ur": "Urdu", | ||
| } | ||
| DEFAULT_LANGUAGES = tuple(LANGUAGE_NAMES) | ||
| SourceRow = Mapping[str, str | int | list[str]] | ||
|
|
||
|
|
||
| def _verifier_metadata(record: SourceRow) -> dict: | ||
| task_id = record.get("task_id") | ||
| tests = record.get("test_list") | ||
| setup = record.get("test_setup_code") | ||
| code = record.get("code") | ||
| if type(task_id) is not int or task_id < 1: | ||
| raise ValueError("task_id must be a positive integer") | ||
| if not isinstance(tests, list) or len(tests) != 3 or any(not isinstance(t, str) or not t.strip() for t in tests): | ||
| raise ValueError(f"Expected three nonempty original tests for task {task_id}") | ||
| if not isinstance(setup, str) or not isinstance(code, str): | ||
| raise ValueError(f"Missing setup or reference code for task {task_id}") | ||
| functions = {node.name for node in ast.parse(code).body if isinstance(node, ast.FunctionDef)} | ||
| calls = { | ||
| node.func.id | ||
| for node in ast.walk(ast.parse(tests[0])) | ||
| if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) | ||
| } | ||
| entry_points = functions & calls | ||
| if len(entry_points) != 1: | ||
| raise ValueError(f"Cannot identify one tested function for task {task_id}") | ||
| # The setup can depend on classes supplied by the candidate, so run it inside check(). | ||
| # Preserve a candidate function named check before the runner installs its callback. | ||
| test = ( | ||
| 'def check(candidate, _original_check=globals().get("check")):\n' | ||
| " if _original_check is not None:\n" | ||
| ' globals()["check"] = _original_check\n' + indent(setup + "\n" + "\n".join(tests), " ") + "\n" | ||
| ) | ||
| ast.parse(test) | ||
| return {"task_id": f"Mbpp/{task_id}", "entry_point": entry_points.pop(), "test": test} | ||
|
|
||
|
|
||
| def build_rows( | ||
| records: Sequence[SourceRow], | ||
| *, | ||
| languages: Sequence[str] = DEFAULT_LANGUAGES, | ||
| task_ids: Sequence[int] | None = None, | ||
| ) -> list[dict]: | ||
| """Build translated prompts and unchanged original MBPP assertions.""" | ||
| if ( | ||
| isinstance(languages, str) | ||
| or not languages | ||
| or any(code not in (*DEFAULT_LANGUAGES, "en") for code in languages) | ||
| ): | ||
| raise ValueError(f"languages must be a nonempty sequence from {(*DEFAULT_LANGUAGES, 'en')}") | ||
| if len(set(languages)) != len(languages): | ||
| raise ValueError("languages must be unique") | ||
| if len(records) != EXPECTED_ROWS: | ||
| raise ValueError(f"Expected {EXPECTED_ROWS} MBPP tasks, got {len(records)}") | ||
| metadata = {} | ||
| for record in records: | ||
| meta = _verifier_metadata(record) | ||
| identity = record["task_id"] | ||
| if identity in metadata: | ||
| raise ValueError(f"Duplicate task_id: {identity}") | ||
| metadata[identity] = meta | ||
| if task_ids is not None: | ||
| if not task_ids or any(type(value) is not int for value in task_ids) or len(set(task_ids)) != len(task_ids): | ||
| raise ValueError("task_ids must contain unique integer IDs") | ||
| if set(task_ids) - metadata.keys(): | ||
| raise ValueError(f"Unknown task_ids: {sorted(set(task_ids) - metadata.keys())}") | ||
| selected = set(metadata if task_ids is None else task_ids) | ||
| rows = [] | ||
| for language in languages: | ||
| column = "text" if language == "en" else f"text_{LANGUAGE_NAMES[language]}_translation" | ||
| for record in records: | ||
| identity = record["task_id"] | ||
| text = record.get(column) | ||
| if not isinstance(text, str) or not text.strip(): | ||
| raise ValueError(f"Missing problem text: {language}/{identity}") | ||
| if identity not in selected: | ||
| continue | ||
| setup = record["test_setup_code"] | ||
| example = (setup + "\n" if setup else "") + record["test_list"][0] | ||
| question = ('"""\n' + text + "\n" + example + '\n"""\n').replace(" ", "\t") | ||
| rows.append( | ||
| { | ||
| "question": question, | ||
| "verifier_metadata": {**metadata[identity], "task_id": f"Mbpp/{language}/{identity}"}, | ||
| "language": language, | ||
| "uuid": f"{SOURCE_ID}/{language}/{identity}", | ||
| } | ||
| ) | ||
| return rows | ||
|
|
||
|
|
||
| def prepare( | ||
| *, | ||
| languages: Sequence[str] = DEFAULT_LANGUAGES, | ||
| task_ids: Sequence[int] | None = None, | ||
| output_fpath: str | None = None, | ||
| ) -> Path: | ||
| """Download the latest translations and write tasks plus source provenance.""" | ||
| config = maybe_get_global_config_dict() | ||
| token = config.get(HF_TOKEN_KEY_NAME) if config is not None else None | ||
| source = hf_hub_download( | ||
| repo_id=SOURCE_ID, | ||
| filename="test.parquet", | ||
| repo_type="dataset", | ||
| token=token or get_token(), | ||
| ) | ||
| records = load_dataset("parquet", data_files={"test": source}, split="test").to_list() | ||
| rows = build_rows(records, languages=languages, task_ids=task_ids) | ||
| # Do not resolve this symlink: the snapshot path carries the downloaded commit, | ||
| # while its target is a content-addressed blob. No second lookup of mutable HEAD. | ||
| source_path = Path(source) | ||
| revision = source_path.parent.name | ||
| provenance = { | ||
| "repo_id": SOURCE_ID, | ||
| "filename": "test.parquet", | ||
| "resolved_revision": revision | ||
| if source_path.parent.parent.name == "snapshots" and re.fullmatch(r"[0-9a-f]{40}", revision) | ||
| else None, | ||
| "sha256": hashlib.sha256(source_path.read_bytes()).hexdigest(), | ||
| "languages": list(languages), | ||
| "task_ids": list(task_ids) if task_ids is not None else None, | ||
| "rows": len(rows), | ||
| } | ||
| output = Path(output_fpath) if output_fpath else OUTPUT_FPATH | ||
| output.parent.mkdir(parents=True, exist_ok=True) | ||
| with output.open("w", encoding="utf-8") as stream: | ||
| for row in rows: | ||
| stream.write(json.dumps(row, ensure_ascii=False) + "\n") | ||
| provenance_path = output.with_suffix(".provenance.json") | ||
| provenance_path.write_text(json.dumps(provenance, indent=2) + "\n", encoding="utf-8") | ||
| print(f"Wrote {len(rows)} problems to {output}") | ||
| print(f"Dataset revision: {provenance['resolved_revision'] or 'unknown'}; provenance: {provenance_path}") | ||
| return output | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| parser.add_argument("--languages", nargs="+", default=DEFAULT_LANGUAGES) | ||
| parser.add_argument("--task-ids", nargs="+", type=int) | ||
| parser.add_argument("--output-fpath") | ||
| prepare(**vars(parser.parse_args())) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.