Skip to content

Commit fc6772c

Browse files
committed
Merge remote-tracking branch 'origin/main' into smohsenitahe/cp-image-tools-pivot-terminal-to-main
Signed-off-by: Danial Mohseni Taheri <smohsenitahe@nvidia.com>
2 parents 1013cdf + 0da3b9a commit fc6772c

592 files changed

Lines changed: 64788 additions & 4120 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/add-benchmark/references/patterns.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ from nemo_gym.base_resources_server import (
2525
SimpleResourcesServer, BaseResourcesServerConfig,
2626
BaseVerifyRequest, BaseVerifyResponse,
2727
)
28+
from nemo_gym.server_utils import is_nemo_gym_fastapi_entrypoint
2829

2930
class MyConfig(BaseResourcesServerConfig):
3031
pass
@@ -40,6 +41,9 @@ class MyServer(SimpleResourcesServer):
4041

4142
if __name__ == "__main__":
4243
MyServer.run_webserver()
44+
elif is_nemo_gym_fastapi_entrypoint(__file__):
45+
# With num_workers > 1, uvicorn imports this module in each worker and serves its module-level `app`.
46+
app = MyServer.run_webserver() # noqa: F401
4347
```
4448

4549
### Subprocess execution server (e.g. `code_gen`)

‎.agents/skills/pr-review/SKILL.md‎

Lines changed: 235 additions & 19 deletions
Large diffs are not rendered by default.

‎.github/autobot/review-config.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
auto-review:
33
enabled: true
44
mode: comment
5-
on: status_change
5+
on: [opened, status_change]
66
when: ready_for_review
77

88
approval:

‎.github/workflows/config/.secrets.baseline‎

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.gitignore‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,5 +243,13 @@ resources_servers/cvdp/data/gym_cvdp_nonagentic_code_generation_no_commercial.js
243243
# Generated Terminal-Bench 4 dataset
244244
/benchmarks/terminal_bench_4/data/
245245

246-
# Default shared datasets folder for Harbor-format tasks (see nemo_gym/tasks/harbor).
246+
# The default pulled datasets folder before environments/harbor/ (see below); kept ignored for checkouts that
247+
# still hold datasets there (environment_servers/harbor/README.md, "Configuration and artifacts", says how to move them).
247248
datasets/
249+
250+
# Pulled Harbor datasets (nemo_gym/tasks/harbor/hub.py) land in environments/harbor/<dataset>/ next to the
251+
# committed per-dataset entries, which keep only their README and YAML files.
252+
environments/harbor/*/*
253+
!environments/harbor/*/README.md
254+
!environments/harbor/*/*.yaml
255+
environments/harbor/.harbor/

‎AGENTS.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,15 @@ Humans: see [Development Setup → Use of AI and LLM Tools](https://docs.nvidia.
2020
- Docs live under `fern/versions/latest/pages/`. Bleeding-edge nav is `fern/versions/main.yml`. See `fern/README.md` and the `nemo-gym-docs` skill.
2121
- Do not introduce licenses incompatible with Apache-2.0. New source files need the standard NVIDIA SPDX header.
2222

23+
## Code provenance
24+
25+
- Avoid copying or adapting code from other repositories. Write an independent implementation whenever possible.
26+
- When copying or adapting external code is necessary, you **must** record the source URL, exact revision (commit SHA),
27+
and applicable license in **both the PR description and the header of every file containing that code**. Identify
28+
the original source file(s) and the affected Gym files in the PR description.
29+
- Retain all required copyright notices, license notices, and attribution. Follow the Apache-2.0 compatibility and
30+
vendoring requirements in [CONTRIBUTING.md](CONTRIBUTING.md#licensing-of-contributions).
31+
2332
## Pull Requests
2433

2534
- For normal authored PRs, use a Conventional Commit-style title: `type(optional-scope): imperative summary`.

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
249249
| Asr With Pc | other | ASR with WER scoring (standard, case-sensitive, punctuation+capitalization) | Improve transcription quality with structural detail | - | - | - | <a href='resources_servers/asr_with_pc/configs/asr_with_pc.yaml'>asr_with_pc.yaml</a> | - |
250250
| Assaybench | knowledge | AssayBench -- rank the hit genes of a CRISPR screen from its plain-text description (arXiv:2605.10876) | Tests whether a model can link mechanistic biology to measured phenotypic outcomes, scored with the paper's adjusted nDCG | - | - | - | <a href='resources_servers/assaybench/configs/assaybench.yaml'>assaybench.yaml</a> | - |
251251
| Bbq | safety | BBQ comparative QA with Answer and Explanation Quality checks | Evidence-grounded and fairness-safe comparative reasoning | - | - | - | <a href='resources_servers/bbq/configs/bbq.yaml'>bbq.yaml</a> | - |
252+
| Beam | long_context | BEAM (Beyond a Million Tokens) long-term-memory probing questions over long multi-session chats, scored by an LLM judge with the upstream per-question rubrics. | Improve long-term conversational memory (recall, updates, contradictions, temporal and multi-session reasoning). | - | - | - | <a href='resources_servers/beam/configs/beam.yaml'>beam.yaml</a> | - |
252253
| Bigcodebench | coding | Verifies model-generated Python solutions against the BigCodeBench unittest suite. | Improve practical, library-rich Python coding capabilities. | - | - | - | <a href='resources_servers/bigcodebench/configs/bigcodebench.yaml'>bigcodebench.yaml</a> | - |
253254
| Bird Sql | coding | Text-to-SQL with execution-based evaluation on BIRD dev (1534 SQLite tasks). Binary reward from unordered result-set equality. | Improve text-to-SQL capabilities on BIRD's realistic dev split using execution-based binary reward without an LLM judge. | - | - | - | <a href='resources_servers/bird_sql/configs/bird_sql.yaml'>bird_sql.yaml</a> | - |
254255
| Blackjack | games | Blackjack. Model hits or stands. Reward +1 win, 0 draw, -1 loss/bust. | Example gymnasium-style multi-step environment | - | - | - | <a href='resources_servers/blackjack/configs/blackjack.yaml'>blackjack.yaml</a> | - |
@@ -372,6 +373,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
372373
| Reasoning Gym | knowledge | LangGraph reflection agent compatible with resource servers that do not use tools; provides iterative reflection for diverse agent training data and test time scaling, extensible to use tools or other agent architectures | Iterative test time scaling for improved performance in reasoning tasks | ✓ | - | Apache 2.0 | <a href='resources_servers/reasoning_gym/configs/reflection_agent.yaml'>reflection_agent.yaml</a> | - |
373374
| Reasoning Gym | knowledge | LangGraph ReWOO agent compatible with resource servers that do not use tools; enables diverse agent training data and test time scaling vs a simple agent, extensible to use tools or other agent architectures | Iterative test time scaling for improved performance in reasoning tasks | ✓ | - | Apache 2.0 | <a href='resources_servers/reasoning_gym/configs/rewoo_agent.yaml'>rewoo_agent.yaml</a> | - |
374375
| Reasoning Gym | knowledge | Over 100 tasks including algebra, arithmetic, computation, cognition, geometry, graph theory, logic, and many common games. | Improve robustness, generalization, broad knowledge and reasoning | ✓ | - | Creative Commons Attribution 4.0 International | <a href='resources_servers/reasoning_gym/configs/reasoning_gym.yaml'>reasoning_gym.yaml</a> | <a href='https://huggingface.co/datasets/nvidia/Nemotron-RL-ReasoningGym-v1'>Nemotron-RL-ReasoningGym-v1</a> |
376+
| Rna Design | agent | RNA inverse folding: the design tools served over MCP to a harness such as Claude Code. | - | - | - | - | <a href='resources_servers/rna_design/configs/rna_design.yaml'>rna_design.yaml</a> | - |
375377
| Rolemrc | instruction_following | RoleMRC role-play MRC scored with reference metrics (ROUGE/BLEU/METEOR/BERTScore); ROUGE-L is the reward. | Improve role-play instruction following and knowledge-grounded responses. | - | ✓ | - | <a href='resources_servers/rolemrc/configs/rolemrc.yaml'>rolemrc.yaml</a> | - |
376378
| Rolemrc | rlhf | RoleMRC role-play MRC scored by a 5-aspect LLM-as-judge (knowledge range, style, nested/multi-turn instruction, instruction priority). | Improve role-play instruction following under knowledge-range and instruction-priority constraints. | - | ✓ | - | <a href='resources_servers/rolemrc/configs/rolemrc_judge.yaml'>rolemrc_judge.yaml</a> | - |
377379
| Ruler | other | - | - | - | - | - | <a href='resources_servers/ruler/configs/ruler.yaml'>ruler.yaml</a> | - |

‎benchmarks/apex25/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ gym eval prepare --benchmark apex25
3333
```
3434

3535
Writes `data/apex25_benchmark.jsonl` with one row per problem:
36-
`{"question": "...", "expected_answer": "..."}`. The HuggingFace dataset
36+
`{"task_id": "...", "question": "...", "expected_answer": "..."}`, where `task_id` is the dataset's `problem_idx`. The HuggingFace dataset
3737
revision is pinned in `prepare.py` (`HF_REVISION`) for reproducibility.
3838

3939
## Running servers

‎benchmarks/apex25/prepare.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ def prepare() -> Path:
4848
with open(OUTPUT_FPATH, "w") as f:
4949
for row in ds:
5050
out = {
51+
"task_id": str(row["problem_idx"]),
5152
"question": row["problem"],
5253
"expected_answer": str(row["answer"]),
5354
}

0 commit comments

Comments
 (0)