Skip to content

cp: feat(pinchbench): add the benchmark config for the full 147-task suite (2259) into r0.5.0 - #2260

Closed
laszkiewiczp wants to merge 2 commits into
r0.5.0from
cherry-pick-pinchbench-benchmark-config-r0.5.0
Closed

laszkiewiczp wants to merge 2 commits into
r0.5.0from
cherry-pick-pinchbench-benchmark-config-r0.5.0

Conversation

@laszkiewiczp

Copy link
Copy Markdown
Contributor

Cherry pick of #2259 into r0.5.0, opened in parallel with it rather than after merge — please review the two together.

git cherry-pick -x of the source commit; it applies clean, with zero drift between main and r0.5.0 on every path this touches (responses_api_agents/pinchbench/, nemo_gym/benchmarks.py, nemo_gym/cli/eval.py, nemo_gym/config_types.py, benchmarks/.gitignore). The only merge was the single regenerated README.md table row.

r0.5.0 is the right release target: it carries responses_api_agents/pinchbench, so the benchmark config's config_paths chain resolves. r0.4.0 does not — PinchBench landed on main after that branch was cut — so this cannot be backported further.

Verification on this branch

Re-ran the full check against the r0.5.0 base, not just main:

  • ng_prepare_benchmark "+config_paths=[benchmarks/pinchbench/config.yaml]" — 147 rows
  • ng_list_benchmarks lists pinchbench, domain agent, num_repeats 3
  • pre-commit run clean on all touched files, and the update-readme-table hook is idempotent here (no further diff)

Rationale for the config shape is in #2259; nothing was changed in the pick.

🤖 Generated with Claude Code

thompsonb and others added 2 commits July 30, 2026 13:19
Expand the translation benchmarks and standardize scoring around chrF,
  FLORES-200 spBLEU, COMET, and target-language consistency.

  - expand WMT24++ from five defaults to all 55 target locales
- expand FLORES from the six-language pair matrix to 219 English-to-X
devtest pairs across 220 supported FLORES+ configurations
- store generated language names and FLORES devtest availability in
checked-in metadata instead of querying or hardcoding them at runtime
  - preserve complete FLORES language and script codes in prepared rows
  - reject unknown or devtest-incompatible FLORES configurations
- deduplicate WMT target locales while preserving their requested order
  - exclude WMT24++ rows marked as bad source data
- publish WMT preparation output atomically after model prefetch
succeeds
  - add CLI controls for COMET, spBLEU, and GlotLID prefetching

Replace language-specific SacreBLEU tokenization with NFC-normalized
chrF
  and FLORES-200 SentencePiece spBLEU scoring. Use sentence chrF as the
  dense RL reward and report sentence and corpus chrF/spBLEU separately.
Emit a startup warning because the new spBLEU values are not comparable
  with the previous BLEU metrics.

  Add configurable target-language consistency scoring:

  - use CLD2 for WMT24++ locale-to-language validation
  - use GlotLID probabilities for FLORES languages and dialect mappings
- load language-detection backends lazily and reuse them across requests
- report per-rollout scores and per-pair, source, target, and global
aggregate metrics
- warn when a language pair falls below the configured consistency
threshold
  - document CLD2 limitations for regional language variants

Prefetch the shared SentencePiece, COMET, and GlotLID assets for offline
  verification. Update translation dependencies, configs, documentation,
  and example rollouts for the new scoring contract. Add metadata
  generation and mapping-validation utilities plus unit coverage for
  preparation, scoring, aggregation, backend loading, and language-code
  handling.

BREAKING CHANGE: The wmt_translation per-rollout reward now uses
sentence-
  level chrF instead of sentence-level BLEU; spBLEU remains a separately
  reported metric. The sentence_bleu field and */bleu aggregate keys are
  replaced by sentence_chrf, sentence_spbleu, */chrF, and */spBLEU.

---------

Signed-off-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com>
Co-authored-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com>
PinchBench shipped as an agent server (responses_api_agents/pinchbench) but had
no entry under benchmarks/, so it was neither discoverable via `gym list
benchmarks` nor preparable via `gym eval prepare`.

Add the Layer-3 benchmark definition: config.yaml chains to the agent config and
narrows it to the benchmark dataset via `_inherit_from` (the tau2 shape, the
existing precedent for a benchmark chaining to a responses_api_agents config),
and prepare.py clones the skill at v2.0.0 -- the ref Dockerfile.benchmark bakes
into the per-task image -- to write all 147 manifest tasks.

The eight run-time knobs the agent config interpolates are declared with
`oc.env:...,null` defaults so `gym list` and `gym eval prepare` resolve without
them; a null still reaching the agent fails fast, since the fields are required
`str` in PinchBenchAgentConfig. CLI overrides keep working because Hydra parses
them against an empty base before config_paths loads and merges them last.

Preparation fails loudly if the manifest is not 147 tasks, or if the regenerated
rows do not byte-match the committed example.jsonl, so upstream drift and
divergence from dataset_preprocess.py cannot pass silently.

Also declare domain/description/value on the agent config: listing metadata has
to live on the base node, since adding it to the inheriting benchmark node
raises `Key 'domain' is not in struct`. The README table row is regenerated by
the update-readme-table hook.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
(cherry picked from commit 8efb861)
@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@laszkiewiczp

Copy link
Copy Markdown
Contributor Author

Superseded by #2261.

This branch was based on main rather than r0.5.0, so it carried an unrelated second commit (902b98510) whose sign-off does not match its author — that is the DCO failure here, not the PinchBench commit. The repo ruleset blocks force-push and branch deletion on all branches, so it could not be fixed in place; #2261 is the same change on a correctly based single-commit branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants