Repository navigation
cp: feat(pinchbench): add the benchmark config for the full 147-task suite (2259) into r0.5.0 - #2260
Closed
laszkiewiczp wants to merge 2 commits into
Closed
laszkiewiczp wants to merge 2 commits into
laszkiewiczp wants to merge 2 commits into
Conversation
Expand the translation benchmarks and standardize scoring around chrF, FLORES-200 spBLEU, COMET, and target-language consistency. - expand WMT24++ from five defaults to all 55 target locales - expand FLORES from the six-language pair matrix to 219 English-to-X devtest pairs across 220 supported FLORES+ configurations - store generated language names and FLORES devtest availability in checked-in metadata instead of querying or hardcoding them at runtime - preserve complete FLORES language and script codes in prepared rows - reject unknown or devtest-incompatible FLORES configurations - deduplicate WMT target locales while preserving their requested order - exclude WMT24++ rows marked as bad source data - publish WMT preparation output atomically after model prefetch succeeds - add CLI controls for COMET, spBLEU, and GlotLID prefetching Replace language-specific SacreBLEU tokenization with NFC-normalized chrF and FLORES-200 SentencePiece spBLEU scoring. Use sentence chrF as the dense RL reward and report sentence and corpus chrF/spBLEU separately. Emit a startup warning because the new spBLEU values are not comparable with the previous BLEU metrics. Add configurable target-language consistency scoring: - use CLD2 for WMT24++ locale-to-language validation - use GlotLID probabilities for FLORES languages and dialect mappings - load language-detection backends lazily and reuse them across requests - report per-rollout scores and per-pair, source, target, and global aggregate metrics - warn when a language pair falls below the configured consistency threshold - document CLD2 limitations for regional language variants Prefetch the shared SentencePiece, COMET, and GlotLID assets for offline verification. Update translation dependencies, configs, documentation, and example rollouts for the new scoring contract. Add metadata generation and mapping-validation utilities plus unit coverage for preparation, scoring, aggregation, backend loading, and language-code handling. BREAKING CHANGE: The wmt_translation per-rollout reward now uses sentence- level chrF instead of sentence-level BLEU; spBLEU remains a separately reported metric. The sentence_bleu field and */bleu aggregate keys are replaced by sentence_chrf, sentence_spbleu, */chrF, and */spBLEU. --------- Signed-off-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com> Co-authored-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com>
PinchBench shipped as an agent server (responses_api_agents/pinchbench) but had no entry under benchmarks/, so it was neither discoverable via `gym list benchmarks` nor preparable via `gym eval prepare`. Add the Layer-3 benchmark definition: config.yaml chains to the agent config and narrows it to the benchmark dataset via `_inherit_from` (the tau2 shape, the existing precedent for a benchmark chaining to a responses_api_agents config), and prepare.py clones the skill at v2.0.0 -- the ref Dockerfile.benchmark bakes into the per-task image -- to write all 147 manifest tasks. The eight run-time knobs the agent config interpolates are declared with `oc.env:...,null` defaults so `gym list` and `gym eval prepare` resolve without them; a null still reaching the agent fails fast, since the fields are required `str` in PinchBenchAgentConfig. CLI overrides keep working because Hydra parses them against an empty base before config_paths loads and merges them last. Preparation fails loudly if the manifest is not 147 tasks, or if the regenerated rows do not byte-match the committed example.jsonl, so upstream drift and divergence from dataset_preprocess.py cannot pass silently. Also declare domain/description/value on the agent config: listing metadata has to live on the base node, since adding it to the inheriting benchmark node raises `Key 'domain' is not in struct`. The README table row is regenerated by the update-readme-table hook. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> (cherry picked from commit 8efb861)
Contributor
Author
|
Superseded by #2261. This branch was based on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry pick of #2259 into
r0.5.0, opened in parallel with it rather than after merge — please review the two together.git cherry-pick -xof the source commit; it applies clean, with zero drift betweenmainandr0.5.0on every path this touches (responses_api_agents/pinchbench/,nemo_gym/benchmarks.py,nemo_gym/cli/eval.py,nemo_gym/config_types.py,benchmarks/.gitignore). The only merge was the single regeneratedREADME.mdtable row.r0.5.0is the right release target: it carriesresponses_api_agents/pinchbench, so the benchmark config'sconfig_pathschain resolves.r0.4.0does not — PinchBench landed onmainafter that branch was cut — so this cannot be backported further.Verification on this branch
Re-ran the full check against the
r0.5.0base, not justmain:ng_prepare_benchmark "+config_paths=[benchmarks/pinchbench/config.yaml]"— 147 rowsng_list_benchmarkslistspinchbench, domainagent,num_repeats3pre-commit runclean on all touched files, and theupdate-readme-tablehook is idempotent here (no further diff)Rationale for the config shape is in #2259; nothing was changed in the pick.
🤖 Generated with Claude Code