Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md

Large diffs are not rendered by default.

45 changes: 26 additions & 19 deletions benchmarks/flores200/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
# FLORES-200 Translation Benchmark

Multilingual segment-level translation across the default six-language set
(`en`, `de`, `es`, `fr`, `it`, `ja`) from the
English-to-X segment-level translation from `eng_Latn` into every FLORES+
configuration in the bundled metadata that provides the
[`openlanguagedata/flores_plus`](https://huggingface.co/datasets/openlanguagedata/flores_plus)
`devtest` split. The benchmark interleaves all 30 directed pairs (6 sources ×
6 targets, minus self-pairs) yielding 30 360 rows
(30 × 1012 sentences).
`devtest` split. The current metadata contains 220 such configurations, so the
benchmark interleaves 219 directed pairs and yields 221 628 rows
(219 × 1012 sentences).

Verification reuses the shared `wmt_translation` resource server: deterministic
corpus-level BLEU (sacrebleu) per language pair plus the cross-pair
aggregations `xx->xx`, `<src>->xx`, `xx->{tgt}`, optionally augmented by
xCOMET-XXL neural QE when `compute_comet: true` is set on the server.
corpus-level spBLEU and chrF (sacrebleu) per language pair plus the cross-pair
aggregations `xx->xx`, `<src>->xx`, `xx->{tgt}`. spBLEU consistently uses
the `flores200` SentencePiece tokenizer. Metrics are optionally augmented by
xCOMET-XXL neural QE when `compute_comet: true` is set on the server. Each
rollout also reports the probability GlotLID assigns to its expected target
language, aggregated under `/language_consistency` metric keys.

See `resources_servers/wmt_translation/README.md` for verifier details and
the Ray GPU-scheduled COMET path.
Expand All @@ -21,24 +24,28 @@ the Ray GPU-scheduled COMET path.
gym eval prepare --benchmark flores200
```

Writes `data/flores200_devtest_benchmark.jsonl` and pre-fetches the
xCOMET-XXL checkpoint plus its xlm-roberta-xxl tokenizer into `HF_HOME`
(when `unbabel-comet` is installed in the active env). That keeps the
resource server's Ray actors fully offline at runtime — no HF Hub calls
during `verify()`, no rate-limit retries.
Writes `data/flores200_devtest_benchmark.jsonl` and pre-fetches SacreBLEU's
FLORES-200 SentencePiece model into its cache, the xCOMET-XXL checkpoint plus
its xlm-roberta-xxl tokenizer into `HF_HOME` (when `unbabel-comet` is installed
in the active env), and the GlotLID model used for language-consistency
scoring. That keeps verification fully offline at runtime.

The default downloads six FLORES+ language configs (`eng_Latn`, `deu_Latn`,
`spa_Latn`, `fra_Latn`, `ita_Latn`, `jpn_Jpan`) and produces a JSONL whose
order, fields, and contents are byte-equivalent to the reference
[NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills) prepare script
(`nemo_skills/dataset/flores200/prepare.py`) under the same defaults.
Each row stores complete FLORES+ codes in `source_language` and
`target_language`, plus their English display names in `source_lang_name` and
`target_lang_name`.

The checked-in `data/flores_language_names.json` and
`data/flores_devtest_languages.json` files are generated by
`resources_servers/wmt_translation/scripts/generate_flores_language_metadata.py`.
Preparation reads these files locally, so it does not make per-language
metadata requests. Regenerate them deliberately when updating FLORES+.

## Running servers

The xCOMET-XXL actor pool requires the `extra_gpu` Ray resource which is
only advertised on multi-node SLURM deployments via NeMo-Skills'
`get_ray_server_cmd`. Local / single-node runs disable COMET via Hydra
override and rely on corpus-BLEU only:
override and rely on local spBLEU and chrF:

```bash
gym env start \
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/flores200/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ config_paths:

flores200_wmt_translation_resources_server:
_inherit_from: wmt_translation
resources_servers:
wmt_translation:
language_consistency_backend: flores_glotlid

flores200_wmt_translation_simple_agent:
_inherit_from: wmt_translation_simple_agent
Expand Down
10 changes: 5 additions & 5 deletions benchmarks/flores200/data/example_rollouts.jsonl

Large diffs are not rendered by default.

222 changes: 222 additions & 0 deletions benchmarks/flores200/data/flores_devtest_languages.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
[
"ace_Arab",
"ace_Latn",
"acm_Arab",
"acq_Arab",
"aeb_Arab",
"afr_Latn",
"als_Latn",
"amh_Ethi",
"apc_Arab_nort3139",
"apc_Arab_sout3123",
"apd_Arab",
"arb_Arab",
"arb_Latn",
"arg_Latn",
"ars_Arab",
"ary_Arab",
"arz_Arab",
"asm_Beng",
"ast_Latn",
"awa_Deva",
"ayr_Latn",
"azb_Arab",
"azj_Latn",
"bak_Cyrl",
"bam_Latn",
"ban_Latn",
"bel_Cyrl",
"bem_Latn",
"ben_Beng",
"bho_Deva",
"bjn_Arab",
"bjn_Latn",
"bod_Tibt",
"bos_Latn",
"bug_Latn",
"bul_Cyrl",
"cat_Latn",
"cat_Latn_vale1252",
"ceb_Latn",
"ces_Latn",
"chv_Cyrl",
"cjk_Latn",
"ckb_Arab",
"cmn_Hans",
"cmn_Hant",
"crh_Latn",
"cym_Latn",
"dan_Latn",
"deu_Latn",
"dik_Latn",
"dyu_Latn",
"dzo_Tibt",
"ekk_Latn",
"ell_Grek",
"eng_Latn",
"epo_Latn",
"eus_Latn",
"ewe_Latn",
"fao_Latn",
"fij_Latn",
"fil_Latn",
"fin_Latn",
"fon_Latn",
"fra_Latn",
"fur_Latn",
"fuv_Latn",
"gaz_Latn",
"gla_Latn",
"gle_Latn",
"glg_Latn",
"gug_Latn",
"guj_Gujr",
"hat_Latn",
"hau_Latn",
"heb_Hebr",
"hin_Deva",
"hne_Deva",
"hrv_Latn",
"hun_Latn",
"hye_Armn",
"ibo_Latn",
"ilo_Latn",
"ind_Latn",
"isl_Latn",
"ita_Latn",
"jav_Latn",
"jpn_Jpan",
"kaa_Latn",
"kab_Latn",
"kac_Latn",
"kam_Latn",
"kan_Knda",
"kas_Arab",
"kas_Deva",
"kat_Geor",
"kaz_Cyrl",
"kbp_Latn",
"kea_Latn",
"khk_Cyrl",
"khk_Mong",
"khm_Khmr",
"kik_Latn",
"kin_Latn",
"kir_Cyrl",
"kjh_Cyrl",
"kmb_Latn",
"kmr_Latn",
"knc_Arab",
"knc_Latn",
"kor_Hang",
"ktu_Latn",
"lao_Laoo",
"lij_Latn",
"lim_Latn",
"lin_Latn",
"lit_Latn",
"lld_Latn",
"lmo_Latn",
"ltg_Latn",
"ltz_Latn",
"lua_Latn",
"lug_Latn",
"luo_Latn",
"lus_Latn",
"lvs_Latn",
"mag_Deva",
"mai_Deva",
"mal_Mlym",
"mar_Deva",
"mfe_Latn",
"mhr_Cyrl",
"min_Arab",
"min_Latn",
"mkd_Cyrl",
"mlt_Latn",
"mni_Beng",
"mos_Latn",
"mri_Latn",
"mya_Mymr",
"myv_Cyrl",
"nld_Latn",
"nno_Latn",
"nob_Latn",
"nob_Latn_radical",
"npi_Deva",
"nqo_Nkoo",
"nso_Latn",
"nus_Latn",
"nya_Latn",
"oci_Latn",
"oci_Latn_aran1260",
"ory_Orya",
"pag_Latn",
"pan_Guru",
"pap_Latn",
"pbt_Arab",
"pes_Arab",
"plt_Latn",
"pol_Latn",
"por_Latn",
"prs_Arab",
"quy_Latn",
"ron_Latn",
"run_Latn",
"rus_Cyrl",
"sag_Latn",
"san_Deva",
"sat_Olck",
"scn_Latn",
"shn_Mymr",
"sin_Sinh",
"slk_Latn",
"slv_Latn",
"smo_Latn",
"sna_Latn",
"snd_Arab",
"som_Latn",
"sot_Latn",
"spa_Latn",
"srd_Latn",
"srp_Cyrl",
"ssw_Latn",
"sun_Latn",
"swe_Latn",
"swh_Latn",
"szl_Latn",
"tam_Taml",
"taq_Latn",
"taq_Tfng",
"tat_Cyrl",
"tel_Telu",
"tgk_Cyrl",
"tha_Thai",
"tir_Ethi",
"tpi_Latn",
"tsn_Latn",
"tso_Latn",
"tuk_Latn",
"tum_Latn",
"tur_Latn",
"twi_Latn_akua1239",
"twi_Latn_asan1239",
"tyv_Cyrl",
"uig_Arab",
"ukr_Cyrl",
"umb_Latn",
"urd_Arab",
"uzn_Latn",
"vec_Latn",
"vie_Latn",
"vmw_Latn",
"war_Latn",
"wol_Latn",
"xho_Latn",
"ydd_Hebr",
"yor_Latn",
"yue_Hant",
"zgh_Tfng",
"zsm_Latn",
"zul_Latn"
]
Loading
Loading