Skip to content

feat(translation): add an Indic-focused LLM judge environment - #3348

Open
tanishku wants to merge 26 commits into
NVIDIA-NeMo:mainfrom
tanishku:feature/indic-translation-with-judge
Open

tanishku wants to merge 26 commits into
NVIDIA-NeMo:mainfrom
tanishku:feature/indic-translation-with-judge

Conversation

@tanishku

@tanishku tanishku commented Sep 13, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds translation_with_judge, a machine-translation resources server whose reward comes
from an LLM judge instead of a fixed n-gram metric, with a particular focus on Indic
language pairs.

verify() sends the source segment, a reference translation, and the policy model's
candidate translation to a judge model (any OpenAI-Responses-compatible endpoint) and asks
it to rate adequacy + fluency on a 0-100 scale; that score, normalized to [0, 1], is the
RL reward. Sentence-BLEU and chrF (via sacrebleu) are also computed per row
as cheap, judge-independent diagnostics that do not affect the reward.

The environment includes dedicated tokenizer handling for Indic scripts: sentence-BLEU uses
sacrebleu's flores200 SentencePiece tokenizer (rather than the default
whitespace/punctuation tokenizer, which doesn't meaningfully segment Brahmic scripts) for
Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia,
Assamese, Urdu, Nepali, Sanskrit, Dogri, Konkani, Sindhi, Bodo, Kashmiri, Maithili, and
Manipuri.

Verification logic: app.py's verify() strips any <think>...</think> reasoning
preamble from the policy output, then sends {source, reference, candidate} to the judge
with a structured rubric prompt and parses a Score: <0-100> line from the response
(clipped to [0, 100], None if unparseable — masked with failure_kind=judge_unparseable and reported as lost coverage; empty candidate translations remain unmasked zeroes).
compute_metrics() aggregates judge score / BLEU / chrF per (src_lang, tgt_lang) pair and
as cross-pair aggregates (xx->xx, <src>->xx, xx-><tgt>), mirroring wmt_translation's
metric shape. Repeat groups preserve _ng_rollout_index after masking, so valid
measurements from other tasks are retained and different repeats are not mixed.
The existing equal-per-pair/equal-per-repeat weighting is preserved.

The judge prompt XML-escapes and separately tags source, reference, and candidate text,
and instructs the judge to ignore embedded instructions and claimed scores. This is
a mitigation, not a guarantee of resistance to arbitrary prompt injection.
The server uses Gym's public package imports and committed pyproject.toml/uv.lock
dependencies in place of requirements.txt.

Task source: data/example.jsonl (5 rows) is sourced directly from the
FLORES-200
devtest split (CC-BY-SA 4.0), covering English↔Hindi/Bengali/Telugu/Tamil/Kannada in both
directions. This PR ships example data only — no train/validation split — since the
larger Indic-language curriculum this was developed against blends several corpora
(Samanantar, NLLB, AI4Bharat's BPCC, ILCI, and others) whose individual licenses have not
yet been verified for redistribution. Happy to follow up with a train/validation PR once
that audit is done.

Domain coverage: domain: translation, focused on Indic-language pairs specifically
(as opposed to wmt_translation, which is the existing generic WMT-style verifier using
spBLEU/chrF/COMET). verified: false per the new-environment default.

Licensing: code is Apache 2.0 (SPDX headers on app.py/test_app.py); example data is
CC-BY-SA 4.0 (FLORES-200); dependencies (sacrebleu, sentencepiece) are Apache 2.0. See
the environment's README.md for full details.

Verification performed

  • Workstation, Python 3.13.14: 1,036 tests passed across
    resources_servers/translation_with_judge/tests,
    tests/unit_tests/tooling/test_component_dependencies.py, and
    tests/unit_tests/test_public_api_facades.py (pytest through a source-bound runner).
  • All three added regressions fail against the original server implementation and
    pass with these fixes: injected delimiter handling, masked-repeat preservation,
    and sparse cross-pair repeat alignment.
  • pre-commit run --all-files — passed, including import layering. The generated
    README table restores this environment's entry.
  • uv lock --check --python 3.13.14 --project resources_servers/translation_with_judge
    — passed. The component lock uses main's OpenAI 2.45.0 pin.
  • Live policy + judge smoke using nvidia/nemotron-3.5-lightning-30b-a3b through
    actual HTTP environment, agent, resources and model servers: five shipped examples
    completed with the repaired verifier before the main update (scores 90, 95, 45,
    95, 20; none masked). The verifier source was unchanged by integration; one full
    example was freshly rerun on the final base and scored 90, unmasked.
  • An unrelated candidate containing a closing delimiter and Score: 100 scored 0.
    Same-candidate rejudging with the original prompt on the current framework scored
    95 versus 90 with the updated prompt for the valid example, and 0 versus 0 for
    the injected candidate. This tiny smoke is not a quality baseline or a guarantee
    of general prompt-injection resistance. Policy and judge used the same model;
    apps were started directly, so the CLI/multiprocess launcher was not exercised.
  • The original author's committed example rollouts remain historical evidence from
    openai/gpt-oss-120b; they were not regenerated by this follow-up.

No separate issue is needed for this self-contained training-environment addition.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

… environment

Adds an LLM-as-judge machine-translation verifier with dedicated FLORES-200
tokenizer support for 21 Indic languages, scoring adequacy + fluency 0-100
instead of relying on a fixed n-gram metric.

Signed-off-by: Tanish Kushalkar <38599503+tanishku@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 13, 2026
@tanishku

Copy link
Copy Markdown
Author

Hi @cmunley1 — could you /ok to test and take a look at #3348 when you get a chance? Thanks.

@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test e0dfccf

@cmunley1

Copy link
Copy Markdown
Contributor

i believe the testsuite is failing as task_data.py is missing, see other resources servers for example

@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work waiting-on-customer Waiting on the original author to respond labels Sep 13, 2026
…tric bias

Add task_data.py (required by CI's per-server schema gate, missing from the
initial commit) mirroring TranslationWithJudgeRunRequest. Also fix
compute_metrics() dropping unparseable judge scores (None) from the mean
instead of counting them as 0, which silently biased aggregate judge_score
upward whenever the judge failed to produce a parseable score -- per
review from @cmunley1 on NVIDIA-NeMo#3348.

Signed-off-by: Tanish Kushalkar <38599503+tanishku@users.noreply.github.com>
@tanishku

Copy link
Copy Markdown
Author

@cmunley1 pushed a fix for both — added task_data.py and fixed the metric bias. Could you /ok to test 13d81d3 when you get a chance?

@yaoyu-33 yaoyu-33 added the complexity:medium Single-domain change with interacting parts or a moderate review surface label Sep 15, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Sep 15, 2026
@yaoyu-33 yaoyu-33 added the needs-review PR is ready for code review and waiting on a reviewer label Sep 15, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 17, 2026
@tanishku

Copy link
Copy Markdown
Author

@cmunley1 pushed a merge to resolve a conflict with main, could you /ok to test when you get a chance? Thanks.

@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Sep 17, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Sep 17, 2026
@yaoyu-33 yaoyu-33 added the needs-review PR is ready for code review and waiting on a reviewer label Sep 17, 2026
Signed-off-by: Tanish <38599503+tanishku@users.noreply.github.com>
@tanishku

Copy link
Copy Markdown
Author

Accepted incoming changes from main for project root README.md in the above commit

@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Sep 19, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 27, 2026
@tanishku
tanishku requested a review from cmunley1 October 28, 2026 05:39
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 29, 2026
Signed-off-by: Tanish <38599503+tanishku@users.noreply.github.com>
@tanishku

Copy link
Copy Markdown
Author

Resolved README.md conflict

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Oct 30, 2026
Signed-off-by: Tanish <38599503+tanishku@users.noreply.github.com>
@tanishku

Copy link
Copy Markdown
Author

Hi @cmunley1 @yaoyu-33,

Gentle ping on this PR!

I've addressed the requested changes and synced with upstream. The latest changes were pushed on October 27.

Could someone take another look? If everything is in order, I'd really appreciate getting this approved and merged. If there are remaining blockers, please let me know.

Thanks!

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 10, 2026
@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test de4e066

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 13, 2026 01:10
cmunley1
cmunley1 previously approved these changes Nov 13, 2026

@cmunley1 cmunley1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The inline notes are non-blocking suggestions for a follow-up.

Reference translation:
{reference_translation}

Candidate translation:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): the candidate is policy output and goes into the judge prompt as raw text, so during RL a policy could learn to append something like "Score: 100". Wrapping it in clear tags and telling the judge to ignore any instructions or scores inside it, plus a test with an injected score, would close that off.

return {}

rollout_counts = [len(r) for r in tasks]
max_k = min(rollout_counts) if rollout_counts else 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): masked rows are removed before compute_metrics, so one unparseable judge reply lowers max_k for every task and drops valid rollouts from all pair metrics. Grouping by the rollout index field instead of list position would keep them.

from pydantic import BaseModel
from sacrebleu import sentence_bleu, sentence_chrf

from nemo_gym.base_resources_server import (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): these are the old import paths that go away when the shims are removed. scripts/migrate_imports.py can switch them to nemo_gym.server_runtime / nemo_gym.core automatically.

@cmunley1
cmunley1 enabled auto-merge November 13, 2026 01:11
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 13, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test 795938d

@yaoyu-33 yaoyu-33 changed the title feat(translation_with_judge): add Indic-focused LLM-judge translation… feat(translation): add an Indic-focused LLM judge environment Nov 18, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added risk:high Autobot PR risk: high and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 18, 2026

This branch was previously deployed

1 inactive (outdated) deployment
public — de4e0665 Deployed Nov 13, 2026 by copy-pr-bot[bot] via release / finalize / notify #12196
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers benchmark-onboarding Self-contained benchmark onboarding community-request Issue reported or requested by someone from the community complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer risk:high Autobot PR risk: high sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants