Skip to content

feat(xtoken): cross-tokenizer collator options and pad text to longest row - #4728

Merged
yuki-97 merged 18 commits into
mainfrom
avenkateshha/xtoken-csd-shared-collator
Oct 29, 2026
Merged

yuki-97 merged 18 commits into
mainfrom
avenkateshha/xtoken-csd-shared-collator

Conversation

@avenkateshha

@avenkateshha avenkateshha commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds three opt-in options to CrossTokenizerCollator for strict text lengths, SFT-style supervision of the full final assistant turn, and text-mode special tokens (for example a beginning-of-text marker), and pads text batches to their longest row instead of the full context length. The options are off by default; X-Token's chat batches are unchanged, and its text batches keep identical values at every real position.

Stacked on #4721 (base branch avenkateshha/xtoken-csd-same-tokenizer). The follow-up CSD PR (#4680, to be restacked on this branch) turns the options on and drops the matching code from its collator subclass.

Changes

nemo_rl/data/cross_tokenizer/cross_tokenizer_collate.py: three keyword-only CrossTokenizerCollator arguments, all False by default: the two below and add_special_tokens_in_text, which tokenizes text-mode rows with each tokenizer's own special tokens (for example Llama's <|begin_of_text|>) and raises in chat mode. The algorithm that builds the collator sets them in code. There are no new YAML keys, and the X-Token trainer still passes none of them.

  • reject_overlength_text (text mode): raises ValueError naming the side and sample idx when the student or any teacher tokenization, or that length rounded up to the side's make_sequence_length_divisible_by, is longer than that side's context length. Without it, each side is still truncated to its own context length.
    • Same-tokenizer teachers check the shared student tokenization against their own context length and divisor.
    • Lengths are counted on the same text with the same add_special_tokens setting as the batch tokenization. Rows that fit produce the same batch as with the option off.
    • Chat mode already rejects overlength rows, so the option does not change chat batches.
  • ce_covers_assistant_turn (ordinary chat mode): the student token_mask (CE) covers every token of the final assistant turn, including template markers (role header, the template's think scaffold, end-of-turn, trailing newline). A new kd_token_mask key carries the previous mask (final content + end-of-turn). Alignment tensors and teacher_{i}_token_mask are unchanged.
    • The turn starts where the rendering of all earlier messages ends (same template controls and tools) and runs to the end of the full rendering. With no earlier messages, the turn is the whole rendering.
    • The collator raises instead of guessing when the earlier-messages rendering is not a prefix of the full rendering, when a token carries non-whitespace history text into the turn, or when content/end-of-turn targets lie before the turn.
    • Earlier assistant turns stay context, as today. include_thinking_in_loss still controls KD; CE covers whatever the template renders in the final turn, including reasoning.
    • __init__ raises ValueError for mode="text" (no template turn) and for native_thinking_alignment=true (it defines its own supervised regions).

Stock Qwen3-4B, multi-turn, final turn:

collator = CrossTokenizerCollator(..., config=CrossTokenizerCollatorConfig(mode="chat"), ce_covers_assistant_turn=True)
batch = collator([datum])
tok.decode(batch["input_ids"][0][batch["token_mask"][0].bool()])
# '<|im_start|>assistant\n<think>\n\n</think>\n\nSame answer<|im_end|>\n'
tok.decode(batch["input_ids"][0][batch["kd_token_mask"][0].bool()])
# 'Same answer<|im_end|>'

Renames nemo_rl/data/xtoken/ to nemo_rl/data/cross_tokenizer/: the package now holds data code shared by both cross-tokenizer algorithms (X-Token, and CSD in #4680), so it is named after what it serves rather than one of them; the xtoken name dates from review on #3286, when only X-Token used it. Imports, pyrefly.toml and the X-Token guide are updated; no behavior change.

Behavior with the options off

Chat and native-thinking batches are bitwise identical to the base commit, with no new keys. For the first commit this was checked by collating the same inputs with the base commit and with this branch: hermetic text, ordinary chat and native-thinking batches, plus real Llama-3.2-1B-Instruct / Qwen3-4B text, chat (both directions) and Qwen3 native-thinking batches; keys, dtypes, shapes and bytes all matched. Text batches now pad to the batch's longest row, rounded up to make_sequence_length_divisible_by, instead of the full context length. Every real position keeps identical values; the only other difference is fewer padding-only alignment pairs, which hold no tokens and are already skipped by the KD loss. One downstream effect: X-Token's default P-KL loss picks its top-k teacher vocabulary over every position of the microbatch, padding included, so for text rows shorter than the context its KD values shift slightly; rows that fill the context, as in the exemplar and nightly data, are unchanged. The three hermetic batches are pinned as CRC32 snapshots in the unit tests; the text snapshot is unchanged because its longest row fills each side's context.

Testing

CPU runs in the NeMo-RL container:

  • tests/unit/data/test_cross_tokenizer_collate.py, test_cross_tokenizer_chat_eot.py, test_cross_tokenizer_native_chat.py, test_collate_fn.py, tests/unit/algorithms/x_token/, tests/unit/algorithms/test_loss_functions.py: 539 passed, 48 skipped (GPU-only). test_qwen3_history_template.py: 31 passed, run with Hugging Face Hub access because its pinned Qwen/Qwen3-30B-A3B-Thinking-2507 tokenizer is not in the offline cache.
  • New tests:
    • Default-off snapshots for text, ordinary chat and native-thinking chat.
    • reject_overlength_text: student, cross-tokenizer teacher and smaller-context same-tokenizer teacher each raise; rows that fit (exactly or with room) match the default batch; chat batches are unchanged.
    • ce_covers_assistant_turn: full final turn per row of a two-row batch on hermetic templates and on stock Qwen3-4B (same- and cross-tokenizer teachers), kd_token_mask equals the default token_mask, every other tensor equals the default, multi-turn supervises only the final turn, reasoning excluded from KD is still in CE, a role token may absorb history whitespace, no-history conversations, and the non-prefix / boundary-crossing / targets-before-turn errors. Text mode and native thinking alignment are rejected.
  • ruff (all three pre-commit hooks) and pyrefly: clean.
  • Manual check of ce_covers_assistant_turn on stock templates: Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, Gemma-3-4b-it and Phi-4 give the full final turn. Phi-4-mini-instruct raises the prefix error because its template ends every rendering with <|endoftext|>.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 15, 2026
@avenkateshha

Copy link
Copy Markdown
Contributor Author

/ok to test 22bfba9

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 15, 2026 16:19
@avenkateshha avenkateshha added the CI:L0 Run doctests and unit tests label Oct 15, 2026

@avenkateshha avenkateshha left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a clean way to let #4680 share the collator, and the default-off CRC snapshots make "X-Token unchanged" easy to trust. Checked on CPU in the NeMo-RL container: all 210 tests in the changed files pass, and the PR body's 539 passed / 48 skipped reproduces. Inline: one recipe/setup fix for the CP case (L661), one check fix plus docstring (L328-340, L121-129), one crash nit (L670), and three test additions.

PR description

  • The title says "opt-in", but padding text batches to the longest row is a default change for X-Token (L661 and L670 are its consequences), and the title leaves out add_special_tokens_in_text and the package rename. Consider e.g. feat(xtoken): cross-tokenizer collator options and pad text to longest row.
  • "Behavior with the options off": padding positions aren't fully inert in the loss. _compute_p_kl picks one top-k teacher vocabulary for the whole microbatch over every teacher row, padding included (select_teacher_topk_indices(...) with no valid_mask). So for text rows shorter than ctx, the narrower padding shifts P-KL values at real positions slightly (Qwen3-0.6B-Base, rows of 52/21 tokens: 656/8192 top-k ids differ, real-position target TV ≤ 1e-2, argmax unchanged). Packed full-context data (exemplar, nightlies, L1 smoke) is unaffected. Worth one sentence in that section. Optional follow-up, not for this PR: pass a real-position valid_mask there, as the same-tokenizer path does (loss_utils.py#L412-L413).

CI

  • Per the cicd skill ("Changes that could affect model behaviour → CI:L1"), consider CI:L1: the text-width change is a default behaviour change and the testing so far is CPU-only. The L1 X-Token smoke runs CP=1, so it won't cover the L661 case.
  • Current failures on this head are not from this diff: L0_Unit_Tests_Other fails on test_nightly_compute_stays_below_4815_hours (4819 GPU-hours; tests/test_suites isn't touched here) and L0_Unit_Tests_Megatron_4 on a Gym submodule fetch.

Generated by Claude Code

@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-same-tokenizer branch from 2b871b8 to fef7b51 Compare October 15, 2026 21:10
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-shared-collator branch from 22bfba9 to 5ef0bb3 Compare October 15, 2026 21:34
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-same-tokenizer branch from 451625c to ddd556f Compare October 16, 2026 08:47
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-shared-collator branch 3 times, most recently from 2d5df0d to 205dc30 Compare October 16, 2026 12:26
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-same-tokenizer branch from 447cc5a to 75d5b8b Compare October 19, 2026 10:52
@avenkateshha avenkateshha changed the title feat(xtoken): add opt-in overlength rejection and full-turn CE to collator feat(xtoken): cross-tokenizer collator options and pad text to longest row Oct 19, 2026
@avenkateshha avenkateshha added CI:L1 Run doctests, unit tests, and functional tests and removed CI:L0 Run doctests and unit tests labels Oct 19, 2026
@avenkateshha

Copy link
Copy Markdown
Contributor Author

Review summary items from #4728 (review):

  • Title: retitled to feat(xtoken): cross-tokenizer collator options and pad text to longest row, since padding text to the longest row is a default change.
  • Description: "Behavior with the options off" now notes that X-Token's default P-KL top-k vocabulary includes padding positions, so KD values shift slightly for text rows shorter than the context.
  • Label: CI:L0 -> CI:L1.

@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-shared-collator branch from a04b753 to e6fe651 Compare October 19, 2026 11:00
avenkateshha and others added 4 commits October 26, 2026 00:34
Move the same-tokenizer KD math out of CrossTokenizerDistillationLossFn
into nemo_rl/algorithms/x_token/loss_utils.py so other losses can reuse
it:

- same_tokenizer_topk_kl / same_tokenizer_full_vocab_kl take the KD mask,
  sample mask and global KD-token count as explicit keyword arguments.
- The top-K vocab list is unchanged: the per-vocab max over every teacher
  row (select_teacher_topk_indices), MAX-reduced across CP.

X-Token passes its token_mask as the KD mask and global_valid_toks as
the KD-token count. _direct_topk_kl, _direct_full_vocab_kl and
_same_vocab_masked_kl are removed; _compute_same_vocab_kl and the
averaged_logits fast path call the shared helpers. Losses and gradients
are bitwise identical to before for any KD and sample mask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The averaged_logits fast path passes temperature, reverse_kl, the KD and
sample masks and the KD-token count to same_tokenizer_full_vocab_kl
explicitly. Its test stubbed the helper without looking at the
arguments, so hard-coding reverse_kl=False or temperature=1.0 at the
call site went unnoticed. Record the stub's keyword arguments and check
them against non-default values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
…b KD

same_tokenizer_full_vocab_kl sliced the teacher to the student logits
width, so columns that a student lm_head pads past the tokenizer vocab
stayed in both softmaxes, while same_tokenizer_topk_kl already drops
them via vocab_size. Give the full-vocab helper the same vocab_size
argument: cut student and teacher to it and renormalize the student.
The averaged_logits fast path passes len(student_tokenizer).

Only kd_loss_mode=averaged_logits with same-tokenizer teachers and a
padded student (e.g. Qwen) changes; unpadded students are bitwise
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
vocab_topk was documented as P-KL only and inert when gold_loss=True,
but every same-tokenizer teacher uses it for its top-K KL whatever
gold_loss is; only the all-same-tokenizer averaged_logits path uses the
full vocab. Update the CrossTokenizerDistillationLossConfig field doc,
the vocab_topk comments in both X-Token exemplar configs and the guide,
which also described the multi-teacher example's same-tokenizer teacher
as a full-vocab KL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-same-tokenizer branch from 75d5b8b to 6c27c72 Compare October 26, 2026 07:48
avenkateshha and others added 6 commits October 26, 2026 00:49
…lator

Add two keyword-only CrossTokenizerCollator options. Both are off by
default, so existing batches stay bitwise identical:

- reject_overlength_text: in text mode, raise ValueError naming the side
  and sample idx when a student or teacher tokenization exceeds that
  side's context length, instead of truncating. Same-tokenizer teachers
  check the shared student tokenization against their own context length.
- ce_covers_assistant_turn: in ordinary chat mode, the CE token_mask
  covers the whole final assistant turn, including template markers. The
  previous content + end-of-turn mask is emitted as kd_token_mask, and
  alignment and teacher masks are unchanged. Text mode and native
  thinking alignment reject the option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Add a keyword-only add_special_tokens_in_text option to
CrossTokenizerCollator, off by default so existing batches are unchanged.
When on, text mode tokenizes the student and every cross-tokenizer
teacher with each tokenizer's own special tokens (for example a
beginning-of-text marker at the start of each row), and
reject_overlength_text counts them.

Added tokens keep (0, 0) offsets, so the aligner pairs them by role; a
marker with no teacher counterpart forms its own chunk with an empty
teacher side, which the KD loss skips. Chat mode rejects the option
because chat templates insert their own markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Text mode now right-pads each batch to its longest tokenization, rounded
up to make_sequence_length_divisible_by, instead of to the full context
length, for the student and every cross-tokenizer teacher. Chat mode
already used this width rule; both now share one helper. Truncation to
the context length is unchanged, so a truncated row can still round the
width past a context that is not a multiple of the divisor.

Values at every real position are unchanged, and chat and native
thinking batches are bitwise identical. The aligner also pairs trailing
padding with padding; those pairs hold no tokens and the KD loss already
skips them, so narrower rows simply emit fewer of them in
alignment_*_pair_valid and alignment_*_pair_is_correct.

With reject_overlength_text, a row is also rejected when its length
rounded up to the side's divisor exceeds that side's context, so a
padded batch is never wider than its context. Same-tokenizer teachers
check the shared tokenization against their own context length and
divisor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The pinned Qwen tokenizer revisions in test_qwen3_history_template.py
moved down one line when the test gained an import. Update their
allowlisted line numbers (39 -> 40, 91 -> 92) so secret scanning passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The package holds the cross-tokenizer collator and chat renderer, which
are now shared data code for both cross-tokenizer distillation
algorithms, so name it after what it serves rather than one of them.

Move nemo_rl/data/xtoken/ to nemo_rl/data/cross_tokenizer/ and update
every reference: imports in the trainer, the package and the tests, the
pyrefly project-includes, and the collator path in the X-Token guide.
Import blocks are re-sorted by ruff's isort rules; no file changes
length. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The option description said that with the option off each side is
truncated to its own context length, and that chat mode always rejects
overlength rows after defining overlength to include the length rounded
up to the sequence-length divisor. A same-tokenizer teacher instead
shares the student's truncation, and chat mode checks only the raw
token count. Say so, and limit the padded-width guarantee to the
student and cross-tokenizer teachers, whose batches the check bounds.
No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
avenkateshha and others added 5 commits October 26, 2026 00:49
When every text in a batch tokenizes to no tokens, the tokenizer
returns offset_mapping as a (B, 0) tensor without the start/end axis.
Text batches now pad to the longest row, so such a batch is padded to
one sequence-length multiple, and concatenating (B, pad, 2) offsets onto
the (B, 0) tensor raised "Tensors must have same number of dimensions:
got 2 and 3". Reshape the offsets to (B, T, 2) from the token ids'
shape, so the batch collates to all-padding rows as it did when text
batches padded to the full context length. Non-empty batches are
unchanged.

Add a regression test for an all-empty text batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
A same-tokenizer teacher's forward reuses the shared student batch,
which is padded to the student's sequence-length divisor, but
reject_overlength_text rounded its rows up only to the teacher's own
divisor. With a student divisor of 8 and a teacher context of 12 with
divisor 2, a 9-token row passed the check while the teacher received a
16-wide batch.

Check each row rounded up to both the student's divisor and the
teacher's own, which applies when the teacher's rows are packed. The
error messages are unchanged, and the option description now states
that no padded batch is wider than its context.

Add a regression test where the student and teacher divisors differ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The add_special_tokens_in_text tests only used a teacher tokenizer that
adds no special tokens, so dropping add_special_tokens from the teacher
tokenization or from the teacher's reject_overlength_text count still
passed every test. The second silently truncates a teacher row that
the option should reject.

Add a test with a word-level teacher that starts each row with <bos>:
the marker appears in the teacher row, forms its own chunk with no
student token, and is counted by reject_overlength_text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
ce_covers_assistant_turn finds the final assistant turn by rendering
the earlier messages with the same preserve_thinking and tools controls
as the full rendering. No test used either control, so rendering the
earlier messages without one still passed every test.

Add tests with earlier reasoning kept by the template and with tools in
the rendering; both check that CE covers exactly the final turn and that
kd_token_mask keeps the default token_mask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Text batches now pad to their longest row, so batch widths follow the
data. Both TP4CP2 X-Token recipes run CP=2 with
make_sequence_length_divisible_by 1. That was harmless while text
batches padded to the 1024-token context, but now a batch whose longest
row has an odd length reaches the CP-divisibility check in the teacher
forward or the loss and stops training mid-run.

Set make_sequence_length_divisible_by to 2 for the policy and every
teacher in both recipes; their packed rows fill the context, so their
batch widths are unchanged. Check at setup that each divisor is a
multiple of its model's context_parallel_size, using the student's
divisor for a same-vocab teacher, which forwards the student batch, so
a mismatched config fails before any cluster is built.

Add setup tests for the student, cross-tokenizer teacher and
same-vocab teacher cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
@avenkateshha
avenkateshha force-pushed the avenkateshha/xtoken-csd-shared-collator branch from 16ea4c3 to d7e38de Compare October 26, 2026 07:56
Base automatically changed from avenkateshha/xtoken-csd-same-tokenizer to main October 27, 2026 16:48

@RayenTian RayenTian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — a high-quality PR. The default-off CRC snapshots make "X-Token unchanged" easy to trust, and the new overlength-rejection and full-turn CE logic held up under end-to-end checks on real stock Qwen3 templates (the final turn is covered exactly, kd_token_mask stays the content+EOT subset, and inline thinking correctly raises toward native_thinking_alignment rather than silently mis-supervising). Only two low-priority notes inline: one pre-existing consistency nit (from #3286, not introduced here) and one small test gap.

Generated by Claude Code

Check chat rows after rounding to the student's sequence-length divisor before accepting them for same-tokenizer teachers. Preserve the existing raw-length error and reject oversized padded inputs without truncating conversations.

Add regression coverage for 9 tokens padded to 16 against a teacher context of 12, mixed row lengths, and padded widths that fit exactly or below a non-multiple context. Validation: 131 focused collator tests passed; Ruff lint, imports, and formatting passed.

Signed-off-by: Adithyakrishna Hanasoge <avenkateshha@nvidia.com>
Add a public-collator regression test for a chat template whose full
conversation rendering is identical to its history-only rendering.

Verify that ordinary content supervision accepts the nonempty answer,
while full-turn CE raises the exact no-final-turn-tokens error. This
exercises the missing guard without mocking the renderer or constructing
an invalid rendered document.

Signed-off-by: Adithyakrishna Hanasoge <avenkateshha@nvidia.com>

@RayenTian RayenTian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Approved.

@yuki-97 yuki-97 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Oct 29, 2026
@yuki-97
yuki-97 enabled auto-merge (squash) October 29, 2026 09:59
@yuki-97

yuki-97 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

/ok to test 265cf15

@yuki-97
yuki-97 merged commit 0b305c9 into main Oct 29, 2026
161 of 163 checks passed
@yuki-97
yuki-97 deleted the avenkateshha/xtoken-csd-shared-collator branch October 29, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants