Repository navigation
feat(xtoken): cross-tokenizer collator options and pad text to longest row - #4728
Conversation
|
/ok to test 22bfba9 |
avenkateshha
left a comment
There was a problem hiding this comment.
Thanks, this is a clean way to let #4680 share the collator, and the default-off CRC snapshots make "X-Token unchanged" easy to trust. Checked on CPU in the NeMo-RL container: all 210 tests in the changed files pass, and the PR body's 539 passed / 48 skipped reproduces. Inline: one recipe/setup fix for the CP case (L661), one check fix plus docstring (L328-340, L121-129), one crash nit (L670), and three test additions.
PR description
- The title says "opt-in", but padding text batches to the longest row is a default change for X-Token (L661 and L670 are its consequences), and the title leaves out
add_special_tokens_in_textand the package rename. Consider e.g.feat(xtoken): cross-tokenizer collator options and pad text to longest row. - "Behavior with the options off": padding positions aren't fully inert in the loss.
_compute_p_klpicks one top-k teacher vocabulary for the whole microbatch over every teacher row, padding included (select_teacher_topk_indices(...)with novalid_mask). So for text rows shorter than ctx, the narrower padding shifts P-KL values at real positions slightly (Qwen3-0.6B-Base, rows of 52/21 tokens: 656/8192 top-k ids differ, real-position target TV ≤ 1e-2, argmax unchanged). Packed full-context data (exemplar, nightlies, L1 smoke) is unaffected. Worth one sentence in that section. Optional follow-up, not for this PR: pass a real-positionvalid_maskthere, as the same-tokenizer path does (loss_utils.py#L412-L413).
CI
- Per the cicd skill ("Changes that could affect model behaviour →
CI:L1"), considerCI:L1: the text-width change is a default behaviour change and the testing so far is CPU-only. The L1 X-Token smoke runs CP=1, so it won't cover the L661 case. - Current failures on this head are not from this diff:
L0_Unit_Tests_Otherfails ontest_nightly_compute_stays_below_4815_hours(4819 GPU-hours;tests/test_suitesisn't touched here) andL0_Unit_Tests_Megatron_4on a Gym submodule fetch.
Generated by Claude Code
2b871b8 to
fef7b51
Compare
22bfba9 to
5ef0bb3
Compare
451625c to
ddd556f
Compare
2d5df0d to
205dc30
Compare
447cc5a to
75d5b8b
Compare
|
Review summary items from #4728 (review):
|
a04b753 to
e6fe651
Compare
Move the same-tokenizer KD math out of CrossTokenizerDistillationLossFn into nemo_rl/algorithms/x_token/loss_utils.py so other losses can reuse it: - same_tokenizer_topk_kl / same_tokenizer_full_vocab_kl take the KD mask, sample mask and global KD-token count as explicit keyword arguments. - The top-K vocab list is unchanged: the per-vocab max over every teacher row (select_teacher_topk_indices), MAX-reduced across CP. X-Token passes its token_mask as the KD mask and global_valid_toks as the KD-token count. _direct_topk_kl, _direct_full_vocab_kl and _same_vocab_masked_kl are removed; _compute_same_vocab_kl and the averaged_logits fast path call the shared helpers. Losses and gradients are bitwise identical to before for any KD and sample mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The averaged_logits fast path passes temperature, reverse_kl, the KD and sample masks and the KD-token count to same_tokenizer_full_vocab_kl explicitly. Its test stubbed the helper without looking at the arguments, so hard-coding reverse_kl=False or temperature=1.0 at the call site went unnoticed. Record the stub's keyword arguments and check them against non-default values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
…b KD same_tokenizer_full_vocab_kl sliced the teacher to the student logits width, so columns that a student lm_head pads past the tokenizer vocab stayed in both softmaxes, while same_tokenizer_topk_kl already drops them via vocab_size. Give the full-vocab helper the same vocab_size argument: cut student and teacher to it and renormalize the student. The averaged_logits fast path passes len(student_tokenizer). Only kd_loss_mode=averaged_logits with same-tokenizer teachers and a padded student (e.g. Qwen) changes; unpadded students are bitwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
vocab_topk was documented as P-KL only and inert when gold_loss=True, but every same-tokenizer teacher uses it for its top-K KL whatever gold_loss is; only the all-same-tokenizer averaged_logits path uses the full vocab. Update the CrossTokenizerDistillationLossConfig field doc, the vocab_topk comments in both X-Token exemplar configs and the guide, which also described the multi-teacher example's same-tokenizer teacher as a full-vocab KL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
75d5b8b to
6c27c72
Compare
…lator Add two keyword-only CrossTokenizerCollator options. Both are off by default, so existing batches stay bitwise identical: - reject_overlength_text: in text mode, raise ValueError naming the side and sample idx when a student or teacher tokenization exceeds that side's context length, instead of truncating. Same-tokenizer teachers check the shared student tokenization against their own context length. - ce_covers_assistant_turn: in ordinary chat mode, the CE token_mask covers the whole final assistant turn, including template markers. The previous content + end-of-turn mask is emitted as kd_token_mask, and alignment and teacher masks are unchanged. Text mode and native thinking alignment reject the option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Add a keyword-only add_special_tokens_in_text option to CrossTokenizerCollator, off by default so existing batches are unchanged. When on, text mode tokenizes the student and every cross-tokenizer teacher with each tokenizer's own special tokens (for example a beginning-of-text marker at the start of each row), and reject_overlength_text counts them. Added tokens keep (0, 0) offsets, so the aligner pairs them by role; a marker with no teacher counterpart forms its own chunk with an empty teacher side, which the KD loss skips. Chat mode rejects the option because chat templates insert their own markers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Text mode now right-pads each batch to its longest tokenization, rounded up to make_sequence_length_divisible_by, instead of to the full context length, for the student and every cross-tokenizer teacher. Chat mode already used this width rule; both now share one helper. Truncation to the context length is unchanged, so a truncated row can still round the width past a context that is not a multiple of the divisor. Values at every real position are unchanged, and chat and native thinking batches are bitwise identical. The aligner also pairs trailing padding with padding; those pairs hold no tokens and the KD loss already skips them, so narrower rows simply emit fewer of them in alignment_*_pair_valid and alignment_*_pair_is_correct. With reject_overlength_text, a row is also rejected when its length rounded up to the side's divisor exceeds that side's context, so a padded batch is never wider than its context. Same-tokenizer teachers check the shared tokenization against their own context length and divisor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The pinned Qwen tokenizer revisions in test_qwen3_history_template.py moved down one line when the test gained an import. Update their allowlisted line numbers (39 -> 40, 91 -> 92) so secret scanning passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The package holds the cross-tokenizer collator and chat renderer, which are now shared data code for both cross-tokenizer distillation algorithms, so name it after what it serves rather than one of them. Move nemo_rl/data/xtoken/ to nemo_rl/data/cross_tokenizer/ and update every reference: imports in the trainer, the package and the tests, the pyrefly project-includes, and the collator path in the X-Token guide. Import blocks are re-sorted by ruff's isort rules; no file changes length. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The option description said that with the option off each side is truncated to its own context length, and that chat mode always rejects overlength rows after defining overlength to include the length rounded up to the sequence-length divisor. A same-tokenizer teacher instead shares the student's truncation, and chat mode checks only the raw token count. Say so, and limit the padded-width guarantee to the student and cross-tokenizer teachers, whose batches the check bounds. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
When every text in a batch tokenizes to no tokens, the tokenizer returns offset_mapping as a (B, 0) tensor without the start/end axis. Text batches now pad to the longest row, so such a batch is padded to one sequence-length multiple, and concatenating (B, pad, 2) offsets onto the (B, 0) tensor raised "Tensors must have same number of dimensions: got 2 and 3". Reshape the offsets to (B, T, 2) from the token ids' shape, so the batch collates to all-padding rows as it did when text batches padded to the full context length. Non-empty batches are unchanged. Add a regression test for an all-empty text batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
A same-tokenizer teacher's forward reuses the shared student batch, which is padded to the student's sequence-length divisor, but reject_overlength_text rounded its rows up only to the teacher's own divisor. With a student divisor of 8 and a teacher context of 12 with divisor 2, a 9-token row passed the check while the teacher received a 16-wide batch. Check each row rounded up to both the student's divisor and the teacher's own, which applies when the teacher's rows are packed. The error messages are unchanged, and the option description now states that no padded batch is wider than its context. Add a regression test where the student and teacher divisors differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
The add_special_tokens_in_text tests only used a teacher tokenizer that adds no special tokens, so dropping add_special_tokens from the teacher tokenization or from the teacher's reject_overlength_text count still passed every test. The second silently truncates a teacher row that the option should reject. Add a test with a word-level teacher that starts each row with <bos>: the marker appears in the teacher row, forms its own chunk with no student token, and is counted by reject_overlength_text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
ce_covers_assistant_turn finds the final assistant turn by rendering the earlier messages with the same preserve_thinking and tools controls as the full rendering. No test used either control, so rendering the earlier messages without one still passed every test. Add tests with earlier reasoning kept by the template and with tools in the rendering; both check that CE covers exactly the final turn and that kd_token_mask keeps the default token_mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
Text batches now pad to their longest row, so batch widths follow the data. Both TP4CP2 X-Token recipes run CP=2 with make_sequence_length_divisible_by 1. That was harmless while text batches padded to the 1024-token context, but now a batch whose longest row has an odd length reaches the CP-divisibility check in the teacher forward or the loss and stops training mid-run. Set make_sequence_length_divisible_by to 2 for the policy and every teacher in both recipes; their packed rows fill the context, so their batch widths are unchanged. Check at setup that each divisor is a multiple of its model's context_parallel_size, using the student's divisor for a same-vocab teacher, which forwards the student batch, so a mismatched config fails before any cluster is built. Add setup tests for the student, cross-tokenizer teacher and same-vocab teacher cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>
16ea4c3 to
d7e38de
Compare
RayenTian
left a comment
There was a problem hiding this comment.
Thanks for this — a high-quality PR. The default-off CRC snapshots make "X-Token unchanged" easy to trust, and the new overlength-rejection and full-turn CE logic held up under end-to-end checks on real stock Qwen3 templates (the final turn is covered exactly, kd_token_mask stays the content+EOT subset, and inline thinking correctly raises toward native_thinking_alignment rather than silently mis-supervising). Only two low-priority notes inline: one pre-existing consistency nit (from #3286, not introduced here) and one small test gap.
Generated by Claude Code
Check chat rows after rounding to the student's sequence-length divisor before accepting them for same-tokenizer teachers. Preserve the existing raw-length error and reject oversized padded inputs without truncating conversations. Add regression coverage for 9 tokens padded to 16 against a teacher context of 12, mixed row lengths, and padded widths that fit exactly or below a non-multiple context. Validation: 131 focused collator tests passed; Ruff lint, imports, and formatting passed. Signed-off-by: Adithyakrishna Hanasoge <avenkateshha@nvidia.com>
Add a public-collator regression test for a chat template whose full conversation rendering is identical to its history-only rendering. Verify that ordinary content supervision accepts the nonempty answer, while full-turn CE raises the exact no-final-turn-tokens error. This exercises the missing guard without mocking the renderer or constructing an invalid rendered document. Signed-off-by: Adithyakrishna Hanasoge <avenkateshha@nvidia.com>
|
/ok to test 265cf15 |
What does this PR do ?
Adds three opt-in options to
CrossTokenizerCollatorfor strict text lengths, SFT-style supervision of the full final assistant turn, and text-mode special tokens (for example a beginning-of-text marker), and pads text batches to their longest row instead of the full context length. The options are off by default; X-Token's chat batches are unchanged, and its text batches keep identical values at every real position.Stacked on #4721 (base branch
avenkateshha/xtoken-csd-same-tokenizer). The follow-up CSD PR (#4680, to be restacked on this branch) turns the options on and drops the matching code from its collator subclass.Changes
nemo_rl/data/cross_tokenizer/cross_tokenizer_collate.py: three keyword-onlyCrossTokenizerCollatorarguments, allFalseby default: the two below andadd_special_tokens_in_text, which tokenizes text-mode rows with each tokenizer's own special tokens (for example Llama's<|begin_of_text|>) and raises in chat mode. The algorithm that builds the collator sets them in code. There are no new YAML keys, and the X-Token trainer still passes none of them.reject_overlength_text(text mode): raisesValueErrornaming the side and sampleidxwhen the student or any teacher tokenization, or that length rounded up to the side'smake_sequence_length_divisible_by, is longer than that side's context length. Without it, each side is still truncated to its own context length.add_special_tokenssetting as the batch tokenization. Rows that fit produce the same batch as with the option off.ce_covers_assistant_turn(ordinary chat mode): the studenttoken_mask(CE) covers every token of the final assistant turn, including template markers (role header, the template's think scaffold, end-of-turn, trailing newline). A newkd_token_maskkey carries the previous mask (final content + end-of-turn). Alignment tensors andteacher_{i}_token_maskare unchanged.include_thinking_in_lossstill controls KD; CE covers whatever the template renders in the final turn, including reasoning.__init__raisesValueErrorformode="text"(no template turn) and fornative_thinking_alignment=true(it defines its own supervised regions).Stock Qwen3-4B, multi-turn, final turn:
Renames
nemo_rl/data/xtoken/tonemo_rl/data/cross_tokenizer/: the package now holds data code shared by both cross-tokenizer algorithms (X-Token, and CSD in #4680), so it is named after what it serves rather than one of them; thextokenname dates from review on #3286, when only X-Token used it. Imports,pyrefly.tomland the X-Token guide are updated; no behavior change.Behavior with the options off
Chat and native-thinking batches are bitwise identical to the base commit, with no new keys. For the first commit this was checked by collating the same inputs with the base commit and with this branch: hermetic text, ordinary chat and native-thinking batches, plus real Llama-3.2-1B-Instruct / Qwen3-4B text, chat (both directions) and Qwen3 native-thinking batches; keys, dtypes, shapes and bytes all matched. Text batches now pad to the batch's longest row, rounded up to
make_sequence_length_divisible_by, instead of the full context length. Every real position keeps identical values; the only other difference is fewer padding-only alignment pairs, which hold no tokens and are already skipped by the KD loss. One downstream effect: X-Token's default P-KL loss picks its top-k teacher vocabulary over every position of the microbatch, padding included, so for text rows shorter than the context its KD values shift slightly; rows that fill the context, as in the exemplar and nightly data, are unchanged. The three hermetic batches are pinned as CRC32 snapshots in the unit tests; the text snapshot is unchanged because its longest row fills each side's context.Testing
CPU runs in the NeMo-RL container:
tests/unit/data/test_cross_tokenizer_collate.py,test_cross_tokenizer_chat_eot.py,test_cross_tokenizer_native_chat.py,test_collate_fn.py,tests/unit/algorithms/x_token/,tests/unit/algorithms/test_loss_functions.py: 539 passed, 48 skipped (GPU-only).test_qwen3_history_template.py: 31 passed, run with Hugging Face Hub access because its pinnedQwen/Qwen3-30B-A3B-Thinking-2507tokenizer is not in the offline cache.reject_overlength_text: student, cross-tokenizer teacher and smaller-context same-tokenizer teacher each raise; rows that fit (exactly or with room) match the default batch; chat batches are unchanged.ce_covers_assistant_turn: full final turn per row of a two-row batch on hermetic templates and on stock Qwen3-4B (same- and cross-tokenizer teachers),kd_token_maskequals the defaulttoken_mask, every other tensor equals the default, multi-turn supervises only the final turn, reasoning excluded from KD is still in CE, a role token may absorb history whitespace, no-history conversations, and the non-prefix / boundary-crossing / targets-before-turn errors. Text mode and native thinking alignment are rejected.ruff(all three pre-commit hooks) andpyrefly: clean.ce_covers_assistant_turnon stock templates: Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, Gemma-3-4b-it and Phi-4 give the full final turn. Phi-4-mini-instruct raises the prefix error because its template ends every rendering with<|endoftext|>.🤖 Generated with Claude Code