Repository navigation
👑 [experimental] GOLD Trainer #4349
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 12 commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
c3e4744
intial
kashif 1966d1f
initial tests
kashif 59bef64
add test_uldloss_hybrid_config_beta_zero
kashif 954aa88
update sleep mode
kashif f8b7212
Merge branch 'main' into gold
kashif 007f80a
chatml test and docs
kashif 406d3ed
fix formatting and mark heavier tests as slow
kashif d2f4e6b
edge case when student/teacher have no overlap in hybrid loss
kashif 0ed3b34
typos
kashif 05a044c
style in doc
qgallouedec b32acf6
citation as class attr
qgallouedec 9fde612
use student_model_revision
kashif 5bcd11f
Dropped the deprecated formatting_func
kashif be88e8e
use _set_signature_columns_if_needed
kashif bae7df5
Removed the dataset_kwargs["skip_prepare_dataset"] hack
kashif e141e52
use create_model_from_path helper
kashif 7871dfd
better names
kashif dfb1310
moved imports to top level
kashif 624ec4c
remove student_ prefix from vllm configs
kashif 7061253
Update trl/experimental/gold/gold_config.py
kashif e36a11c
Update trl/experimental/gold/gold_trainer.py
kashif 9acf7eb
use max_completion_length
kashif 4e69b12
use relative imports
kashif 62f18e8
avoid hasattr
kashif a2acc33
use ensure_master_addr_port
kashif aab8b02
Update trl/experimental/gold/gold_trainer.py
kashif e2dfff5
Update trl/experimental/gold/gold_trainer.py
kashif 0a0c79a
minimal example
kashif cb587c2
remove padding when calling generate_batch
kashif 3382b1e
use lightweight _GoldDataCollator
kashif 260e920
initial gold.py script
kashif eb9ad55
fix args
kashif e504832
Revert "use lightweight _GoldDataCollator"
kashif 841939f
update the _set_signature_columns_if_needed
kashif 0e7b4b8
fix the if statement
kashif e325322
test generate_on_policy_outputs
kashif 78ecfda
add test_generate_on_policy_outputs_masks_prompt_smollm
kashif 232f8b6
use TinyLlama
kashif 76fb22b
update _paper link
kashif 33b009a
add default lr
kashif 060f9ba
nit
qgallouedec b8217fe
Merge branch 'main' into gold
qgallouedec 4f27fce
fix autodoc
qgallouedec File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # General Online Logit Distillation (GOLD) Trainer | ||
|
|
||
| [](https://huggingface.co/models?other=sft,gold) | ||
|
|
||
| ## Overview | ||
|
|
||
| General Online Logit Distillation (GOLD) is an extension of Universal Logit Distillation (ULD) that supports | ||
| student/teacher pairs with different tokenizers. It aligns the textual spans produced by both tokenizers and merges the | ||
| associated logits so no completion tokens are dropped. This enables cross-tokenizer knowledge distillation, including | ||
| mixed model families (for example, LLaMA students with Qwen teachers). | ||
|
|
||
| Key capabilities: | ||
|
|
||
| 1. **Cross-tokenizer alignment** – GOLD incrementally decodes the student and teacher tokens, groups passages with the same visible text, and merges probabilities inside each group. This guarantees loss terms are computed over the full completion even when token boundaries differ. | ||
| 2. **Hybrid ULD loss** – when `uld_use_hybrid_loss` is enabled, GOLD compares exact vocabulary matches directly and falls back to the original sorted-probability ULD loss for unmatched tokens. This improves stability for students whose vocabularies only partially overlap with the teacher. | ||
| 3. **Seamless integration with GKD** – GOLD inherits the on-policy vs. off-policy scheduling from the [`GKDTrainer`](./gkd_trainer.md), so you can combine sequence-level KD, generalized JSD, and cross-tokenizer distillation in a single training run. | ||
|
|
||
| > [!NOTE] | ||
| > GOLD is currently part of the `trl.experimental` namespace. APIs may change without notice while the feature is iterated on. | ||
|
|
||
| ## Usage tips | ||
|
|
||
| The [`GOLDTrainer`] subclasses [`SFTTrainer`] and accepts the same datasets as other TRL trainers (lists of ChatML style | ||
| messages). Important configuration flags on [`GOLDConfig`] include: | ||
|
|
||
| * `use_uld_loss` – toggles Universal Logit Distillation. Set this to `True` for cross-tokenizer setups. | ||
| * `teacher_tokenizer_name_or_path` – required when `use_uld_loss=True`; GOLD uses the teacher tokenizer to align tokens. | ||
| * `uld_use_hybrid_loss`, `uld_hybrid_matched_weight`, `uld_hybrid_unmatched_weight` – enables and weights the hybrid | ||
| matched/unmatched loss. | ||
| * `beta`, `lmbda`, `seq_kd` – inherited from `GKDConfig`, controlling the generalized JSD interpolation and on-policy | ||
| sampling ratio. | ||
|
|
||
| A minimal end-to-end example: | ||
|
|
||
| ```python | ||
| from datasets import load_dataset | ||
| from trl import GOLDConfig, GOLDTrainer | ||
| from transformers import AutoModelForCausalLM, AutoTokenizer | ||
|
|
||
| student_name = "meta-llama/Llama-3.2-1B-Instruct" | ||
| teacher_name = "Qwen/Qwen2.5-0.5B-Instruct" | ||
|
|
||
| tokenizer = AutoTokenizer.from_pretrained(student_name) | ||
| if tokenizer.pad_token is None: | ||
| tokenizer.pad_token = tokenizer.eos_token | ||
|
|
||
| model = AutoModelForCausalLM.from_pretrained(student_name) | ||
| teacher_model = AutoModelForCausalLM.from_pretrained(teacher_name) | ||
|
|
||
| train_dataset = load_dataset( | ||
| "HuggingFaceTB/OpenR1-Math-220k-default-verified", | ||
| "all", | ||
| split="train[:1024]", | ||
| ) | ||
|
|
||
| training_args = GOLDConfig( | ||
| output_dir="gold-model", | ||
| per_device_train_batch_size=1, | ||
| teacher_model=teacher_name, | ||
| teacher_tokenizer_name_or_path=teacher_name, | ||
| use_uld_loss=True, | ||
| uld_use_hybrid_loss=True, | ||
| uld_hybrid_matched_weight=0.6, | ||
| uld_hybrid_unmatched_weight=0.4, | ||
| ) | ||
|
|
||
| trainer = GOLDTrainer( | ||
| model=model, | ||
| teacher_model=teacher_model, | ||
| args=training_args, | ||
| processing_class=tokenizer, | ||
| train_dataset=train_dataset, | ||
| ) | ||
| trainer.train() | ||
| ``` | ||
|
|
||
| ### Expected dataset type | ||
|
|
||
| GOLD requires a [conversational](dataset_formats#conversational) [language modeling](dataset_formats#language_modeling) dataset, e.g.: | ||
|
|
||
| ```python | ||
| {"messages": [{"role": "user", "content": "What color is the sky?"}, | ||
| {"role": "assistant", "content": "It is blue."}]} | ||
| ``` | ||
|
|
||
| `GOLDTrainer` keeps the raw messages so the ChatML collator can construct prompts and completions with the correct | ||
| boundaries. | ||
|
|
||
| ## GOLDTrainer | ||
|
|
||
| [[autodoc]] GOLDTrainer | ||
| - train | ||
| - generate_on_policy_outputs | ||
| - save_model | ||
| - push_to_hub | ||
|
|
||
| ## GOLDConfig | ||
|
|
||
| [[autodoc]] GOLDConfig | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.