Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c3e4744
intial
kashif Oct 27, 2025
1966d1f
initial tests
kashif Oct 27, 2025
59bef64
add test_uldloss_hybrid_config_beta_zero
kashif Oct 27, 2025
954aa88
update sleep mode
kashif Oct 28, 2025
f8b7212
Merge branch 'main' into gold
kashif Oct 28, 2025
007f80a
chatml test and docs
kashif Oct 28, 2025
406d3ed
fix formatting and mark heavier tests as slow
kashif Oct 28, 2025
d2f4e6b
edge case when student/teacher have no overlap in hybrid loss
kashif Oct 28, 2025
0ed3b34
typos
kashif Oct 28, 2025
05a044c
style in doc
qgallouedec Oct 28, 2025
b32acf6
citation as class attr
qgallouedec Oct 28, 2025
9fde612
use student_model_revision
kashif Oct 28, 2025
5bcd11f
Dropped the deprecated formatting_func
kashif Oct 28, 2025
be88e8e
use _set_signature_columns_if_needed
kashif Oct 28, 2025
bae7df5
Removed the dataset_kwargs["skip_prepare_dataset"] hack
kashif Oct 28, 2025
e141e52
use create_model_from_path helper
kashif Oct 28, 2025
7871dfd
better names
kashif Oct 28, 2025
dfb1310
moved imports to top level
kashif Oct 28, 2025
624ec4c
remove student_ prefix from vllm configs
kashif Oct 28, 2025
7061253
Update trl/experimental/gold/gold_config.py
kashif Oct 28, 2025
e36a11c
Update trl/experimental/gold/gold_trainer.py
kashif Oct 28, 2025
9acf7eb
use max_completion_length
kashif Oct 28, 2025
4e69b12
use relative imports
kashif Oct 28, 2025
62f18e8
avoid hasattr
kashif Oct 28, 2025
a2acc33
use ensure_master_addr_port
kashif Oct 28, 2025
aab8b02
Update trl/experimental/gold/gold_trainer.py
kashif Oct 28, 2025
e2dfff5
Update trl/experimental/gold/gold_trainer.py
kashif Oct 28, 2025
0a0c79a
minimal example
kashif Oct 28, 2025
cb587c2
remove padding when calling generate_batch
kashif Oct 29, 2025
3382b1e
use lightweight _GoldDataCollator
kashif Oct 29, 2025
260e920
initial gold.py script
kashif Oct 29, 2025
eb9ad55
fix args
kashif Oct 29, 2025
e504832
Revert "use lightweight _GoldDataCollator"
kashif Oct 29, 2025
841939f
update the _set_signature_columns_if_needed
kashif Oct 29, 2025
0e7b4b8
fix the if statement
kashif Oct 29, 2025
e325322
test generate_on_policy_outputs
kashif Oct 29, 2025
78ecfda
add test_generate_on_policy_outputs_masks_prompt_smollm
kashif Oct 29, 2025
232f8b6
use TinyLlama
kashif Oct 29, 2025
76fb22b
update _paper link
kashif Oct 29, 2025
33b009a
add default lr
kashif Oct 29, 2025
060f9ba
nit
qgallouedec Oct 29, 2025
b8217fe
Merge branch 'main' into gold
qgallouedec Oct 29, 2025
4f27fce
fix autodoc
qgallouedec Oct 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@
- sections:
- local: bco_trainer
title: BCO
- local: gold_trainer
title: GOLD
- local: openenv
title: OpenEnv Integration
title: Experimental
99 changes: 99 additions & 0 deletions docs/source/gold_trainer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# General Online Logit Distillation (GOLD) Trainer

[![All_models-GOLD-blue](https://img.shields.io/badge/All_models-GOLD-blue)](https://huggingface.co/models?other=sft,gold)

## Overview

General Online Logit Distillation (GOLD) is an extension of Universal Logit Distillation (ULD) that supports
student/teacher pairs with different tokenizers. It aligns the textual spans produced by both tokenizers and merges the
associated logits so no completion tokens are dropped. This enables cross-tokenizer knowledge distillation, including
mixed model families (for example, LLaMA students with Qwen teachers).

Key capabilities:

1. **Cross-tokenizer alignment** – GOLD incrementally decodes the student and teacher tokens, groups passages with the same visible text, and merges probabilities inside each group. This guarantees loss terms are computed over the full completion even when token boundaries differ.
2. **Hybrid ULD loss** – when `uld_use_hybrid_loss` is enabled, GOLD compares exact vocabulary matches directly and falls back to the original sorted-probability ULD loss for unmatched tokens. This improves stability for students whose vocabularies only partially overlap with the teacher.
3. **Seamless integration with GKD** – GOLD inherits the on-policy vs. off-policy scheduling from the [`GKDTrainer`](./gkd_trainer.md), so you can combine sequence-level KD, generalized JSD, and cross-tokenizer distillation in a single training run.

> [!NOTE]
> GOLD is currently part of the `trl.experimental` namespace. APIs may change without notice while the feature is iterated on.

## Usage tips

The [`GOLDTrainer`] subclasses [`SFTTrainer`] and accepts the same datasets as other TRL trainers (lists of ChatML style
messages). Important configuration flags on [`GOLDConfig`] include:

* `use_uld_loss` – toggles Universal Logit Distillation. Set this to `True` for cross-tokenizer setups.
* `teacher_tokenizer_name_or_path` – required when `use_uld_loss=True`; GOLD uses the teacher tokenizer to align tokens.
* `uld_use_hybrid_loss`, `uld_hybrid_matched_weight`, `uld_hybrid_unmatched_weight` – enables and weights the hybrid
matched/unmatched loss.
* `beta`, `lmbda`, `seq_kd` – inherited from `GKDConfig`, controlling the generalized JSD interpolation and on-policy
sampling ratio.

A minimal end-to-end example:

```python
from datasets import load_dataset
from trl import GOLDConfig, GOLDTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

student_name = "meta-llama/Llama-3.2-1B-Instruct"
teacher_name = "Qwen/Qwen2.5-0.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(student_name)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(student_name)
teacher_model = AutoModelForCausalLM.from_pretrained(teacher_name)
Comment thread
kashif marked this conversation as resolved.
Resolved

train_dataset = load_dataset(
"HuggingFaceTB/OpenR1-Math-220k-default-verified",
"all",
split="train[:1024]",
)

training_args = GOLDConfig(
output_dir="gold-model",
per_device_train_batch_size=1,
teacher_model=teacher_name,
teacher_tokenizer_name_or_path=teacher_name,
use_uld_loss=True,
uld_use_hybrid_loss=True,
uld_hybrid_matched_weight=0.6,
uld_hybrid_unmatched_weight=0.4,
)

trainer = GOLDTrainer(
model=model,
teacher_model=teacher_model,
args=training_args,
processing_class=tokenizer,
train_dataset=train_dataset,
)
trainer.train()
```

### Expected dataset type

GOLD requires a [conversational](dataset_formats#conversational) [language modeling](dataset_formats#language_modeling) dataset, e.g.:

```python
{"messages": [{"role": "user", "content": "What color is the sky?"},
{"role": "assistant", "content": "It is blue."}]}
```

`GOLDTrainer` keeps the raw messages so the ChatML collator can construct prompts and completions with the correct
boundaries.

## GOLDTrainer

[[autodoc]] GOLDTrainer
- train
- generate_on_policy_outputs
- save_model
- push_to_hub

## GOLDConfig

[[autodoc]] GOLDConfig
Loading