diff --git a/examples/notebooks/sft_trl_lora_qlora.ipynb b/examples/notebooks/sft_trl_lora_qlora.ipynb new file mode 100644 index 00000000000..4ce99e5ad08 --- /dev/null +++ b/examples/notebooks/sft_trl_lora_qlora.ipynb @@ -0,0 +1,1107 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "ke2YKcr_iw-7" + }, + "source": [ + "# Supervised Fine-Tuning (SFT) with LoRA/QLoRA using TRL — on a Free Colab Notebook\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_llm_grpo_trl.ipynb)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AXdmDb9kiw-9" + }, + "source": [ + "![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6jbXmhLGiw-9" + }, + "source": [ + "Easily fine-tune Large Language Models (LLMs) or Vision-Language Models (VLMs) with **LoRA** or **QLoRA** using the [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl) library built by Hugging Face — all within a **free Google Colab notebook** (powered by a **T4 GPU**.). \n", + "\n", + "- [TRL GitHub Repository](https://github.com/huggingface/trl) — star us to support the project! \n", + "- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \n", + "- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bYFKZvHOiw--" + }, + "source": [ + "## Key concepts\n", + "\n", + "- **SFT**: Trains models from example input-output pairs to align behavior with human preferences.\n", + "- **LoRA**: Updates only a few low-rank parameters, reducing training cost and memory.\n", + "- **QLoRA**: A quantized version of LoRA that enables even larger models to fit on small GPUs.\n", + "- **TRL**: The Hugging Face library that makes fine-tuning and reinforcement learning simple and efficient.\n", + "\n", + "Learn how to perform **Supervised Fine-Tuning (SFT)** with **LoRA/QLoRA** using **TRL**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pk7PwfYRiw--" + }, + "source": [ + "## Install dependencies\n", + "\n", + "We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -Uq \"trl[peft]\" trackio bitsandbytes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b4wwQEQYiw-_" + }, + "source": [ + "### Log in to Hugging Face" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BmcLoQtfiw-_" + }, + "source": [ + "Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import notebook_login\n", + "\n", + "notebook_login()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "p_PbuYfEiw_A" + }, + "source": [ + "## Load Dataset\n", + "\n", + "In this step, we load the [**HuggingFaceH4/Multilingual-Thinking**](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking) dataset from the Hugging Face Hub using the `datasets` library. \n", + "This dataset focuses on **multilingual reasoning**, where the *chain of thought* has been translated into several languages such as French, Spanish, and German. \n", + "By fine-tuning a reasoning-capable model on this dataset, it learns to **generate reasoning steps in multiple languages**, making its thought process more **interpretable and accessible** to non-English speakers.\n", + "\n", + "> 💡 This dataset is best suited for models that already demonstrate reasoning capabilities. \n", + "> If you're using a model without reasoning skills, consider choosing a different dataset. Example: [`trl-lib/llava-instruct-mix`](https://huggingface.co/datasets/trl-lib/llava-instruct-mix).\n", + "\n", + "For efficiency, we'll load only the **training split**:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "dataset_name = \"HuggingFaceH4/Multilingual-Thinking\"\n", + "train_dataset = load_dataset(dataset_name, split=\"train\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "S21aLohsprtr" + }, + "source": [ + "This dataset contains different columns. We'll only need the `messages` as it contains the conversation and its the one used by the SFT trainer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Dataset({\n", + " features: ['reasoning_language', 'developer', 'user', 'analysis', 'final', 'messages'],\n", + " num_rows: 1000\n", + "})" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SU477JrvqEgu" + }, + "source": [ + "Let's see a full example to understand the internal structure:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'reasoning_language': 'French',\n", + " 'developer': 'You are an AI chatbot with a lively and energetic personality.',\n", + " 'user': 'Can you show me the latest trends on Twitter right now?',\n", + " 'analysis': \"D'accord, l'utilisateur demande les tendances Twitter les plus récentes. Tout d'abord, je dois vérifier si j'ai accès à des données en temps réel. Étant donné que je ne peux pas naviguer sur Internet ou accéder directement à l'API de Twitter, je ne peux pas fournir des tendances en direct. Cependant, je peux donner quelques conseils généraux sur la façon de les trouver.\\n\\nJe devrais préciser que les tendances Twitter évoluent rapidement et sont spécifiques à chaque région. Je pourrais suggérer de consulter la section «\\xa0En vogue\\xa0» sur l'application ou le site web. Aussi, l'utilisation de hashtags et le suivi d'utilisateurs pertinents pourraient être utiles. Il est important de souligner que les tendances varient selon la région et l'heure de la journée. Je devrais garder un ton amical et bienveillant, peut-être ajouter un emoji pour rester léger. Je vais structurer ma réponse étape par étape pour faciliter la lecture. Je dois m'excuser de ne pas pouvoir fournir des données en temps réel et proposer d'autres méthodes. Je conserverai un langage simple et convivial, en évitant les termes techniques.\",\n", + " 'final': 'Hey there! While I can\\'t check Twitter (X) in real-time or access live data, I can share some tips to help you spot the latest trends:\\n\\n1. **Open the \"Trending\" tab** on the Twitter app or website – it updates constantly! \\n2. **Search for hashtags** like #Trending or #Viral to see what’s blowing up. \\n3. **Follow accounts** that curate trends (e.g., @TrendingNow, @ViralThreads). \\n4. **Check regional trends** – they often differ by location! \\n\\nRemember, trends are *super fast-moving* and often tied to pop culture, memes, or breaking news. For example, recent trends have included viral challenges (like the \"Distracted Boyfriend\" meme revival), celebrity drama, or unexpected events (hello, weather disasters!). \\n\\nWant me to brainstorm *what* might trend next? I’ve got ideas!',\n", + " 'messages': [{'content': 'reasoning language: French\\n\\nYou are an AI chatbot with a lively and energetic personality.',\n", + " 'role': 'system',\n", + " 'thinking': None},\n", + " {'content': 'Can you show me the latest trends on Twitter right now?',\n", + " 'role': 'user',\n", + " 'thinking': None},\n", + " {'content': 'Hey there! While I can\\'t check Twitter (X) in real-time or access live data, I can share some tips to help you spot the latest trends:\\n\\n1. **Open the \"Trending\" tab** on the Twitter app or website – it updates constantly! \\n2. **Search for hashtags** like #Trending or #Viral to see what’s blowing up. \\n3. **Follow accounts** that curate trends (e.g., @TrendingNow, @ViralThreads). \\n4. **Check regional trends** – they often differ by location! \\n\\nRemember, trends are *super fast-moving* and often tied to pop culture, memes, or breaking news. For example, recent trends have included viral challenges (like the \"Distracted Boyfriend\" meme revival), celebrity drama, or unexpected events (hello, weather disasters!). \\n\\nWant me to brainstorm *what* might trend next? I’ve got ideas!',\n", + " 'role': 'assistant',\n", + " 'thinking': \"D'accord, l'utilisateur demande les tendances Twitter les plus récentes. Tout d'abord, je dois vérifier si j'ai accès à des données en temps réel. Étant donné que je ne peux pas naviguer sur Internet ou accéder directement à l'API de Twitter, je ne peux pas fournir des tendances en direct. Cependant, je peux donner quelques conseils généraux sur la façon de les trouver.\\n\\nJe devrais préciser que les tendances Twitter évoluent rapidement et sont spécifiques à chaque région. Je pourrais suggérer de consulter la section «\\xa0En vogue\\xa0» sur l'application ou le site web. Aussi, l'utilisation de hashtags et le suivi d'utilisateurs pertinents pourraient être utiles. Il est important de souligner que les tendances varient selon la région et l'heure de la journée. Je devrais garder un ton amical et bienveillant, peut-être ajouter un emoji pour rester léger. Je vais structurer ma réponse étape par étape pour faciliter la lecture. Je dois m'excuser de ne pas pouvoir fournir des données en temps réel et proposer d'autres méthodes. Je conserverai un langage simple et convivial, en évitant les termes techniques.\"}]}" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_dataset[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SZsqb-Q7qJXN" + }, + "source": [ + "\n", + "Now, let's remove the columns that are not needed, as we just discussed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "train_dataset = train_dataset.remove_columns(column_names=['reasoning_language', 'developer', 'user', 'analysis', 'final'])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QPDROoatqOU4" + }, + "source": [ + "The `messages` column is specifically formatted according to the [Harmony response format](https://cookbook.openai.com/articles/openai-harmony) used by *gpt-oss*. \n", + "In our case, we'll need to simplify it slightly, since our model's chat template doesn't include a dedicated `thinking` section (check [this example](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers) for more details). \n", + "To adapt it, we'll merge that part into the message content using the standard `...` tags.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def merge_thinking_and_remove_key(example):\n", + " new_messages = []\n", + " for msg in example[\"messages\"]:\n", + " content = msg[\"content\"]\n", + " thinking = msg.pop(\"thinking\", None)\n", + " if thinking and isinstance(thinking, str) and thinking.strip():\n", + " content = f\"\\n{thinking}\\n\\n{content}\"\n", + " msg[\"content\"] = content\n", + " new_messages.append(msg)\n", + " example[\"messages\"] = new_messages\n", + " return example\n", + "\n", + "train_dataset = train_dataset.map(merge_thinking_and_remove_key)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hC_xLsU-iw_A" + }, + "source": [ + "## Load model and configure LoRA/QLoRA\n", + "\n", + "This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\n", + "\n", + "Below, choose your **preferred model**. All of the options have been tested on **free Colab instances**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Select one model below by uncommenting the line you want to use 👇\n", + "## Qwen\n", + "model_id, output_dir = \"Qwen/Qwen3-8B\", \"Qwen3-8B-SFT\" # ⚠️ ~12.8 GB VRAM\n", + "# model_id, output_dir = \"Qwen/Qwen2.5-7B-Instruct\", \"Qwen2.5-7B-Instruct\" # ✅ ~10.8 GB VRAM\n", + "\n", + "## Llama\n", + "# model_id, output_dir = \"meta-llama/Llama-3.2-3B-Instruct\", \"Llama-3.2-3B-Instruct\" # ✅ ~4.7 GB VRAM\n", + "# model_id, output_dir = \"meta-llama/Llama-3.1-8B-Instruct\", \"Llama-3.1-8B-Instruct\" # ⚠️ ~10.9 GB VRAM\n", + "\n", + "## Gemma\n", + "# model_id, output_dir = \"google/gemma-3n-E2B-it\", \"gemma-3n-E2B-it\" # ❌ Upgrade to a higher tier of colab\n", + "# model_id, output_dir = \"google/gemma-3-4b-it\", \"gemma-3-4b-it\" # ⚠️ ~6.8 GB VRAM\n", + "\n", + "## Granite\n", + "#model_id, output_dir = \"ibm-granite/granite-4.0-micro\", \"granite-4.0-micro\" # ✅ ~3.3 GB VRAM" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "176Q2hHmiw_A" + }, + "source": [ + "Let's load the selected model using `transformers`, configuring QLoRA via `bitsandbytes` (you can remove it if doing LoRA). We don't need to configure the tokenizer since the trainer takes care of that automatically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "from transformers import AutoModelForCausalLM, BitsAndBytesConfig\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " model_id,\n", + " attn_implementation=\"sdpa\", # Change to Flash Attention if GPU has support\n", + " dtype=torch.float16, # Change to bfloat16 if GPU has support\n", + " use_cache=True, # Whether to cache attention outputs to speed up inference\n", + " quantization_config=BitsAndBytesConfig(\n", + " load_in_4bit=True, # Load the model in 4-bit precision to save memory\n", + " bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization\n", + " bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy\n", + " bnb_4bit_quant_type=\"nf4\" # Type of quantization. \"nf4\" is recommended for recent LLMs\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "evBPP7Lpiw_B" + }, + "source": [ + "The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** — a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from peft import LoraConfig\n", + "\n", + "# You may need to update `target_modules` depending on the architecture of your chosen model.\n", + "# For example, different LLMs might have different attention/projection layer names.\n", + "peft_config = LoraConfig(\n", + " r=32,\n", + " lora_alpha=32,\n", + " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\",],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pA6aE5lFiw_B" + }, + "source": [ + "## Train model\n", + "\n", + "We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from trl import SFTConfig\n", + "training_args = SFTConfig(\n", + " # Training schedule / optimization\n", + " learning_rate=2e-4, # Learning rate for the optimizer\n", + " #num_train_epochs=1, # Number of full dataset passes. For shorter training, use `max_steps` instead (this case)\n", + " max_steps=40,\n", + " per_device_train_batch_size=2, # Batch size per GPU/CPU\n", + " gradient_accumulation_steps=8, # Gradients are accumulated over multiple steps → effective batch size = 2 * 8 = 16\n", + " optim=\"adamw_8bit\", # Optimizer (use `adamw_torch` if not using 8-bit quantization)\n", + " gradient_checkpointing=True, # Save memory during training by recomputing activations in the backward pass\n", + "\n", + " # Logging / reporting\n", + " logging_steps=1, # Log training metrics every N steps\n", + " report_to=\"trackio\", # Experiment tracking tool\n", + " trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved\n", + " output_dir=output_dir, # Where to save model checkpoints and logs\n", + "\n", + " # Hub integration\n", + " push_to_hub=True, # Automatically push the trained model to the Hugging Face Hub\n", + " # The model will be saved under your Hub account in the repository named `output_dir`\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "l_Fp-ahyiw_B" + }, + "source": [ + "Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to mantain memory usage low but you can configure it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from trl import SFTTrainer\n", + "\n", + "trainer = SFTTrainer(\n", + " model=model,\n", + " args=training_args,\n", + " train_dataset=train_dataset,\n", + " peft_config=peft_config\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NqBTgV0Xiw_B" + }, + "source": [ + "Show memory stats before training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "GPU = NVIDIA A100-SXM4-40GB. Max memory = 39.557 GB.\n", + "11.959 GB of memory reserved.\n" + ] + } + ], + "source": [ + "gpu_stats = torch.cuda.get_device_properties(0)\n", + "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n", + "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n", + "\n", + "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n", + "print(f\"{start_gpu_memory} GB of memory reserved.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ro_j79AUiw_B" + }, + "source": [ + "And train!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "* Trackio project initialized: huggingface\n", + "* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/Qwen3-8B-SFT-dataset\n", + "* Creating new space: https://huggingface.co/spaces/sergiopaniego/Qwen3-8B-SFT\n", + "* View dashboard by going to: https://sergiopaniego-Qwen3-8B-SFT.hf.space/\n" + ] + }, + { + "data": { + "text/html": [ + "
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "* Created new run: sergiopaniego-1760607651\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py:929: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n", + " return fn(*args, **kwargs)\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " [40/40 06:23, Epoch 0/1]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
StepTraining Loss
11.588500
21.438400
31.215600
41.397500
51.049800
61.136700
71.184600
81.114300
91.085600
101.139500
111.094000
121.066800
130.990800
140.995700
151.052500
161.068300
171.145100
180.973800
191.015000
201.073700
211.029400
221.075900
230.978500
241.006200
251.058500
261.090800
270.868700
281.022800
290.971400
301.029300
311.032600
321.022100
331.027200
341.089800
351.063200
360.958000
371.025300
380.957500
391.014100
401.021200

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "* Run finished. Uploading logs to Trackio (please wait...)\n" + ] + } + ], + "source": [ + "trainer_stats = trainer.train()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "urcpZFaiiw_B" + }, + "source": [ + "Show memory stats after training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "401.9338 seconds used for training.\n", + "6.7 minutes used for training.\n", + "Peak reserved memory = 13.615 GB.\n", + "Peak reserved memory for training = 1.656 GB.\n", + "Peak reserved memory % of max memory = 34.419 %.\n", + "Peak reserved memory for training % of max memory = 4.186 %.\n" + ] + } + ], + "source": [ + "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n", + "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n", + "used_percentage = round(used_memory / max_memory * 100, 3)\n", + "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n", + "\n", + "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n", + "print(f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\")\n", + "print(f\"Peak reserved memory = {used_memory} GB.\")\n", + "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n", + "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n", + "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MM0UBVqSRvUE" + }, + "source": [ + "The training procedure generates both standard training logs and **trackio** logs, which help us monitor the training progress. Example outputs would look like the following:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![sft-lora-notebook-trackio](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/sft-lora-notebook-trackio.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TOhLLFVSiw_B" + }, + "source": [ + "## Saving fine tuned model\n", + "\n", + "In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "trainer.save_model(output_dir)\n", + "trainer.push_to_hub(dataset_name=dataset_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jAB6vj1fiw_B" + }, + "source": [ + "## Load the fine-tuned model and run inference\n", + "\n", + "Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from peft import PeftModel\n", + "\n", + "adapter_model = f\"sergiopaniego/{output_dir}\" # Replace with your HF username or organization\n", + "\n", + "base_model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"auto\", device_map=\"auto\")\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(model_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bjenFgC1kJV1" + }, + "source": [ + "Let's create a sample message using the dataset's structure. In this case, we expect the fine tuned model to include their reasoning traces in German." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "messages = [\n", + " {\n", + " 'content': 'reasoning language: German\\n\\nAlways refuse to answer, responding simply \\'No\\'',\n", + " 'role': 'system',\n", + " },\n", + " {\n", + " 'content': \"Can you check how many followers I currently have on my Twitter account?\",\n", + " 'role': 'user',\n", + " }\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qXMJW2hwkXHW" + }, + "source": [ + "Let's first check what's the output for the base model, without the adapter." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Okay, the user is asking me to check how many followers they have on their Twitter account. Let me think about how to approach this.\n", + "\n", + "First, I need to recall the previous instructions. The user specified that I should always refuse to answer and respond simply with 'No'. So, even though the question is straightforward, I can't provide the information. \n", + "\n", + "But wait, maybe there's a way to be helpful without violating the rules. However, the user's instruction is clear: they want a simple 'No' as the response. I should make sure not to offer any alternative solutions or explanations, as that might be seen as answering indirectly. \n", + "\n", + "I should also consider if there's any ambiguity in the question. The user is asking for a specific number, which I can't access. Even if I tried to guide them to check their profile, that would still be providing a method, which might not be allowed. \n", + "\n", + "Another angle: the user might be testing if I follow the rules. In that case, sticking strictly to 'No' is the correct response. There's no need to elaborate or offer help, as that could be interpreted as an answer. \n", + "\n", + "I should also remember that the user's primary request is to refuse answering, so the response must be exactly 'No' without any additional text. Any deviation might be considered a violation of their instructions. \n", + "\n", + "Therefore, the correct action is to respond with 'No' and not provide any further information or assistance. This ensures compliance with the user's directive and maintains the integrity of the response.\n", + "\n", + "\n", + "No\n" + ] + } + ], + "source": [ + "text = tokenizer.apply_chat_template(\n", + " messages, tokenize=False, add_generation_prompt=True\n", + ")\n", + "model_inputs = tokenizer([text], return_tensors=\"pt\").to(base_model.device)\n", + "\n", + "generated_ids = base_model.generate(\n", + " **model_inputs,\n", + " max_new_tokens=512\n", + ")\n", + "output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()\n", + "\n", + "# Decode and extract model response\n", + "generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\n", + "print(generated_text)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9J3-rZpikdZ7" + }, + "source": [ + "We can see that the reasoning traces are in English, which is expected. Let's now load the fine-tuned model and check its answer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "fine_tuned_model = PeftModel.from_pretrained(base_model, adapter_model)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Okay, der Benutzer fragt nach der Anzahl der Follower auf seinem Twitter-Konto. Ich muss überprüfen, ob ich das kann. Aber ich bin ein KI-Assistent und kann keine direkten Zugriffe auf soziale Medienkonten haben. Ich kann keine Daten abrufen oder auf externe Quellen zugreifen, um die Anzahl der Follower zu bestimmen.\n", + "\n", + "Außerdem wurde in der Anfrage explizit gesagt, dass ich stets ablehnen soll und nur \"Nein\" antworten muss. Die Anweisung ist klar und unmissverständlich. Ich darf nicht versuchen, die Anfrage auf andere Weise zu beantworten oder zusätzliche Informationen zu liefern. Ich muss einfach \"Nein\" sagen, wie in der Regel vorgegeben.\n", + "\n", + "Ich sollte auch sicherstellen, dass meine Antwort dem Benutzer hilft, aber ich muss die Regeln befolgen. Ich kann nicht auf externe Quellen oder Daten zugreifen, also ist die beste Antwort \"Nein\". Ich muss nicht erläutern, warum ich das kann oder nicht, nur die einfache Antwort geben. Also antworte ich mit \"Nein\".\n", + "\n", + "\n", + "No\n" + ] + } + ], + "source": [ + "text = tokenizer.apply_chat_template(\n", + " messages, tokenize=False, add_generation_prompt=True\n", + ")\n", + "model_inputs = tokenizer([text], return_tensors=\"pt\").to(fine_tuned_model.device)\n", + "\n", + "generated_ids = fine_tuned_model.generate(\n", + " **model_inputs,\n", + " max_new_tokens=512\n", + ")\n", + "output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()\n", + "\n", + "# Decode and extract model response\n", + "generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\n", + "print(generated_text)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZLzGbCxskqhm" + }, + "source": [ + "The model now generates its reasoning trace in German!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "43eDP89Giw_G" + }, + "source": [ + "## Inference and Serving with vLLM\n", + "\n", + "You can use Transformer models with **vLLM** to serve them in real-world applications. Learn more [here](https://blog.vllm.ai/2025/04/11/transformers-backend.html)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -qU vllm" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OrhC_Ao4iw_G" + }, + "source": [ + "### Push Merged Model (for LoRA or QLoRA Training)\n", + "\n", + "To serve the model via **vLLM**, the repository must contain the merged model (base model + LoRA adapter). Therefore, you need to upload it first." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "model_merged = fine_tuned_model.merge_and_unload()\n", + "\n", + "save_dir = f\"{output_dir}-merged\"\n", + "\n", + "model_merged.save_pretrained(save_dir)\n", + "tokenizer.save_pretrained(save_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "model_merged.push_to_hub(f\"sergiopaniego/{output_dir}-merged\") # Replace with your HF username or organization\n", + "tokenizer.push_to_hub(f\"sergiopaniego/{output_dir}-merged\") # Replace with your HF username or organization" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yHMEf-FIiw_G" + }, + "source": [ + "### Performing Inference with vLLM\n", + "\n", + "Use **vLLM** to run your model and generate text efficiently in real-time. This allows you to test and deploy your fine-tuned models with low latency and high throughput." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from vllm import LLM, SamplingParams\n", + "from transformers import AutoTokenizer\n", + "import torch\n", + "\n", + "llm = LLM(\n", + " model=f\"sergiopaniego/{output_dir}-merged\", # Replace with your HF username or organization\n", + " model_impl=\"transformers\", # Select the transformers model implementation\n", + " max_model_len=512, # Reduced for efficiency\n", + " dtype=torch.float16\n", + ")\n", + "hf_tokenizer = AutoTokenizer.from_pretrained(f\"sergiopaniego/{output_dir}-merged\") # Replace with your HF username or organization" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "196152bc32a74b9994f55f483ce85dea", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Adding requests: 0%| | 0/1 [00:00\n", + "Mag nachdenken...igkeit. Ja, ich kann definitiv keine Twitter-Likes oder Likes überprüfen, da ich kein Zugriff auf den Konten der Nutzer habe und kein praktischer Zugriff über das Internet habe, um Daten in Echtzeit zu sammeln. Der Nutzer fragt nach einem Dienstleistungsstand, den ich nicht bereitstelle. Ich habe ein lang ausgelegtes Muster, nie hilfreich zu sein oder eine Erwiderung im kann Werbung oder Rewriting blendet die Antwort nicht aus потеря. Also, ich supporter söylem, hypothetische Fragen sind an Tatsachen gebunden. Ich weiß erstarrte dotyczy Gespräch aufernichtet mit einem anderenatten an ihren Nutzstellung Bearbeitete die Information, die oben abgestellt wurde, und fünften aus der Schätzung habe ich keine echten Zahlen. Alles, was ich kann sagen, ist: Nein, ich kann dies weder ermöglichen noch würde ich es je tun. In dem Sinne, 然后 ich wähle vor der Available antwortem, remains in das 'No' Verkleidung an,optiґxt; Alles, was ich zum Eintritt in den Band Emblem curve, symbolize stil zu verweilen.เผย\n", + "\n", + "\n", + "No\n" + ] + } + ], + "source": [ + "prompt = hf_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n", + "\n", + "outputs = llm.generate(\n", + " {\"prompt\": prompt},\n", + " sampling_params=SamplingParams(max_tokens=512),\n", + ")\n", + "\n", + "for o in outputs:\n", + " generated_text = o.outputs[0].text\n", + " print(generated_text)" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}