Notebook-first tutorial for LLM algorithms and systems.
面向大模型算法与系统的 Notebook-first 教程。
中文版 (Chinese) | English Version
这是一个面向大模型入门到进阶的算法实战教程，以 LLM 为主线，帮助读者通过可运行、可验证、可回顾的 Notebook，从“会看”走到“会写、会调、会优化”。A practical tutorial with theory, walkthroughs, test cases, and solutions.
- 主线清晰：从基础能力到 Triton / CUDA 系统优化，形成完整学习链。
- 工程导向：以 Notebook 实战为载体，强调动手实现与性能意识。
- 覆盖广泛：从 PyTorch、Transformer 到推理优化、显存管理与底层实现都有对应内容。
- 求职面试者：巩固 LLM 算法工程师、AI 架构师、算子开发工程师的高频考点。
- AI 研发人员：从代码底层理解显存优化、分布式通信与 Triton/CUDA 算子。
如果你不想从 Part 顺序硬读，可以先从下面这些专题进入： 如果你只想先解决一个问题，可以直接从专题进入。
这套教程分为纵深主线、横切专题和共学沉淀三层：Part 0 和 Part 1 是共同前置，Part 2 -> Part 5 是主线实战层，topic_discussion 承载 profiling、编译与图优化、推理优化、通信与并行、显存优化与性能调优等跨 Part 主题，team_study 则单独作为动态共学沉淀层，当前主要对应 Part 2。整体关系可以理解为前置打底 -> PyTorch 主线 -> Triton -> CUDA，横切专题和组队学习分别服务于性能分析、编译视野和共学沉淀。
纵向主线负责把知识按层次搭起来，保证学习路径完整、能力递进清晰；横向专题负责把分散在不同 Part 里的方法论和案例重新串联起来，补足故事性、整体性和跨章节的理解闭环。
这套教程不要求从 00 开始按顺序硬读。00 主要是前置补齐区，如果你已有基础，可以直接从最相关的部分开始；下面这张表会直接告诉你：每一部分学什么、包含哪些组、适合谁、当前进度如何。
| 专题 | 覆盖范围 | 内容定位 | 适合对象 | 状态 |
|---|---|---|---|---|
Profiling 专题 |
所有part | 导读：intro；正文：casebook。性能意识、profiling 方法、瓶颈定位经验。 | 想系统补性能意识与排障方法的学习者。 | 🛠 建设中 |
编译与图优化专题 |
Part 1-4 | 导读：intro；正文：casebook。图优化、融合、lowering、调度和代码生成视角。 | 想理解图级优化与编译链路的学习者。 | 🛠 建设中 |
推理优化专题 |
Part 1-2 | 导读：intro；正文：casebook。FlashAttention、解码、PagedAttention、cache 优化。 | 想系统理解推理加速路径的学习者。 | 🛠 建设中 |
通信与并行专题 |
Part 1-2 | 导读：intro；正文：casebook。NCCL、AllReduce、ZeRO、Pipeline、TP。 | 想理解多卡训练和通信边界的学习者。 | 🛠 建设中 |
显存优化与性能调优专题 |
Part 0-2 | 导读：intro；正文：casebook。VRAM、activation、checkpointing、offload、benchmark。 | 想系统优化显存和端到端性能的学习者。 | 🛠 建设中 |
| 模块 | 覆盖范围 | 内容定位 | 适合对象 | 状态 |
|---|---|---|---|---|
组队学习专题 |
不固定 | part2_l1_202606 / part2_l1_202607 / part2_l2_202607 |
想通过共学沉淀知识、题目与复盘记录的学习者。 | 🛠 建设中 |
- 2026-07-10：[最新更新点]收紧了中文版首页的教材总览与状态列，校正了
Part 0/Part 1的组名、节数和0E标题，并同步了相关导航与最近更新说明。 - 2026-06-26：[最新更新点]收紧了中文版首页的教材总览、状态列和 mermaid 关系图，明确了
Part 0-1的前置关系、Part 2-5的主线关系，以及横向专题和组队学习的定位。 - 2026-06-15：推进第零部分 / 第一部分的分组与导读收口，统一部分级导航，并完成网页底部评论区接入 GitHub Discussions，同时持续扩展第一部分的正文、桥接页与 Notebook 结构。
- 2026-06-13：修复 dead link，并为未完成页面补充占位页，避免学习入口出现 404。
- 2026-04-21：更新 Colab 徽章链接，统一指向官方
datawhalechina仓库。 - 2026-04-20：上线站点首页与部分导学；新增第零部分前置知识与第一部分练习内容，完善在线阅读入口与学习路径。
- 2026-04-18 ~ 2026-04-19：集中重构第二部分 / 第三部分内容，优化 Notebook、答案区与算子实现说明。
- 2026-04-02：完成教程核心 Notebook、文档与测试脚本的初始搭建。
路径兼容说明：第三部分已从
03_CUDA_and_Triton_Kernels更名为03_Triton_Kernels，CUDA / 系统优化内容拆分到第四部分。旧网页路径会保留迁移入口，建议新链接统一使用03_Triton_Kernels。
如果你想开始学习，不需要从 00 按顺序起步；在线站点的导学和目录是入口，不是硬性起点。Part 0 适合补基础，Part 1 / 2 / 3 / 4 可以按你的目标直接切入。需要运行 Notebook 时，Part 0 / 1 / 2 可以优先走 CPU-first，Part 3 / 4 需要 GPU 环境。环境与平台差异见 使用指南。
- 在左侧侧边栏选择你当前最关心的部分
- 点击 📖 完整导学 了解该部分的阅读顺序
- 直接从对应 group 进入，不必先补完全部前置
- 如果后面遇到知识缺口，再回到 Part 0 / Part 1 补基础
- 环境和平台差异见 使用指南
访问在线站点：
https://datawhalechina.github.io/llm-algo-leetcode/
适合：
- 先看目录再决定从哪一部分切入
- 先读部分导学，按目标跳转到对应 group
- Part 0 / 1 / 2 可以直接用 Colab CPU 跑练习
- Part 3 / 4 需要 Colab GPU runtime
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
conda env create -f environment.yml
conda activate llm_algo
jupyter lab适合：
- 想在本地完整跑 Part 0 / 1 / 2 的 Notebook
- 想自己控制 Python / PyTorch / CUDA 版本
- 想做更稳定的离线调试
- Part 3 / 4 需要本地 NVIDIA GPU
如果你希望和仓库当前推荐环境保持一致，可以使用 CNB 统一环境入口。
适合：
- 团队协作
- 统一实验镜像
- 需要减少本地环境差异
- Part 0 / 1 / 2 可以用 CNB CPU
- Part 3 / 4 需要 CNB GPU 会话
CNB 的具体使用方式和适用范围见 使用指南。
| 姓名 | 职责 | 简介 |
|---|---|---|
| lynn_jingjing | 项目发起人 | 一个算法工程师 |
本仓库所有 .ipynb 文件中的文字内容（Markdown 单元格、公式、图示说明）采用 CC BY 4.0 协议；代码内容（Code 单元格、可执行实现）采用 Apache-2.0 协议。使用、转载、改编时，请按单元格类型分别遵守对应协议。文字协议见 LICENSE，代码协议见 LICENSE-CODE。
All .ipynb files in this repository are mixed-content notebooks: Markdown cells (tutorial text, formulas, and figure captions) are licensed under CC BY 4.0, while Code cells (executable implementations) are licensed under Apache-2.0. Please comply with the corresponding license by cell type when using, redistributing, or adapting this repository. See LICENSE for text and LICENSE-CODE for code.
This is a practical LLM algorithm tutorial from beginner to advanced, built around runnable, verifiable notebooks that help you move from "reading" to "writing, debugging, and optimizing".
- Clear Main Line: A complete learning chain from prerequisites to Triton / CUDA system optimization.
- Engineering-Oriented: Notebook-based practice with hands-on implementation and performance awareness.
- Broad Coverage: Covers PyTorch, Transformers, inference optimization, VRAM management, and low-level implementation.
- Job Seekers: Reinforce common interview topics for LLM algorithm engineers, AI architects, and kernel developers.
- AI Practitioners: Understand VRAM optimization, distributed communication, and Triton/CUDA operators from the code level.
This tutorial is organized into a vertical main line and cross-cutting tracks: the main line connects Part 0 -> Part 4 (with Part 5 reserved), topic_discussion covers profiling, compiler and graph optimization, inference optimization, communication and parallelism, and memory and performance tuning topics, and team_study is maintained as a separate collaborative-learning lane. The overview is summarized in the asset and topic tables below.
You do not need to start from 00 in strict order. 00 is the prerequisite lane; if you already have the background, jump directly to the part that matches your goal. The table below summarizes each part, its groups, its audience, and its status.
| Topic | Coverage | Content Positioning | Suitable For | Status |
|---|---|---|---|---|
| Topic Discussion Axis | All parts | Cross-Part topic discussion and case stitching. | Learners who want to consolidate methods and cases across parts. | 🛠 In progress |
| Profiling Topic | All parts | Guide: intro; casebook: casebook. Performance awareness, profiling methods, bottleneck localization. | Learners who want systematic performance diagnosis and debugging methods. | 🛠 In progress |
| Compiler and Graph Optimization Topic | Part 1-4 | Guide: intro; casebook: casebook. Graph optimization, fusion, lowering, scheduling, and code generation. | Learners who want compiler and graph-level optimization vision. | 🛠 In progress |
| Inference Optimization Topic | Part 1-2 | Guide: intro; casebook: casebook. FlashAttention, decoding, PagedAttention, and cache optimization. | Learners who want to understand practical inference acceleration. | 🛠 In progress |
| Communication and Parallelism Topic | Part 1-2 | Guide: intro; casebook: casebook. NCCL, AllReduce, ZeRO, Pipeline, and Tensor Parallelism. | Learners who want to understand multi-GPU scaling and communication cost. | 🛠 In progress |
| Memory and Performance Tuning Topic | Part 0-2 | Guide: intro; casebook: casebook. VRAM, activation, checkpointing, offload, and benchmark. | Learners who want to optimize memory usage and end-to-end performance. | 🛠 In progress |
| Module | Coverage | Content Positioning | Suitable For | Status |
|---|---|---|---|---|
| Team Study Topic | Not fixed | part2_l1_202606 / part2_l1_202607 / part2_l2_202607 | Learners who want to accumulate knowledge and review records through collaborative study. | 🛠 In progress |
- 2026-07-10: [Latest update] tightened the English homepage asset overview and status columns, aligned the part/group counts with the current source structure, and refreshed the topic and team-study status tables.
- 2026-06-26: [Latest update] improved the Chinese homepage overview and clarified the learning path across Parts 3 and 4, making the entry points and study order more intuitive.
- 2026-06-15: Finalized the Part 0 / 1 grouping and guide cleanup, unified the part-level navigation, connected the page comments to GitHub Discussions, and continued expanding Part 1 content, bridge pages, and notebook structure.
- 2026-06-13: Fixed dead links and added placeholder pages for unfinished content to prevent 404s in learning entry points.
- 2026-04-21: Updated Colab badges to point to the official
datawhalechinarepository. - 2026-04-20: Launched the site homepage and part guides; added Part 0 prerequisites and Part 1 practice content to unify the learning path.
- 2026-04-18 ~ 2026-04-19: Refactored Part 2 / 3 content, polishing notebooks, answer sections, and operator implementation notes.
- 2026-04-02: Completed the initial tutorial notebooks, docs, and test scripts.
Path compatibility note: Part 3 has been renamed from
03_CUDA_and_Triton_Kernelsto03_Triton_Kernels, and CUDA / system optimization content has moved to Part 4. Old web paths keep migration pages, but new links should use03_Triton_Kernels.
You do not need to start from Part 0 in order; Part 0 is the prerequisite lane, and you can jump directly to the part that matches your goal.
Visit the online platform:
https://datawhalechina.github.io/llm-algo-leetcode/
Suitable for:
- Skimming the table of contents first and then jumping to the part you need
- Reading the part guides first
- Part 0 / 1 / 2 can run on Colab CPU
- Part 3 / 4 need a Colab GPU runtime
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
conda env create -f environment.yml
conda activate llm_algo
jupyter labSuitable for:
- Running Part 0 / 1 / 2 locally on CPU
- Controlling your own Python / PyTorch / CUDA versions
- More stable offline debugging
- Part 3 / 4 require a local NVIDIA GPU
For environment details and platform differences, see the Chinese guide section or docs/guide.md.
If you want the same runtime style used by the repository, use the CNB unified environment.
Suitable for:
- Team collaboration
- Consistent experiment images
- Lower local environment drift
- Part 0 / 1 / 2 can use CNB CPU
- Part 3 / 4 need a CNB GPU session
See docs/guide.md for the exact environment rules and scope.
- docs/guide.md - environment and learning modes
- docs/contributing.md - how to contribute to development and testing
- docs/maintenance.md - maintenance rules for parts, links, tests, and releases
- Automated Test Script Index - entry points for automated verification scripts
| Name | Role | Description |
|---|---|---|
| lynn_jingjing | Project initiator | An algorithm engineer |
(Feel free to add your name here! )
Tutorial text in this repository is licensed under CC BY 4.0, and code is licensed under Apache-2.0. .ipynb files are mixed-content notebooks, so please follow the corresponding license by cell type.