[mpashkovskii]

Summary

Adds a single developer guide, docs/design/rocm_decode_megakernel.md, documenting how to
write a decode megakernel ("MonoKernel" / specialized decode) for a dense transformer on AMD
ROCm (MI300X / MI355X, CDNA3 / CDNA4) inside vLLM — the technique behind the in-flight AMD mono
decode work (DeepSeek-V4.1 #60397/#61082, Qwen3.8-27B dense #61573, Kimi-K3 #60683/#61435,
GLM-5.2 #60889) and the NVIDIA CuTeDSL specialized decode (#61867), standardized by RFC #60904.

It is top-down: architecture → compute contract → toolchain → a from-zero FlyDSL bootcamp →
step-by-step implementation → testing → an executable task breakdown → pitfalls → reference
appendices. It uses a Gemma-style dense FP8 model as the worked example (the closest template
is the dense Qwen3.8-27B path), but the technique sections are model-agnostic.

Status: DRAFT. This lands the material as one page so reviewers can react to the content and
agree on its home/vocabulary; a proposed multi-page split to fit the docs layout is below and
would be done as a follow-up (or folded into this PR) once there's agreement.

Why this is not a duplicate

• Checked gh pr list --repo vllm-project/vllm --state all --search "<mono decode / megakernel / specialized decode / kernel authoring> docs/guide in:title" and
  gh issue view 60904 — no existing documentation PR covers this.
• RFC [RFC]: A standard for Specialized (MonoKernels) Decodes in vllm/models #60904 ("A standard for Specialized (MonoKernels) Decodes in vllm/models") is the
  umbrella design discussion; the related PRs ([ROCm][Perf][DSv4.1] Mono decode layer for DeepSeek-V4.1-Flash on MI355X #60397, [ROCm][Perf][Qwen3.8] Add dense mono decode for Qwen3.8-27B on MI355X. #61573, [ROCm][Perf][Kimi-K3] Specialized decode for Kimi-K3 on MI355X #60683, [ROCm] MonoKernel for GLM 5.2 #60889, [Core] Shared abstractions for specialized decode kernels #61311, [Perf] Kimi K3 Specialized Decode: CuTeDSL Optimized GB300 (1.5-2x) #61867…)
  are code, not end-to-end authoring documentation. This page is the missing how-to that ties
  them together and points readers at the canonical sources.

Proposed split (the analysis requested)

The page is intentionally one file today, but it maps cleanly onto vLLM's docs conventions, which
separate docs/design/ ("how it works" — e.g. paged_attention.md, cuda_graphs.md,
model_runner_v2.md, fused_moe_modular_kernel.md) from docs/contributing/ ("how to build"
— e.g. model/, profiling.md). A megakernel guide has both halves, so the natural seam is:

docs/design/specialized_decode/ — conceptual

Page	From current parts	Scope
index.md	Part 0 (TL;DR) + Part 1	What/why, the AMD mono family, execution model, tagged-mailbox + peer-memory sync, the K1/K2 split, where it plugs into vLLM
compute_contract.md	Part 2	What to fuse (model-agnostic) + the Gemma dense worked example
toolchain.md	Part 3	FlyDSL, the reusable device primitives, CDNA3/4 hardware contract, AITER

docs/contributing/specialized_decode/ — how-to

Page	From current parts	Scope
index.md	new (scope/prereqs + links)	Entry point, prerequisites, nav
flydsl_bootcamp.md	Part 3.5	EX0–EX7 hands-on warm-ups (write → run → measure → understand)
implementation.md	Part 4	Steps A–K (directory layout, eligibility, build keys, K1/K2, runner, TP, integration, warm-up, fail-stop)
testing.md	Part 5	Numerics / accuracy / performance validation
task_breakdown.md	Part 6	The milestone checklist
pitfalls.md	Part 7	Gotchas from the real PRs
reference.md	Appendices A/B/C	File map, PR index, external links

Rationale:

• Design vs contributing seam mirrors the repo and lets each page carry a single # H1 and a
  focused nav entry, instead of this draft's many # PART … sections in one file.
• awesome-nav auto-includes files in a directory; a small .nav.yml per subdir would set
  order/titles. No root nav: edits are needed (the design dir has no .nav.yml and auto-navs
  today).
• Vocabulary: I used directory name specialized_decode to match the in-tree package
  (vllm/models/common/specialized_decode/) and RFC [RFC]: A standard for Specialized (MonoKernels) Decodes in vllm/models #60904's terminology. Open question for
  reviewers: prefer specialized_decode or monokernel? And should the conceptual half
  live in docs/design/ while the how-to lives in docs/contributing/, or should the whole thing
  sit under one tree?
• The Gemma example stays inline (it's a teaching device for the technique), not under
  docs/models/, since the guide is about the method, not a specific model.

Test plan / commands run

• markdownlint-cli2 --config tools/pre_commit/.markdownlint.yaml docs/design/rocm_decode_megakernel.md
  → 0 issues (single H1, code-fence languages, blanks, table style, no bare URLs — matches the
  repo's CI config).
• Heading tree verified: 1× H1 (title) + H2 parts/appendices, proper MD001 increments.
• Not run here (no GPU / full docs toolchain in this environment): mkdocs build --strict and a
  rendered-nav check — please let CI exercise these, or I can add a .nav.yml + run it in the
  split follow-up.

Caveats for reviewers

• The referenced mono-decode PRs are mostly unmerged and still converging (the enable switch is
  moving VLLM_ROCM_MONO_DECODE → --monokernel → --additional-config decode_backend per
  [RFC]: A standard for Specialized (MonoKernels) Decodes in vllm/models #60904). The guide flags this and the stable-vs-in-flux parts explicitly; paths/flags should be
  re-checked against tip before anyone implements from it.
• Content is AMD/ROCm + gfx950-first; CDNA3 (gfx942) notes are included but secondary.

Disclosure

AI assistance was used to research the referenced PRs/sources and draft this document. As the
human submitter I will review every line, keep it in sync with RFC #60904, and own it end-to-end;
the DCO sign-off should be re-applied with the submitter's identity before this leaves draft.

[mergify]

Documentation preview: https://vllm--62256.org.readthedocs.build/en/62256/