Repository navigation
[Fix] Share the MTP draft's embed/lm_head before the KV pool is profiled - #37155
IlyaNizamov wants to merge 1 commit into
Conversation
MTP-style draft models build their own vocab-sized embed_tokens and ParallelLMHead while loading, then hand both back to the target's tensors in init_lm_head(). That release ran inside alloc_memory_pool(), i.e. after Scheduler.init_target_memory_pool() had already profiled free memory and sized the KV cache, so the freed bytes were never offered to the pool and stayed unused for the life of the process. The waste is 2 * vocab_size * hidden_size * dtype / tp per rank: 2.4 GiB per GPU for Qwen3.8-27B-NVFP4 (vocab 248320, hidden 5120) at tp=2, which on 2x16 GB is the difference between an 8k and a 128k context (max_total_num_tokens 13303 -> 136047). Add EagleDraftWorkerBase.share_target_embed_and_head(), guarded so it runs once, and call it from Scheduler.init_memory_pools() before the target pool is sized; the existing call sites in EagleDraftWorker.alloc_memory_pool() and MultiLayerEagleDraftWorker.alloc_memory_pool() now go through the hook and hit the guard. BaseSpecWorker delegates only when the draft worker implements the hook, so NGRAM (no draft worker) and DFlash/DSpark (plain TpModelWorker draft) are unaffected. StandaloneDraftWorker keeps its own vocab layers via its no-op init_lm_head() override, and FrozenKVMTPDraftWorker already binds in __init__ - the ordering this change generalises. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TsVjhGJomr7tEt5LHegz1f
|
I independently reproduced this issue and validated the ordering change from this PR on a different hardware topology. Environment
What I changedThis was a revision-specific port of the mechanism in this PR, not a verbatim application of its full diff. The local patch makes the same ordering change:
The scheduler-side port is equivalent to: if self.draft_worker is not None:
share = getattr(
self.draft_worker,
"share_target_embed_and_head",
None,
)
if share is not None:
share()
self.init_target_memory_pool()The local def share_target_embed_and_head(self):
if getattr(self, "_embed_and_head_shared", False):
return
self.init_token_map()
self.init_lm_head()
self._embed_and_head_shared = TrueI also added a small delegating method on The upstream PR is more complete and general:
My local port only covers the standard For the container deployment, I bind-mounted the two patched source files into the pinned SGLang image. A launch-time monkey patch in the parent process was insufficient because the scheduler starts in a fresh child process and imported the original modules again. Memory resultBefore applying the change, requesting a 245,760-token pool produced: After moving the sharing operation before target KV profiling: This confirms that the draft embedding/LM-head memory was being released only after the target KV pool had already been sized. The same issue therefore occurs on TP=1, not only on the TP=2 configuration described in the PR. A 245,760-token pool left only about 0.42 GiB after CUDA graph capture and later OOMed while Triton compiled a first-request kernel. Reducing the operational pool to 229,376 tokens left enough compilation headroom and was stable in the tests below. This appears to be normal capacity tuning after recovering the previously stranded memory, rather than a problem with the sharing change. Validation at 229,376 tokens
Scope note: this was a single-GPU, single-running-request validation on a pinned SGLang revision. I did not run the full upstream backend and distributed test matrix. This independently confirms both the memory-accounting issue and the effectiveness of performing the embedding/LM-head sharing before target KV-cache profiling. I hope these additional results are useful for review. I can provide the exact launch configuration and the complete revision-specific patch if helpful. |
…/deferred/avoid map - 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619, SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420, mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821. - 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202, PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565, draft embed/lm_head VRAM sgl-project#37155. - sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39). - stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
Motivation
On MTP-style speculative decoding the draft model builds its own vocab-sized
embed_tokens+ParallelLMHeadwhile loading, and gives both back to the target'stensors in
EagleDraftWorker.init_lm_head()→set_embed_and_head(). That releasecurrently happens inside
alloc_memory_pool(), i.e. afterScheduler.init_target_memory_pool()has already profiled free memory and sized theKV cache. The freed bytes are therefore never offered to the KV cache — they stay
unused for the lifetime of the process.
The cost is
2 * vocab_size * hidden_size * dtype / tpper rank, so it grows with thevocabulary. On
Qwen3.8-27B-NVFP4(vocab 248320, hidden 5120, tp=2) that is 2.4 GiBper GPU. The server log shows it directly — memory appears out of nowhere right after
the pool has been carved up:
On a 2x16 GB box this is the difference between an 8k and a 128k context: the same model
served by vLLM with
--speculative-config '{"method": "mtp", ...}'reaches--max-model-len 131072, because vLLM binds the shared vocab layers at construction time.Modifications
Move the share to just before the target pool is profiled, and make the existing call
site idempotent:
EagleDraftWorkerBase.share_target_embed_and_head()— new, guarded by_embed_and_head_shared; runsinit_token_map()(when the subclass has one) andinit_lm_head().BaseSpecWorker.share_target_embed_and_head()— delegates to the draft worker when itimplements the hook. No-op for NGRAM (no draft worker) and for DFlash/DSpark, whose
draft is a plain
TpModelWorker.Scheduler.init_memory_pools()— calls the hook beforeinit_target_memory_pool().EagleDraftWorker.alloc_memory_pool()/MultiLayerEagleDraftWorker.alloc_memory_pool()— call the hook instead of
init_lm_head()directly; the guard makes it a no-op.No change for drafts that keep their own vocab layers:
StandaloneDraftWorker.init_lm_head()is already a no-op override, and
FrozenKVMTPDraftWorkerbinds in__init__, beforeprofiling, which is exactly the ordering this PR generalises.
Accuracy Tests
Not applicable — no change to any forward path or to which tensors the draft ends up
using, only to when the duplicates are released. Output of the shared server before
and after is produced by the same weights;
spec_accept_lengthwas 2.9–3.0 in both.Speed Tests and Profiling
Measured on 2x RTX 5070 Ti (16 GB, tp=2, no NVLink),
unsloth/Qwen3.8-27B-NVFP4,--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --kv-cache-dtype fp8_e4m3:max_total_num_tokens--context-lengthA 115573-token prompt is served in 61 s after the change; before it is rejected because
the pool cannot hold it. Decode throughput at short context is unchanged (~93 tok/s).
Regression-checked on the same box that the hook stays a no-op where it should:
--speculative-algorithm NGRAMstill starts normally (max_total_num_tokens=106396).Note on provenance: the numbers above were produced with the identical change applied
to the v0.5.17 release running on that hardware; the diff here is the same design ported
to
main, where the affected code is unchanged.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01TsVjhGJomr7tEt5LHegz1f
CI States
Latest PR Test (Base): ❌ Run #33329331493
Latest PR Test (Extra): ❌ Run #33329331376
Latest PR Test (AMD ROCm 7.2): ❌ Run #33329331490