Skip to content

[Fix] Share the MTP draft's embed/lm_head before the KV pool is profiled - #37155

Open
IlyaNizamov wants to merge 1 commit into
sgl-project:mainfrom
IlyaNizamov:fix/mtp-draft-embed-before-kv-profile
Open

IlyaNizamov wants to merge 1 commit into
sgl-project:mainfrom
IlyaNizamov:fix/mtp-draft-embed-before-kv-profile

Conversation

@IlyaNizamov

@IlyaNizamov IlyaNizamov commented Aug 30, 2026 •

Copy link
Copy Markdown

Motivation

On MTP-style speculative decoding the draft model builds its own vocab-sized
embed_tokens + ParallelLMHead while loading, and gives both back to the target's
tensors in EagleDraftWorker.init_lm_head() → set_embed_and_head(). That release
currently happens inside alloc_memory_pool(), i.e. after
Scheduler.init_target_memory_pool() has already profiled free memory and sized the
KV cache. The freed bytes are therefore never offered to the KV cache — they stay
unused for the lifetime of the process.

The cost is 2 * vocab_size * hidden_size * dtype / tp per rank, so it grows with the
vocabulary. On Qwen3.8-27B-NVFP4 (vocab 248320, hidden 5120, tp=2) that is 2.4 GiB
per GPU
. The server log shows it directly — memory appears out of nowhere right after
the pool has been carved up:

Memory pool end.                          avail mem=0.29 GB
Capture target verify CUDA graph begin.   avail mem=2.12 GB   <- 1.83 GB arrived too late
...
max_total_num_tokens=13303, context_len=8192, available_gpu_mem=1.82 GB

On a 2x16 GB box this is the difference between an 8k and a 128k context: the same model
served by vLLM with --speculative-config '{"method": "mtp", ...}' reaches
--max-model-len 131072, because vLLM binds the shared vocab layers at construction time.

Modifications

Move the share to just before the target pool is profiled, and make the existing call
site idempotent:

  • EagleDraftWorkerBase.share_target_embed_and_head() — new, guarded by
    _embed_and_head_shared; runs init_token_map() (when the subclass has one) and
    init_lm_head().
  • BaseSpecWorker.share_target_embed_and_head() — delegates to the draft worker when it
    implements the hook. No-op for NGRAM (no draft worker) and for DFlash/DSpark, whose
    draft is a plain TpModelWorker.
  • Scheduler.init_memory_pools() — calls the hook before init_target_memory_pool().
  • EagleDraftWorker.alloc_memory_pool() / MultiLayerEagleDraftWorker.alloc_memory_pool()
    — call the hook instead of init_lm_head() directly; the guard makes it a no-op.

No change for drafts that keep their own vocab layers: StandaloneDraftWorker.init_lm_head()
is already a no-op override, and FrozenKVMTPDraftWorker binds in __init__, before
profiling, which is exactly the ordering this PR generalises.

Accuracy Tests

Not applicable — no change to any forward path or to which tensors the draft ends up
using, only to when the duplicates are released. Output of the shared server before
and after is produced by the same weights; spec_accept_length was 2.9–3.0 in both.

Speed Tests and Profiling

Measured on 2x RTX 5070 Ti (16 GB, tp=2, no NVLink), unsloth/Qwen3.8-27B-NVFP4,
--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --kv-cache-dtype fp8_e4m3:

before after
max_total_num_tokens 13303 136047
largest usable --context-length 8192 131072
stranded VRAM per GPU 1.82 GB 0.21 GB

A 115573-token prompt is served in 61 s after the change; before it is rejected because
the pool cannot hold it. Decode throughput at short context is unchanged (~93 tok/s).

Regression-checked on the same box that the hook stays a no-op where it should:
--speculative-algorithm NGRAM still starts normally (max_total_num_tokens=106396).

Note on provenance: the numbers above were produced with the identical change applied
to the v0.5.17 release running on that hardware; the diff here is the same design ported
to main, where the affected code is unchanged.

Checklist

🤖 Generated with Claude Code

https://claude.ai/code/session_01TsVjhGJomr7tEt5LHegz1f


CI States

Latest PR Test (Base): ❌ Run #33329331493
Latest PR Test (Extra): ❌ Run #33329331376
Latest PR Test (AMD ROCm 7.2): ❌ Run #33329331490

MTP-style draft models build their own vocab-sized embed_tokens and
ParallelLMHead while loading, then hand both back to the target's tensors
in init_lm_head(). That release ran inside alloc_memory_pool(), i.e. after
Scheduler.init_target_memory_pool() had already profiled free memory and
sized the KV cache, so the freed bytes were never offered to the pool and
stayed unused for the life of the process.

The waste is 2 * vocab_size * hidden_size * dtype / tp per rank: 2.4 GiB
per GPU for Qwen3.8-27B-NVFP4 (vocab 248320, hidden 5120) at tp=2, which
on 2x16 GB is the difference between an 8k and a 128k context
(max_total_num_tokens 13303 -> 136047).

Add EagleDraftWorkerBase.share_target_embed_and_head(), guarded so it runs
once, and call it from Scheduler.init_memory_pools() before the target pool
is sized; the existing call sites in EagleDraftWorker.alloc_memory_pool()
and MultiLayerEagleDraftWorker.alloc_memory_pool() now go through the hook
and hit the guard. BaseSpecWorker delegates only when the draft worker
implements the hook, so NGRAM (no draft worker) and DFlash/DSpark (plain
TpModelWorker draft) are unaffected. StandaloneDraftWorker keeps its own
vocab layers via its no-op init_lm_head() override, and
FrozenKVMTPDraftWorker already binds in __init__ - the ordering this
change generalises.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsVjhGJomr7tEt5LHegz1f
@geecraft23

Copy link
Copy Markdown

I independently reproduced this issue and validated the ordering change from this PR on a different hardware topology.

Environment

  • GPU: single NVIDIA RTX 5090 32 GB
  • Model: RadixArk/Qwen3.8-27B-NVFP4
  • SGLang commit: 5f55db35e9
  • CUDA: 13.0.3
  • TP: 1
  • KV cache: fp8_e4m3
  • Multimodal support enabled
  • EAGLE configuration:
    • --speculative-num-steps 3
    • --speculative-eagle-topk 1
    • --speculative-num-draft-tokens 4
    • --enable-linear-replayssm-spec
  • --max-running-requests 1

What I changed

This was a revision-specific port of the mechanism in this PR, not a verbatim application of its full diff.

The local patch makes the same ordering change:

  1. Before Scheduler.init_target_memory_pool() is called, the scheduler asks the EAGLE worker to share the target embedding and LM head.
  2. The EAGLE draft worker runs init_token_map() followed by init_lm_head().
  3. The operation is guarded so the existing call during draft-pool allocation becomes a no-op.
  4. EagleDraftWorker.alloc_memory_pool() calls the guarded sharing method instead of directly calling init_token_map() and init_lm_head().

The scheduler-side port is equivalent to:

if self.draft_worker is not None:
    share = getattr(
        self.draft_worker,
        "share_target_embed_and_head",
        None,
    )
    if share is not None:
        share()

self.init_target_memory_pool()

The local EagleDraftWorker implementation is equivalent to:

def share_target_embed_and_head(self):
    if getattr(self, "_embed_and_head_shared", False):
        return

    self.init_token_map()
    self.init_lm_head()
    self._embed_and_head_shared = True

I also added a small delegating method on EAGLEWorkerV2 so the scheduler can reach the inner EagleDraftWorker.

The upstream PR is more complete and general:

  • It defines the guarded operation on EagleDraftWorkerBase.
  • It adds generic delegation through BaseSpecWorker.
  • It updates both EagleDraftWorker and MultiLayerEagleDraftWorker.
  • It preserves no-op behavior for speculative algorithms that do not use this sharing path.

My local port only covers the standard EAGLEWorkerV2 path used by this Qwen3.8 deployment. I did not modify or validate the MultiLayer EAGLE path.

For the container deployment, I bind-mounted the two patched source files into the pinned SGLang image. A launch-time monkey patch in the parent process was insufficient because the scheduler starts in a fresh child process and imported the original modules again.

Memory result

Before applying the change, requesting a 245,760-token pool produced:

max_total_tokens=245760 is larger than the profiled value 126859
KV Cache ... #tokens: 126859

After moving the sharing operation before target KV profiling:

Shared target embedding/lm_head before target KV profiling
KV Cache ... #tokens: 245760
max_total_num_tokens=245760

This confirms that the draft embedding/LM-head memory was being released only after the target KV pool had already been sized. The same issue therefore occurs on TP=1, not only on the TP=2 configuration described in the PR.

A 245,760-token pool left only about 0.42 GiB after CUDA graph capture and later OOMed while Triton compiled a first-request kernel. Reducing the operational pool to 229,376 tokens left enough compilation headroom and was stable in the tests below. This appears to be normal capacity tuning after recovering the previously stranded memory, rather than a problem with the sharing change.

Validation at 229,376 tokens

  • Health check remained HTTP 200 after all tests.
  • Six independent generation requests, with outputs up to 2,048 tokens:
    • minimum: 121.61 token/s
    • median: 126.09 token/s
    • maximum: 178.15 token/s
    • 6/6 requests exceeded 100 token/s
  • Long-context recall:
    • prompt: 82,549 tokens
    • TTFT: 19.157 seconds
    • decode: 157.17 token/s
    • the model correctly recalled information placed at the beginning
  • Multimodal correctness: 6/6 tests passed, including OCR, counting, colors, and a three-image cross-reference request.
  • Observed speculative metrics after the tests:
    • mean acceptance length: 2.575
    • draft acceptance rate: 0.525
  • Post-test GPU memory: approximately 31,549 MiB used and 540 MiB free.

Scope note: this was a single-GPU, single-running-request validation on a pinned SGLang revision. I did not run the full upstream backend and distributed test matrix.

This independently confirms both the memory-accounting issue and the effectiveness of performing the embedding/LM-head sharing before target KV-cache profiling. I hope these additional results are useful for review. I can provide the exact launch configuration and the complete revision-specific patch if helpful.

detain added a commit to detain/sglang that referenced this pull request Oct 20, 2026
…/deferred/avoid map

- 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619,
  SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420,
  mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821.
- 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202,
  PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565,
  draft embed/lm_head VRAM sgl-project#37155.
- sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39).
- stack_want.txt: execution-status banner. Branch tip 5dd6ec7.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants