Skip to content

Qwen3.8-Flash-Next DSpark: MoE drafter, MTP warm start, HF / vLLM offline capture - #983

Draft
fg11991 wants to merge 6 commits into
sgl-project:mainfrom
fg11991:1015_main_next_flash
Draft

fg11991 wants to merge 6 commits into
sgl-project:mainfrom
fg11991:1015_main_next_flash

Conversation

@fg11991

@fg11991 fg11991 commented Oct 16, 2026

Copy link
Copy Markdown

Branch: fg11991/SpecForge-upstream:1015_main_next_flash, based on sgl-project/SpecForge@fdcc2ed.

Motivation

Qwen3.8-Flash-Next (qwen4_exp) is a 48-layer MoE target with a 4-stream gated
residual, a PLE n-gram embedding and one MTP layer. Training a DSpark drafter
for it in SpecForge was blocked on two fronts:

  1. The drafter. The DSpark drafters here are dense. Flash-Next's MTP layer,
    the natural initialization, is MoE with gated attention and partial RoPE, so
    a dense drafter can neither mirror it nor warm-start from it.
  2. Feature capture. Offline and online preparation both run the target in
    SGLang. SGLang supports qwen4_exp on GPU (support qwen 3.8 flash next sglang#37500), but
    there is no hidden-state capture hook for it, and the Ascend path is still
    an open PR ([Feat] [NPU] Qwen3.8-Flash-Next: Support graph replay, MTP, and Triton kernels sglang#37570). We train on Ascend.

This PR adds the drafter and two capture backends that do not depend on SGLang,
without changing any existing default.

What changes

Commit Change
feat: MoE DSpark drafter for Qwen3.8-Flash-Next QwenSparseMoE (Qwen3-Next routing + sigmoid-gated shared expert, Switch balance loss), attn_output_gate and partial_rotary_factor on the DFlash-family attention (both off by default), padded blocks masked out of the balance term, MoE drafters built directly in the requested dtype, configs qwen3.8-flash-next-dspark-moe{64,512}.json, offline NPU recipe
feat: warm-start ... from the target's MTP layer scripts/warm_start_from_mtp.py writes a pretrained draft directory for model.draft_checkpoint_path
test: Qwen3.8 training render and loss mask ... pins that the qwen3.5 template + ThinkingParser loss mask match what a server prompts, turn by turn
feat: Transformers offline capture backend ... prepare_hidden_states.py --target-backend hf
feat: prepare DSpark features from a running vLLM ... prepare_hidden_states.py --target-backend vllm
docs: ... data-preparation guide section and example README entry

Not in this PR: expert parallelism. The 64-expert arm (2.0B) trains with DDP.
The 512-expert arm (13.0B) needs EP and is shipped only as a config, so warm
start and serving can already target it.

How it works

Drafter

  • MoE FFN. Softmax over all experts, top-10, renormalized weights, plus one
    shared expert scaled by a sigmoid gate. With the same weights it matches
    Transformers' Qwen4ExpTextSparseMoeBlock to 1e-6. Load balancing uses the
    Switch auxiliary loss (router_aux_loss_coef, 0.001) rather than a
    selection bias, so serving has nothing extra to reproduce. moe_aux_loss,
    moe_max_load_ratio and moe_unused_expert_frac are logged as means over
    the MoE layers.
  • Padding. Padded DSpark blocks, which the objective already excludes, are
    masked out of the balance term too. The mask reaches the MoE layers as a
    forward argument, so a gradient-checkpointed recompute uses the same mask;
    a test checks that router gradients match with and without checkpointing.
  • Attention. q_proj emits [query | gate] per head, as in the target. The
    attention output is multiplied by sigmoid(gate) before o_proj. RoPE turns
    64 of the 256 channels, with cos/sin equal to the target's text-only M-RoPE.
  • Anchor alignment. The training block matches the serving query block. The
    bonus token sits in slot 0 at position a, labels start at x_{a+1}, and no
    target state at or after a is visible. A test pins this under both sdpa and
    eager. When we let the anchor row through, the test fails, which is the leak
    that once cost a DeepSeek-V4 DSpark fine-tune its serving acceptance.

Capture taps

DSpark tap i is layers[i + 1].attn_hyper_connection(streams)[0]: the
normalized state layer i + 1 reads out of the 4-stream residual. The configs
use the full-attention layers [3, 15, 23, 35, 43]. Every capture path,
including a future SGLang one, must take exactly this state.

--target-backend hf

  • One process drives the node and spreads consecutive decoder layers over the
    visible devices.
  • The ~95 GiB n-gram table stays in host memory. Module.to() would first copy
    it onto one 64 GB card, so layer placement stops at host-resident subtrees.
  • Two reference paths are replaced for speed:
    • The QSA indexer becomes one vectorized score matrix; its masks are
      bit-identical to the reference.
    • Expert dispatch takes one host readback per forward instead of one per
      expert.
  • The capture equals the reference model run with the same fast paths
    (torch.equal).
  • Rows run one at a time over their real tokens and are saved at their real
    length; each record is trimmed before the host copy, so it stores only its
    own rows.
  • Needs transformers>=5.16.1 in the capture environment only.

--target-backend vllm

  • Talks to a vLLM / vLLM-Ascend server started with extract_hidden_states and
    ExampleHiddenStatesConnector.
  • Server prerequisite. Upstream vLLM / vLLM-Ascend cannot yet serve
    Flash-Next this way: vLLM's Qwen4Exp config check admits only the native MTP
    and n-gram proposers, and the model emits no auxiliary hidden states. The
    qwen38-capture
    branch of fg11991/vllm-ascend carries the changes (8cc7129 allows
    extract_hidden_states, 1de1b0b serves the auxiliary states, plus
    KV-cache sizing fixes for the capture path). It has run on one Atlas 800 A3
    node: W8A8, TP8, --enforce-eager, native MTP off, --max-model-len 2304.
  • The connector writes each file asynchronously, with or without its
    synchronization lock, so the lock is the only sign a file is complete. The
    backend refuses a capture without its .lock file and asks for
    use_synchronization_lock (the connector's default).
  • Requests carry token ids, so the loss mask comes from SpecForge's own
    rendering.
  • The server's layer ids are the taps plus one, followed by num_hidden_layers.
    vLLM numbers a layer's input, SpecForge a layer's output; for Flash-Next the
    ids are [4, 16, 24, 36, 44, 48].
  • Each row is checked: echoed token ids, length and finiteness.
  • Rows are sent unpadded and saved at their real length.
  • The client loads no model. The preparation entry reads the target config
    through load_target_config, falling back to the raw config.json as
    training does, so the repository's pinned Transformers is enough.

Testing

Unit tests (macOS arm64, CPU)

Environment Result
repo pins: torch 2.13.0, transformers 5.12.1 new tests pass; the qwen4_exp-dependent ones skip. Full suite has the same failures as main on this machine (sglang / triton / yunchang are not installable on macOS).
torch 2.13.0, transformers 5.19.0 all Flash-Next tests pass (76 passed, 4 skipped), including the HF capture against the reference model and the MoE block / RoPE against qwen4_exp

The real-tokenizer template test passed on the cluster with the released
Qwen3.8-Flash-Next tokenizer (QWEN38_TOKENIZER_PATH set, 15 passed, none
skipped): each turn's serving prompt is a token prefix of the training ids and
the loss mask covers exactly the generated tokens.

End to end on Ascend

Run with the development branch these commits were ported from. Its drafter,
loss and capture code are the same as here; at expert-parallel size 1 the MoE
outputs, balance loss and all gradients were checked equal on CPU.

Capture (--target-backend vllm): one Atlas 800 A3 node serving the W8A8
checkpoint from the qwen38-capture branch (TP8, --enforce-eager, native
MTP off, --max-model-len 2304, --max-num-seqs 4); client concurrency and
--batch-size 4. 95 conversations were captured in 5 min 45 s, about 3.6 s
per row.

Training: the 64-expert drafter (the recipe in this PR) from random
initialization on those 95 rows; 8 NPUs, DDP (NO_SHARD), batch 1 ×
accumulation 5 × 8 ranks = 40 samples per step, max_length 2304,
transformers 5.16.1, 720 steps at 9.3–11 s per step. That is about 300 passes
over 95 rows, so this is a pipeline and learning check, not a quality result.

Metric Step 2 Step 8 Steps 717–720
ce_loss 12.57 (≈ ln 248320) 8.92 1.26–1.53
acc 0.00 0.036 0.63–0.68
accuracy_position 0 / 6 0.00 / 0.00 0.035 / 0.037 0.63–0.68 / 0.62–0.67
tau_probabilistic 1.00 1.03 4.03–4.29
teacher_agreement 0.00 0.032 0.84–0.85
moe_aux_loss (10 = balanced) 30.0 24.4 13.2–13.6
moe_max_load_ratio 6.39 6.40 2.40–2.47
moe_unused_expert_frac 0.48 0.66 ≈ 0.002
  • The loss starts at the uniform-vocabulary value, so labels and vocabulary
    line up, and has fallen to 1.3–1.5 by the end. Gradient norm spikes to 28
    once in the early steps (step 6, in warmup, before the spike detector has a
    baseline) and sits near 0.04 at the end, with no step skipped there.
  • At initialization most draft slots hold the same mask embedding, so every
    token routes to the same ten experts: the load ratio sits at its ceiling
    E / k = 6.4 and two thirds of the experts go unused by step 8. The balance
    loss then spreads the load until nearly every expert is used.
  • teacher_agreement is how often the drafter's top-1 matches the target's,
    scored against the final state the vLLM capture returned; its climb to 0.85
    is an indirect check that the captured state is the one lm_head reads.

512 experts and MTP warm start (development branch only: the 512-expert
drafter needs expert parallelism, which this PR does not add). Same 95 rows,
EP8 on 8 NPUs, batch 1 × accumulation 5 = 5 samples per step (EP ranks share
their tokens), the same 720-step schedule, 6.6–9 s per step after the first.

  • scripts/warm_start_from_mtp.py on the released W8A8 checkpoint (its MTP
    layer is BF16) copies 7,735 of the drafter's 7,752 tensors: per layer the
    attention projections, q/k norms, router, 512 × 3 expert projections,
    shared expert and its gate. The 17 left are the parts with no MTP
    counterpart: the input projection and its norm, the layer and final norms,
    and the Markov and confidence heads.
  • From scratch and from the warm start the runs see identical batches, so
    their steps compare directly:
Metric Scratch, step 1 Warm start, step 1 Warm start, steps 1–11 Scratch, steps 717–720
ce_loss 13.14 6.19 5.64–7.03 5.94–6.53
acc 0.00 0.14 0.10–0.22 0.15–0.20
tau_probabilistic 1.00 1.95 1.54–2.49 1.57–2.06
teacher_agreement 0.00 0.40 0.28–0.53 0.30–0.39
moe_max_load_ratio (cap 51.2) 49.8 12.9 11.9–25.4 10.1–13.9
moe_unused_expert_frac 0.84 0.43 0.40–0.50 0.38–0.41
  • On its first step the warm-started drafter is where the scratch run ends
    after 720 steps (3,600 samples), and its router starts spread out instead
    of collapsed onto ten experts.
  • Open: while the learning rate warms up, the warm-started router loses
    balance (max load ratio 12.9 → 25.4 and gradient norm 0.13 → about 2 over
    steps 1–11); whether it recovers by the end of the run is not yet known.
    The 512-expert runs also see 8× fewer samples per step than the 64-expert
    run, so the two do not compare step for step, and the scratch 512 run ends
    with about 40% of experts idle in a step against 0.2% for 64 experts. Its
    balance coefficient may need tuning once it trains on real data.

Not run at scale:

  • The hf backend: only the W8A8 checkpoint is available on the cluster, and
    the Transformers reference model cannot load it. It is checked bit for bit
    against Transformers on a small randomly initialized qwen4_exp model.
  • Acceptance length: needs the drafter served by vLLM-Ascend; see the
    follow-ups.

Follow-ups once SGLang runs Flash-Next with hidden-state capture

  1. SGLang capture. Add a qwen4_exp hook to the SGLang capture patch that
    reads the tap above. That turns on the default --target-backend sglang
    and online / disaggregated training for this target.
  2. Cross-check. Compare SGLang-captured rows against the HF backend on the
    same tokens before trusting them. The HF backend stays as the oracle and as
    the fallback for hardware SGLang does not cover.
  3. Expert parallelism. Train the 512-expert arm once EP lands here.
  4. vLLM / vLLM-Ascend. Propose the qwen38-capture changes upstream, so
    the vllm backend no longer depends on a fork.
  5. Serving. Bring the drafter to SGLang speculative decoding for
    Flash-Next. It needs gated attention, partial RoPE and the MoE FFN on the
    DSpark draft path.
  6. Recipe. Once a drafter is trained and evaluated, add a runbook under
    docs/recipes/ with the exact revisions, commands and acceptance numbers.

🤖 Generated with Claude Code

w00958190 added 6 commits October 15, 2026 17:28
Qwen3.8-Flash-Next is MoE (512 routed experts of width 640, top-10, one
sigmoid-gated shared expert), so its DSpark drafter is MoE too. A
DSparkDraftModel config with num_experts > 0 swaps each layer's dense MLP for
QwenSparseMoE: Qwen3-Next routing (softmax over all experts, top-k,
renormalized weights) plus the gated shared expert, numerically equal to
Transformers' Qwen4ExpTextSparseMoeBlock on the same weights. Balance is the
Switch auxiliary loss scaled by router_aux_loss_coef, added to the DSpark
objective. moe_aux_loss, moe_max_load_ratio and moe_unused_expert_frac are
logged as means over the MoE layers. Padded DSpark blocks, which the
objective already excludes, are masked out of the balance term as well; the
mask reaches the MoE layers as a forward argument rather than module state,
so a gradient-checkpointed recompute sees the same mask.

The drafter mirrors the target's MTP layer so it can be warm-started from it:
attn_output_gate adds the gated attention output (q_proj emits
[query | gate] per head, the output is scaled by its sigmoid before o_proj)
and partial_rotary_factor rotates 64 of the 256 head channels, with cos/sin
equal to the target's text-only M-RoPE. Both default off, so existing drafters
are unchanged. The rotary config folds the factor into head_dim and resets it
to 1, so a Transformers release that also applies partial_rotary_factor inside
the rotary embedding cannot shrink the slice twice.

AutoDraftModel.from_config builds drafters with num_experts > 0 directly in
the requested dtype: built in fp32 and cast afterwards, the 13B 512-expert
drafter peaks at twice its bf16 size. Dense drafters keep their fp32
construction, so their random initialization does not move.

configs/qwen3.8-flash-next-dspark-moe{64,512}.json (2.0B / 13.0B) differ only
in num_experts. The offline NPU recipe trains the 64-expert arm with DDP on
one 8-card node; the 512-expert arm needs expert parallelism, which this
repository does not have yet.

Tests pin the training block to the serving query block (anchor at slot 0,
no target state at or after the anchor read, under sdpa and eager), the
balance term against the Switch formula with and without padding (also
under a checkpointed recompute), the gated attention, and -- with transformers>=5.16.1 -- the MoE block and partial RoPE
against Qwen3.8-Flash-Next's own modules.
scripts/warm_start_from_mtp.py reads mtp.layers.0.* from a local Flash-Next
checkpoint and writes a pretrained draft directory for
model.draft_checkpoint_path. Every drafter layer takes the MTP layer's gated
attention, router, routed experts (the stacked gate_up_proj split into
gate_proj / up_proj) and shared expert; the q/k norms gain one, since the
target's RMSNorm scales by 1 + w and the drafter's by w. The input projection,
layer norms and Markov / confidence heads -- nothing in the MTP layer
corresponds -- keep their fresh initialization, and the indexer is dropped.

A drafter with fewer experts names the MTP experts it starts from
(--expert-ids, or --expert-selection stride). The mapping hands out views
into the stacked experts instead of copies; the 512-expert arm peaks at about
32 GiB of host memory.

Tests: the copied MoE computes what Qwen4ExpTextSparseMoeBlock computes on the
same tensors; unmapped tensors are untouched; the script's output
warm-starts a drafter with nothing missing; and the released checkpoint's MTP
shapes, read from its safetensors headers, fit both shipped drafter configs.
Qwen3.8 recipes use the qwen3.5 template entry, which renders a conversation
once with enable_thinking=True and supervises each assistant span. That is
right only if, for every assistant turn, the prompt a server builds is a
prefix of the training text and the supervised span is exactly what the model
generates. The test renders the released Jinja both ways for single-turn,
multi-turn and tool-calling conversations, and runs ThinkingParser.parse() to
check per turn that the serving-prompt tokens prefix the input ids and the
loss mask covers exactly the generated tokens plus the newline after
<|im_end|>. A byte-level tokenizer carrying Qwen3.8's added tokens runs this
offline; with QWEN38_TOKENIZER_PATH set, the real tokenizer runs the same
checks.

It also pins the serving settings that break the match, which a corpus and an
evaluation must avoid: a reasoning_effort other than xhigh,
preserve_thinking=False, and enable_thinking=False.

The template is vendored byte for byte (sha256 pinned; identical in
Qwen3.8-27B and Qwen3.8-Flash-Next), so the whitespace hooks skip vendored
templates.
`prepare_hidden_states.py --target-backend hf` captures DFlash / DSpark
features with the Transformers reference model, for targets no local serving
engine runs; currently Qwen3.8-Flash-Next (qwen4_exp). One process spreads
consecutive decoder layers over the visible accelerators and keeps the
~95 GiB n-gram table in host memory, where its lookup already runs.
Accelerate's device_map cannot express that -- it treats "cpu" as offload and
would stream the table onto the execution device every forward.

Layer placement moves each module's own tensors and stops at host-resident
subtrees: Module.to() would recurse into the n-gram table and copy it onto one
64 GB card first. A table found off the host is refused; the regression tests
dispatch onto the meta device, from which a transient migration cannot come
back.

A DSpark tap i reads layers[i + 1].attn_hyper_connection(streams)[0]: the
normalized state layer i + 1 itself reads out of the 4-stream residual. Taps
whose next layer carries PLE, or that have no next layer, are refused.

Two reference paths are replaced for speed. The QSA indexer's per-query
Python loop becomes one score matrix over fixed 4-token blocks (masks are
bit-identical to the reference in fp32 and bf16; padded or cached inputs fall
back), and the routed experts sort their assignments once instead of a host
sync per expert. The capture test compares rows against a reference running
the same fast paths with torch.equal; the fast paths' agreement with the stock
forward is tested separately, since top-k ties at score 0 resolve differently
between per-query and vectorized calls on aarch64.

Rows run one at a time over their real tokens: right padding is dropped
before the forward and the states are zero-padded back so the rows stack.
The backend says so through unpadded_rows, and the preparation script then
saves each record at its attention-mask length instead of the batch's padded
one. It trims before the host copy, so a saved tensor owns only its kept
rows; a view of the padded copy would serialize the whole padded storage.

Needs transformers>=5.16.1 in the capture environment only; the tests that
build the model skip without it.
Add --target-backend vllm to prepare_hidden_states.py. The backend sends each
row's real tokens to a vLLM / vLLM-Ascend server running extract_hidden_states
with ExampleHiddenStatesConnector (/v1/completions, prompt as token ids,
max_tokens=1), waits on the connector's file lock, and splits the [T, K, H]
capture into the taps concatenated in ascending order and the final state
lm_head reads. The server's layer ids must be the taps plus one (vLLM numbers
a layer's input, SpecForge its output) followed by num_hidden_layers; the
backend prints that value and refuses a capture with the wrong layer count.
Echoed and saved token ids, T, and finiteness are checked per row.

Rows from one batch are requested concurrently and sent without their right
padding (unpadded_rows), so --batch-size can match the server's concurrency.
Rendering, loss masks, the record layout and resumption are the existing
pipeline's.

The client loads no model, so it should not need a Transformers release that
knows the target: the preparation entry now reads the target config through
load_target_config, which falls back to the raw config.json as training
already does, and build_target_model no longer requires a dtype field on it.
With the repository's pinned Transformers, Qwen3.8-Flash-Next (qwen4_exp) was
otherwise refused before any request is sent.

The connector writes each file asynchronously whether or not it keeps its
synchronization lock, so the lock is the only sign a file is complete. A
capture without its .lock file is refused with a pointer to
use_synchronization_lock (the connector's default) instead of being read
while it may still be written.

Tests run the backend and HiddenStatesGenerator.generate against a fake
connector server; the saved records pass normalize_dspark_offline_sample as
3-D hidden states. The entry reaches distributed setup with a target whose
model type Transformers does not register.
The data preparation guide gains a section on the hf and vllm capture
backends -- when to use them, the vLLM layer-id convention, batch size versus
server concurrency, path mapping -- with a Flash-Next command, and on the MTP
warm start. The example-config README introduces the Flash-Next recipe.

It also states what the vllm backend needs from the server: the
synchronization lock kept on, and -- for Qwen3.8-Flash-Next, which upstream
vLLM / vLLM-Ascend cannot yet serve with extract_hidden_states -- the
qwen38-capture branch of fg11991/vllm-ascend, with the configuration it has
been run in.
@fg11991 fg11991 mentioned this pull request Oct 17, 2026
3 of 4 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant