Repository navigation
Conversation
added 6 commits
October 15, 2026 17:28
Qwen3.8-Flash-Next is MoE (512 routed experts of width 640, top-10, one
sigmoid-gated shared expert), so its DSpark drafter is MoE too. A
DSparkDraftModel config with num_experts > 0 swaps each layer's dense MLP for
QwenSparseMoE: Qwen3-Next routing (softmax over all experts, top-k,
renormalized weights) plus the gated shared expert, numerically equal to
Transformers' Qwen4ExpTextSparseMoeBlock on the same weights. Balance is the
Switch auxiliary loss scaled by router_aux_loss_coef, added to the DSpark
objective. moe_aux_loss, moe_max_load_ratio and moe_unused_expert_frac are
logged as means over the MoE layers. Padded DSpark blocks, which the
objective already excludes, are masked out of the balance term as well; the
mask reaches the MoE layers as a forward argument rather than module state,
so a gradient-checkpointed recompute sees the same mask.
The drafter mirrors the target's MTP layer so it can be warm-started from it:
attn_output_gate adds the gated attention output (q_proj emits
[query | gate] per head, the output is scaled by its sigmoid before o_proj)
and partial_rotary_factor rotates 64 of the 256 head channels, with cos/sin
equal to the target's text-only M-RoPE. Both default off, so existing drafters
are unchanged. The rotary config folds the factor into head_dim and resets it
to 1, so a Transformers release that also applies partial_rotary_factor inside
the rotary embedding cannot shrink the slice twice.
AutoDraftModel.from_config builds drafters with num_experts > 0 directly in
the requested dtype: built in fp32 and cast afterwards, the 13B 512-expert
drafter peaks at twice its bf16 size. Dense drafters keep their fp32
construction, so their random initialization does not move.
configs/qwen3.8-flash-next-dspark-moe{64,512}.json (2.0B / 13.0B) differ only
in num_experts. The offline NPU recipe trains the 64-expert arm with DDP on
one 8-card node; the 512-expert arm needs expert parallelism, which this
repository does not have yet.
Tests pin the training block to the serving query block (anchor at slot 0,
no target state at or after the anchor read, under sdpa and eager), the
balance term against the Switch formula with and without padding (also
under a checkpointed recompute), the gated attention, and -- with transformers>=5.16.1 -- the MoE block and partial RoPE
against Qwen3.8-Flash-Next's own modules.
scripts/warm_start_from_mtp.py reads mtp.layers.0.* from a local Flash-Next checkpoint and writes a pretrained draft directory for model.draft_checkpoint_path. Every drafter layer takes the MTP layer's gated attention, router, routed experts (the stacked gate_up_proj split into gate_proj / up_proj) and shared expert; the q/k norms gain one, since the target's RMSNorm scales by 1 + w and the drafter's by w. The input projection, layer norms and Markov / confidence heads -- nothing in the MTP layer corresponds -- keep their fresh initialization, and the indexer is dropped. A drafter with fewer experts names the MTP experts it starts from (--expert-ids, or --expert-selection stride). The mapping hands out views into the stacked experts instead of copies; the 512-expert arm peaks at about 32 GiB of host memory. Tests: the copied MoE computes what Qwen4ExpTextSparseMoeBlock computes on the same tensors; unmapped tensors are untouched; the script's output warm-starts a drafter with nothing missing; and the released checkpoint's MTP shapes, read from its safetensors headers, fit both shipped drafter configs.
Qwen3.8 recipes use the qwen3.5 template entry, which renders a conversation once with enable_thinking=True and supervises each assistant span. That is right only if, for every assistant turn, the prompt a server builds is a prefix of the training text and the supervised span is exactly what the model generates. The test renders the released Jinja both ways for single-turn, multi-turn and tool-calling conversations, and runs ThinkingParser.parse() to check per turn that the serving-prompt tokens prefix the input ids and the loss mask covers exactly the generated tokens plus the newline after <|im_end|>. A byte-level tokenizer carrying Qwen3.8's added tokens runs this offline; with QWEN38_TOKENIZER_PATH set, the real tokenizer runs the same checks. It also pins the serving settings that break the match, which a corpus and an evaluation must avoid: a reasoning_effort other than xhigh, preserve_thinking=False, and enable_thinking=False. The template is vendored byte for byte (sha256 pinned; identical in Qwen3.8-27B and Qwen3.8-Flash-Next), so the whitespace hooks skip vendored templates.
`prepare_hidden_states.py --target-backend hf` captures DFlash / DSpark features with the Transformers reference model, for targets no local serving engine runs; currently Qwen3.8-Flash-Next (qwen4_exp). One process spreads consecutive decoder layers over the visible accelerators and keeps the ~95 GiB n-gram table in host memory, where its lookup already runs. Accelerate's device_map cannot express that -- it treats "cpu" as offload and would stream the table onto the execution device every forward. Layer placement moves each module's own tensors and stops at host-resident subtrees: Module.to() would recurse into the n-gram table and copy it onto one 64 GB card first. A table found off the host is refused; the regression tests dispatch onto the meta device, from which a transient migration cannot come back. A DSpark tap i reads layers[i + 1].attn_hyper_connection(streams)[0]: the normalized state layer i + 1 itself reads out of the 4-stream residual. Taps whose next layer carries PLE, or that have no next layer, are refused. Two reference paths are replaced for speed. The QSA indexer's per-query Python loop becomes one score matrix over fixed 4-token blocks (masks are bit-identical to the reference in fp32 and bf16; padded or cached inputs fall back), and the routed experts sort their assignments once instead of a host sync per expert. The capture test compares rows against a reference running the same fast paths with torch.equal; the fast paths' agreement with the stock forward is tested separately, since top-k ties at score 0 resolve differently between per-query and vectorized calls on aarch64. Rows run one at a time over their real tokens: right padding is dropped before the forward and the states are zero-padded back so the rows stack. The backend says so through unpadded_rows, and the preparation script then saves each record at its attention-mask length instead of the batch's padded one. It trims before the host copy, so a saved tensor owns only its kept rows; a view of the padded copy would serialize the whole padded storage. Needs transformers>=5.16.1 in the capture environment only; the tests that build the model skip without it.
Add --target-backend vllm to prepare_hidden_states.py. The backend sends each row's real tokens to a vLLM / vLLM-Ascend server running extract_hidden_states with ExampleHiddenStatesConnector (/v1/completions, prompt as token ids, max_tokens=1), waits on the connector's file lock, and splits the [T, K, H] capture into the taps concatenated in ascending order and the final state lm_head reads. The server's layer ids must be the taps plus one (vLLM numbers a layer's input, SpecForge its output) followed by num_hidden_layers; the backend prints that value and refuses a capture with the wrong layer count. Echoed and saved token ids, T, and finiteness are checked per row. Rows from one batch are requested concurrently and sent without their right padding (unpadded_rows), so --batch-size can match the server's concurrency. Rendering, loss masks, the record layout and resumption are the existing pipeline's. The client loads no model, so it should not need a Transformers release that knows the target: the preparation entry now reads the target config through load_target_config, which falls back to the raw config.json as training already does, and build_target_model no longer requires a dtype field on it. With the repository's pinned Transformers, Qwen3.8-Flash-Next (qwen4_exp) was otherwise refused before any request is sent. The connector writes each file asynchronously whether or not it keeps its synchronization lock, so the lock is the only sign a file is complete. A capture without its .lock file is refused with a pointer to use_synchronization_lock (the connector's default) instead of being read while it may still be written. Tests run the backend and HiddenStatesGenerator.generate against a fake connector server; the saved records pass normalize_dspark_offline_sample as 3-D hidden states. The entry reaches distributed setup with a target whose model type Transformers does not register.
The data preparation guide gains a section on the hf and vllm capture backends -- when to use them, the vLLM layer-id convention, batch size versus server concurrency, path mapping -- with a Flash-Next command, and on the MTP warm start. The example-config README introduces the Flash-Next recipe. It also states what the vllm backend needs from the server: the synchronization lock kept on, and -- for Qwen3.8-Flash-Next, which upstream vLLM / vLLM-Ascend cannot yet serve with extract_hidden_states -- the qwen38-capture branch of fg11991/vllm-ascend, with the configuration it has been run in.
3 of 4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Qwen3.8-Flash-Next (
qwen4_exp) is a 48-layer MoE target with a 4-stream gatedresidual, a PLE n-gram embedding and one MTP layer. Training a DSpark drafter
for it in SpecForge was blocked on two fronts:
the natural initialization, is MoE with gated attention and partial RoPE, so
a dense drafter can neither mirror it nor warm-start from it.
SGLang. SGLang supports
qwen4_expon GPU (support qwen 3.8 flash next sglang#37500), butthere is no hidden-state capture hook for it, and the Ascend path is still
an open PR ([Feat] [NPU] Qwen3.8-Flash-Next: Support graph replay, MTP, and Triton kernels sglang#37570). We train on Ascend.
This PR adds the drafter and two capture backends that do not depend on SGLang,
without changing any existing default.
What changes
feat: MoE DSpark drafter for Qwen3.8-Flash-NextQwenSparseMoE(Qwen3-Next routing + sigmoid-gated shared expert, Switch balance loss),attn_output_gateandpartial_rotary_factoron the DFlash-family attention (both off by default), padded blocks masked out of the balance term, MoE drafters built directly in the requested dtype, configsqwen3.8-flash-next-dspark-moe{64,512}.json, offline NPU recipefeat: warm-start ... from the target's MTP layerscripts/warm_start_from_mtp.pywrites a pretrained draft directory formodel.draft_checkpoint_pathtest: Qwen3.8 training render and loss mask ...qwen3.5template +ThinkingParserloss mask match what a server prompts, turn by turnfeat: Transformers offline capture backend ...prepare_hidden_states.py --target-backend hffeat: prepare DSpark features from a running vLLM ...prepare_hidden_states.py --target-backend vllmdocs: ...Not in this PR: expert parallelism. The 64-expert arm (2.0B) trains with DDP.
The 512-expert arm (13.0B) needs EP and is shipped only as a config, so warm
start and serving can already target it.
How it works
Drafter
shared expert scaled by a sigmoid gate. With the same weights it matches
Transformers'
Qwen4ExpTextSparseMoeBlockto 1e-6. Load balancing uses theSwitch auxiliary loss (
router_aux_loss_coef, 0.001) rather than aselection bias, so serving has nothing extra to reproduce.
moe_aux_loss,moe_max_load_ratioandmoe_unused_expert_fracare logged as means overthe MoE layers.
masked out of the balance term too. The mask reaches the MoE layers as a
forward argument, so a gradient-checkpointed recompute uses the same mask;
a test checks that router gradients match with and without checkpointing.
q_projemits[query | gate]per head, as in the target. Theattention output is multiplied by
sigmoid(gate)beforeo_proj. RoPE turns64 of the 256 channels, with cos/sin equal to the target's text-only M-RoPE.
bonus token sits in slot 0 at position
a, labels start atx_{a+1}, and notarget state at or after
ais visible. A test pins this under both sdpa andeager. When we let the anchor row through, the test fails, which is the leak
that once cost a DeepSeek-V4 DSpark fine-tune its serving acceptance.
Capture taps
DSpark tap
iislayers[i + 1].attn_hyper_connection(streams)[0]: thenormalized state layer
i + 1reads out of the 4-stream residual. The configsuse the full-attention layers
[3, 15, 23, 35, 43]. Every capture path,including a future SGLang one, must take exactly this state.
--target-backend hfvisible devices.
Module.to()would first copyit onto one 64 GB card, so layer placement stops at host-resident subtrees.
bit-identical to the reference.
expert.
(
torch.equal).length; each record is trimmed before the host copy, so it stores only its
own rows.
transformers>=5.16.1in the capture environment only.--target-backend vllmextract_hidden_statesandExampleHiddenStatesConnector.Flash-Next this way: vLLM's Qwen4Exp config check admits only the native MTP
and n-gram proposers, and the model emits no auxiliary hidden states. The
qwen38-capturebranch of
fg11991/vllm-ascendcarries the changes (8cc7129allowsextract_hidden_states,1de1b0bserves the auxiliary states, plusKV-cache sizing fixes for the capture path). It has run on one Atlas 800 A3
node: W8A8, TP8,
--enforce-eager, native MTP off,--max-model-len 2304.synchronization lock, so the lock is the only sign a file is complete. The
backend refuses a capture without its
.lockfile and asks foruse_synchronization_lock(the connector's default).rendering.
num_hidden_layers.vLLM numbers a layer's input, SpecForge a layer's output; for Flash-Next the
ids are
[4, 16, 24, 36, 44, 48].through
load_target_config, falling back to the rawconfig.jsonastraining does, so the repository's pinned Transformers is enough.
Testing
Unit tests (macOS arm64, CPU)
qwen4_exp-dependent ones skip. Full suite has the same failures asmainon this machine (sglang / triton / yunchang are not installable on macOS).qwen4_expThe real-tokenizer template test passed on the cluster with the released
Qwen3.8-Flash-Next tokenizer (
QWEN38_TOKENIZER_PATHset, 15 passed, noneskipped): each turn's serving prompt is a token prefix of the training ids and
the loss mask covers exactly the generated tokens.
End to end on Ascend
Run with the development branch these commits were ported from. Its drafter,
loss and capture code are the same as here; at expert-parallel size 1 the MoE
outputs, balance loss and all gradients were checked equal on CPU.
Capture (
--target-backend vllm): one Atlas 800 A3 node serving the W8A8checkpoint from the
qwen38-capturebranch (TP8,--enforce-eager, nativeMTP off,
--max-model-len 2304,--max-num-seqs 4); client concurrency and--batch-size4. 95 conversations were captured in 5 min 45 s, about 3.6 sper row.
Training: the 64-expert drafter (the recipe in this PR) from random
initialization on those 95 rows; 8 NPUs, DDP (
NO_SHARD), batch 1 ×accumulation 5 × 8 ranks = 40 samples per step,
max_length2304,transformers 5.16.1, 720 steps at 9.3–11 s per step. That is about 300 passes
over 95 rows, so this is a pipeline and learning check, not a quality result.
ce_lossaccaccuracy_position0 / 6tau_probabilisticteacher_agreementmoe_aux_loss(10 = balanced)moe_max_load_ratiomoe_unused_expert_fracline up, and has fallen to 1.3–1.5 by the end. Gradient norm spikes to 28
once in the early steps (step 6, in warmup, before the spike detector has a
baseline) and sits near 0.04 at the end, with no step skipped there.
token routes to the same ten experts: the load ratio sits at its ceiling
E / k = 6.4 and two thirds of the experts go unused by step 8. The balance
loss then spreads the load until nearly every expert is used.
teacher_agreementis how often the drafter's top-1 matches the target's,scored against the final state the vLLM capture returned; its climb to 0.85
is an indirect check that the captured state is the one
lm_headreads.512 experts and MTP warm start (development branch only: the 512-expert
drafter needs expert parallelism, which this PR does not add). Same 95 rows,
EP8 on 8 NPUs, batch 1 × accumulation 5 = 5 samples per step (EP ranks share
their tokens), the same 720-step schedule, 6.6–9 s per step after the first.
scripts/warm_start_from_mtp.pyon the released W8A8 checkpoint (its MTPlayer is BF16) copies 7,735 of the drafter's 7,752 tensors: per layer the
attention projections, q/k norms, router, 512 × 3 expert projections,
shared expert and its gate. The 17 left are the parts with no MTP
counterpart: the input projection and its norm, the layer and final norms,
and the Markov and confidence heads.
their steps compare directly:
ce_lossacctau_probabilisticteacher_agreementmoe_max_load_ratio(cap 51.2)moe_unused_expert_fracafter 720 steps (3,600 samples), and its router starts spread out instead
of collapsed onto ten experts.
balance (max load ratio 12.9 → 25.4 and gradient norm 0.13 → about 2 over
steps 1–11); whether it recovers by the end of the run is not yet known.
The 512-expert runs also see 8× fewer samples per step than the 64-expert
run, so the two do not compare step for step, and the scratch 512 run ends
with about 40% of experts idle in a step against 0.2% for 64 experts. Its
balance coefficient may need tuning once it trains on real data.
Not run at scale:
hfbackend: only the W8A8 checkpoint is available on the cluster, andthe Transformers reference model cannot load it. It is checked bit for bit
against Transformers on a small randomly initialized
qwen4_expmodel.follow-ups.
Follow-ups once SGLang runs Flash-Next with hidden-state capture
qwen4_exphook to the SGLang capture patch thatreads the tap above. That turns on the default
--target-backend sglangand online / disaggregated training for this target.
same tokens before trusting them. The HF backend stays as the oracle and as
the fallback for hardware SGLang does not cover.
qwen38-capturechanges upstream, sothe
vllmbackend no longer depends on a fork.Flash-Next. It needs gated attention, partial RoPE and the MoE FFN on the
DSpark draft path.
docs/recipes/with the exact revisions, commands and acceptance numbers.🤖 Generated with Claude Code