Skip to content

support qwen 3.8 flash next - #37500

Merged
Qiaolin-Yu merged 47 commits into
mainfrom
qwen4-main-squashed-rebased
Sep 8, 2026
Merged

Qiaolin-Yu merged 47 commits into
mainfrom
qwen4-main-squashed-rebased

Conversation

@Qiaolin-Yu

@Qiaolin-Yu Qiaolin-Yu commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

initial pr: #36497


CI States

Latest PR Test (Base): ✅ Run #34195586068
Latest PR Test (Extra): ✅ Run #34273666129
Latest PR Test (AMD ROCm 7.2): ❌ Run #34195586146

Qiaolin-Yu and others added 7 commits September 2, 2026 00:11
Co-authored-by: Qiaolin-Yu <90088090+Qiaolin-Yu@users.noreply.github.com>
Co-authored-by: ch-wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ispobock <26454835+ispobock@users.noreply.github.com>
Co-authored-by: JustinTong0323 <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: samuellees <26428561+samuellees@users.noreply.github.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: yizhang2077 <25844240+yizhang2077@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Shinto C V <cshintov@gmail.com>
@detain detain mentioned this pull request Sep 14, 2026
5 tasks done
@Xia-Weiwen Xia-Weiwen mentioned this pull request Sep 17, 2026
3 of 5 tasks
adibarra pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Sep 21, 2026
…ge to v0.5.20-cu130

Move the family from the model-branch tag lmsysorg/sglang:qwen38flashnext
(commit 593134d1, 2026-09-03) to the official lmsysorg/sglang:v0.5.20-cu130
release (commit 94602c9c), which ships upstream Qwen3.8-Flash-Next support
(sgl-project/sglang#37500). v0.5.20 removed the deprecated --cuda-graph-max-bs
alias (sgl-project/sglang#38375), so the unshared recipe now passes the same
value through --cuda-graph-max-bs-decode.

将该配置从模型分支标签 lmsysorg/sglang:qwen38flashnext(提交 593134d1,2026-09-03)
迁移到官方发布镜像 lmsysorg/sglang:v0.5.20-cu130(提交 94602c9c),该版本已包含
上游 Qwen3.8-Flash-Next 支持(sgl-project/sglang#37500)。v0.5.20 移除了已弃用的
--cuda-graph-max-bs 别名(sgl-project/sglang#38375),因此该配方独有的脚本改用
--cuda-graph-max-bs-decode 传递相同的值。

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 24, 2026
…rides: fused split for head ratio 3

Upstream 8a1e6e4 (v0.5.20), the GDN part only, plus the row-stride hunk
of upstream 52fecfd (sgl-project#37500) that the fork's port of sgl-project#37500 (e46a4db)
left out.

The 27B's GDN (16 k / 48 v heads, ratio 3) was outside the fused split list
[1, 2, 4]: every decode / DFLASH verify step ran split + 2x .contiguous() +
torch.cat + a z reshape copy on every GDN layer. The contiguous fused split
kernel now walks a non-power-of-two head group one HEAD_V head at a time
(V_POW2 constexpr prunes the dead branch; power-of-two groups keep the wide
vector access), and qwen3_5 enables ratio 3 on CUDA
(_GDN_FUSED_QKVZBA_RATIOS = (1, 2, 3, 4) on CUDA, (1, 2, 4) elsewhere).
Prefill keeps the sgl-project#36267 strided views; verify then hands views on (sgl-project#33778).

Row strides (sgl-project#37500): the kernel read rows at the dense widths. With
finalize_fused_in_proj (called for the NF model, qwen4_exp; its GDN ratio is
also 3) the split gets two column views of ONE GEMM output (row pitch = qkvz
+ ba width), so routing ratio 3 here without the strides would have read the
wrong rows for every token after the first on NF. The 27B does not call
finalize_fused_in_proj (dense rows); the wrapper now also asserts unit
column strides.

Not ported from sgl-project#34859 (not applicable to INT8 on sm86/sm120): sm120 FP8
GEMV, Hopper bf16 GEMV (SM90 only), SM90 FlashInfer GDN prefill default,
NVFP4 SiLU+quant fusion, DSpark / qwen2_moe / deepseek / modelopt hunks.

Tests (CPU, incg cgroup):
test/registered/unit/layers/attention/test_gdn_fused_split_head_ratios_34859.py
- production kernel under TRITON_INTERPRET vs plain slicing, ratios 1/2/3/4,
  uneven 27B rank-local heads 5/15 and 6/18, full 16/48, dense and
  fused-GEMM-strided rows: 14/14 bit-exact (worker run under incg; the
  shared 3 GiB cgroup OOM-killed the pytest wrapper of this class twice
  while other agents' suites ran). Base kernel: strided rows wrong for
  ratios 1/2/4, ratio 3 does not compile (arange not a power of two).
- qwen3_5 dispatch: decode and target verify take the fused split for
  ratio 3, extend takes the views: 3 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 24, 2026
…cation, cursors, resets

27B ReplaySSM package, slice S2 of REPLAYSSM_PLAN.md. Flag default off; with
it off every pool is byte-identical (pinned).

- server_args: enable_linear_replayssm_spec (+ help) and its static checks
  next to the decode ring's in _handle_linear_attn_backend: exclusive with
  --enable-linear-replayssm, needs a speculative algorithm, linear draft chain
  (topk None/1), Triton decode backend, no PD disaggregation, ring length a
  power of two; a 16-bit mamba-ssm-dtype logs the metal-gate warning.
- MambaPool: under the flag the spec ring replaces intermediate_ssm (not
  allocated): d/rawv [layers, rows, HV, L, V], k/rawk [layers, rows, H, L, K]
  in the activation dtype (low parts only for 16-bit), g fp32; rows =
  spec_state_size + 1 like the state it replaces; HV/H from the rank's own
  temporal shape (uneven GDN TP needs no special case). Request-row cursors
  replayssm_spec_write_pos / cache_base / is_flush. Runtime refusals where
  the resolved values live: ring shorter than the widest verify window
  (adaptive ladder included), KDA (only the GDN route is ported), decode ring.
  reset_state zeroes rings and cursors and tolerates the absent intermediate;
  the poison list covers the new rings; get_contiguous_buf_infos never ships
  them. State gains replayssm_rawv / replayssm_rawk.
- HybridReqToTokenPool forwards the flag (the parameter existed from the
  sgl-project#37500 port but was dropped) and gives a fresh request row an empty ring.
- mixin: _replayssm_spec_for refuses the flag for non-GDN models (ring layout
  is the GDN temporal), and passes it to the pool.
- The intermediate readers not yet wired to the ring refuse loudly instead of
  scattering a None: DFLASH commit (update_mamba_state_after_mtp_verify), GDN
  target verify, the MTP scatter helper, the dual-group lane verify. S3
  replaces the GDN ones with the ring route.

Budget: the mixin still reserves the intermediate bytes (S5 moves them to KV),
so this slice cannot over-commit.

Tests (CPU, incg cgroup):
test/registered/unit/mem_cache/test_replayssm_spec_ring_pool_s2.py: flag-off
pool unchanged; ring shapes/dtypes/price for the 27B uneven ranks (HV 18/6,
15/5); fp32 has no low parts; runtime and static refusals; reset_state;
transfer list; fresh-row reset through the real alloc; refusing reader.
10 passed (7 subtests); on the base sources 14 of 17 fail.
test_mamba_checkpoint_interval.py (pool flush/clear incl. decode ring): 58
passed. CLI: --enable-linear-replayssm-spec parses.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 24, 2026
…cation, cursors, resets

27B ReplaySSM package, slice S2 of REPLAYSSM_PLAN.md. Flag default off; with
it off every pool is byte-identical (pinned).

- server_args: enable_linear_replayssm_spec (+ help) and its static checks
  next to the decode ring's in _handle_linear_attn_backend: exclusive with
  --enable-linear-replayssm, needs a speculative algorithm, linear draft chain
  (topk None/1), Triton decode backend, no PD disaggregation, ring length a
  power of two; a 16-bit mamba-ssm-dtype logs the metal-gate warning.
- MambaPool: under the flag the spec ring replaces intermediate_ssm (not
  allocated): d/rawv [layers, rows, HV, L, V], k/rawk [layers, rows, H, L, K]
  in the activation dtype (low parts only for 16-bit), g fp32; rows =
  spec_state_size + 1 like the state it replaces; HV/H from the rank's own
  temporal shape (uneven GDN TP needs no special case). Request-row cursors
  replayssm_spec_write_pos / cache_base / is_flush. Runtime refusals where
  the resolved values live: ring shorter than the widest verify window
  (adaptive ladder included), KDA (only the GDN route is ported), decode ring.
  reset_state zeroes rings and cursors and tolerates the absent intermediate;
  the poison list covers the new rings; get_contiguous_buf_infos never ships
  them. State gains replayssm_rawv / replayssm_rawk.
- HybridReqToTokenPool forwards the flag (the parameter existed from the
  sgl-project#37500 port but was dropped) and gives a fresh request row an empty ring.
- mixin: _replayssm_spec_for refuses the flag for non-GDN models (ring layout
  is the GDN temporal), and passes it to the pool.
- The intermediate readers not yet wired to the ring refuse loudly instead of
  scattering a None: DFLASH commit (update_mamba_state_after_mtp_verify), GDN
  target verify, the MTP scatter helper, the dual-group lane verify. S3
  replaces the GDN ones with the ring route.

Budget: the mixin still reserves the intermediate bytes (S5 moves them to KV),
so this slice cannot over-commit.

Tests (CPU, incg cgroup):
test/registered/unit/mem_cache/test_replayssm_spec_ring_pool_s2.py: flag-off
pool unchanged; ring shapes/dtypes/price for the 27B uneven ranks (HV 18/6,
15/5); fp32 has no low parts; runtime and static refusals; reset_state;
transfer list; fresh-row reset through the real alloc; refusing reader.
10 passed (7 subtests); on the base sources 14 of 17 fail.
test_mamba_checkpoint_interval.py (pool flush/clear incl. decode ring): 58
passed. CLI: --enable-linear-replayssm-spec parses.

27B line (desk/27b-up-replayssm-line-0924): cherry-picked from ee6bee6
(desk/27b-up-replayssm-0924, on the NF-based staging line fa757eb) onto
this line, clean. The touched hunks are identical on both lines; what differs
in memory_pool.py / hybrid_linear_attn_backend.py (the NF-only H21 tail-adopt
install, the H20 fwd_mark timing) lies outside them and is not taken over.
HybridReqToTokenPool already carried enable_linear_replayssm_spec here too
(sgl-project#37500 port c529777) and dropped it at the MambaPool call, as on NF.

Tests on this line (CPU, incg cgroup, S2 tree, one file at a time):
test_replayssm_spec_ring_pool_s2.py 10 passed (7 subtests);
test_mamba_checkpoint_interval.py 58 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 25, 2026
…cation, cursors, resets

27B ReplaySSM package, slice S2 of REPLAYSSM_PLAN.md. Flag default off; with
it off every pool is byte-identical (pinned).

- server_args: enable_linear_replayssm_spec (+ help) and its static checks
  next to the decode ring's in _handle_linear_attn_backend: exclusive with
  --enable-linear-replayssm, needs a speculative algorithm, linear draft chain
  (topk None/1), Triton decode backend, no PD disaggregation, ring length a
  power of two; a 16-bit mamba-ssm-dtype logs the metal-gate warning.
- MambaPool: under the flag the spec ring replaces intermediate_ssm (not
  allocated): d/rawv [layers, rows, HV, L, V], k/rawk [layers, rows, H, L, K]
  in the activation dtype (low parts only for 16-bit), g fp32; rows =
  spec_state_size + 1 like the state it replaces; HV/H from the rank's own
  temporal shape (uneven GDN TP needs no special case). Request-row cursors
  replayssm_spec_write_pos / cache_base / is_flush. Runtime refusals where
  the resolved values live: ring shorter than the widest verify window
  (adaptive ladder included), KDA (only the GDN route is ported), decode ring.
  reset_state zeroes rings and cursors and tolerates the absent intermediate;
  the poison list covers the new rings; get_contiguous_buf_infos never ships
  them. State gains replayssm_rawv / replayssm_rawk.
- HybridReqToTokenPool forwards the flag (the parameter existed from the
  sgl-project#37500 port but was dropped) and gives a fresh request row an empty ring.
- mixin: _replayssm_spec_for refuses the flag for non-GDN models (ring layout
  is the GDN temporal), and passes it to the pool.
- The intermediate readers not yet wired to the ring refuse loudly instead of
  scattering a None: DFLASH commit (update_mamba_state_after_mtp_verify), GDN
  target verify, the MTP scatter helper, the dual-group lane verify. S3
  replaces the GDN ones with the ring route.

Budget: the mixin still reserves the intermediate bytes (S5 moves them to KV),
so this slice cannot over-commit.

Tests (CPU, incg cgroup):
test/registered/unit/mem_cache/test_replayssm_spec_ring_pool_s2.py: flag-off
pool unchanged; ring shapes/dtypes/price for the 27B uneven ranks (HV 18/6,
15/5); fp32 has no low parts; runtime and static refusals; reset_state;
transfer list; fresh-row reset through the real alloc; refusing reader.
10 passed (7 subtests); on the base sources 14 of 17 fail.
test_mamba_checkpoint_interval.py (pool flush/clear incl. decode ring): 58
passed. CLI: --enable-linear-replayssm-spec parses.

27B line (desk/27b-up-replayssm-line-0924): cherry-picked from ee6bee6
(desk/27b-up-replayssm-0924, on the NF-based staging line fa757eb) onto
this line, clean. The touched hunks are identical on both lines; what differs
in memory_pool.py / hybrid_linear_attn_backend.py (the NF-only H21 tail-adopt
install, the H20 fwd_mark timing) lies outside them and is not taken over.
HybridReqToTokenPool already carried enable_linear_replayssm_spec here too
(sgl-project#37500 port c529777) and dropped it at the MambaPool call, as on NF.

Tests on this line (CPU, incg cgroup, S2 tree, one file at a time):
test_replayssm_spec_ring_pool_s2.py 10 passed (7 subtests);
test_mamba_checkpoint_interval.py 58 passed.

(cherry picked from commit 64ca3dc)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 25, 2026
… Storage, und sein Block liegt im Band seiner Schicht statt im Basis-Tag (H79)

fnNV4f1 (77a8956) starb im ersten P->D-Release an W29 mit W84: 44/16/12
Befunde, alle "SHORT in_proj_qkvz.weight planned_mib=80.000 live_mib=80.469"
bzw. "in_proj_ba.weight planned_mib=0.469 live_mib=80.469". Das NVIDIA-NVFP4-
Checkpoint laesst in_proj_qkvz in BF16 (sgl-project#1483: qkvz=UnquantizedLinearMethod).
Deshalb stapelt finalize_fused_in_proj (Upstream sgl-project#37500) beide Gewichte in EINEN
Block und bindet sie als dessen Zeilen-Views. Plan und Austausch adressieren
jeden Parameter per data_ptr() und mit der Geometrie seines Views
(derive_leg_plan ptr_of, _weg2_join_src/dst_addr, StorageGeom.of). Ihre
Anspruchswerte stimmten also. Nur die Coverage rechnete jedem View den ganzen
Storage an (_nbytes = untyped_storage().nbytes()). Dazu lag der Block im
BASIS-Tag weights: finalize laeuft am Ende von load_weights ausserhalb jedes
weight_chunk_scope. Gemessen am Basis-Pool auf PP0/1/2: 2,91/0,63/1,67 GiB
aktiv gegen 0,60/0,00/0,61 bei x176 ohne Fusion. Plan und Walk fuehren
in_proj_* dagegen per Name im Band der Schicht.

- qwen3_5.finalize_fused_in_proj: der torch.cat laeuft unter
  weight_chunk_scope(layer_id), also im Band-Tag und im Band-Pool. Ausserhalb
  einer Weg2-Chunk-Region ist das ein No-op, dann bleibt die Upstream-Funktion
  unveraendert. Eine ungefusete Schicht (INT4-/FP8-qkvz) kehrt vor jeder
  Allokation und jedem Scope zurueck. Die Funktion gibt jetzt ein Statuswort
  zurueck; qwen4_exp druckt daraus je Rang
  "#H79 GDN-FUSED-IN-PROJ gdn= fused= unbanded= census=".
- weight_exchange.build_coverage: ein Storage, den mindestens zwei
  Plan-Parameter mit VERSCHIEDENEN Byte-Bereichen teilen, wird ueber seine Views
  beurteilt. Jedes Mitglied muss seinen View beanspruchen (sonst SHORT, ein
  Anspruch ueber den View hinaus ist W68), und die Views muessen den Storage
  zusammenhaengend und ueberlappungsfrei von 0 bis zum Ende kacheln. Sonst
  verweigert der neue Befund TILING (GAP/OVERLAP/OVERRUN/NONCONTIGUOUS), und ok
  ist falsch. Nur ein gekachelter Storage gilt als gedeckt. Dann ist
  _fused_in_proj_weight ein view_holder und keine eigene Quelle; bei Luecke
  bleibt es UNCOVERED. Ein Mitglied, Aliase ueber denselben Bereich und die
  Experten-Puffer bleiben auf dem alten Pfad.
- Metallmarker: WEG2-XCHG-TILED je Storage (verdict=tiled bzw. refused), am
  Ende der COVER-Zeile "tiled= view_holders= tiling=" und in der
  PLAN-PARAM-Zeile "view_mib= view_offset_mib=". Alle drei erscheinen nur, wenn
  es etwas zu zaehlen gibt.

INT4-Linie byte-identisch: die COVER- und PLAN-PARAM-Zeilen eines ungefuseten
Modells sind auf 77a8956 und mit H79 Zeichen fuer Zeichen gleich (cmp), und
finalize beruehrt keinen Parameter.

Test test_weg2_fused_in_proj_tiling_h79.py (CPU, Metallformen 16384/96 x 2560,
echte Fusion und echter derive_leg_plan): 14 gruen. Auf 77a8956 sind 10 rot:
die Metallform zeigt dort genau die beiden SHORT-Befunde (80.0/80.469 und
0.469/80.469), der cat laeuft unter weights statt weights_1, und die
Strenge-Tests brauchen die neuen Felder. Gruen auf beiden Baeumen: die zwei
INT4-Tests, die Views und der Plan ohne ba.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…l-project#37500-Gate bleibt ohne Bilder), PLE liest image_token_id an Bild-Pad-Positionen (Forward und Prefetch-Spiegel); --weg2-vision off unveraendert
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail documentation Improvements or additions to documentation high priority jit-kernel memory-pool quant LLM Quantization release-highlight Candidate PR for release note highlight run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) speculative-decoding unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants