Skip to content

[atom vllm]Fix Qwen3.5 block-FP8 correctness under vLLM FULL cudagraph - #1985

Merged
valarLip merged 5 commits into
mainfrom
fix/qwen35-vllm-blockfp8-cudagraph
Aug 26, 2026
Merged

valarLip merged 5 commits into
mainfrom
fix/qwen35-vllm-blockfp8-cudagraph

Conversation

@zhangxinyuanliuhengyu

@zhangxinyuanliuhengyu zhangxinyuanliuhengyu commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes Qwen3.5-35B-A3B block-FP8 decode corruption on vLLM 0.27+ with ATOM plugin under FULL CUDA graph (enforce_eager=False), matching correctness of the known-good vllm-v0.19.0-nightly_20260526 image.

Root cause: latest ATOM routed Qwen3.5 MHA through the isolated AttentionForVllmMHA metadata path, which breaks under padded FULL-graph decode. Additional regressions were in GDN prefill kernel/dispatch, GDN metadata compaction, and MoE MORI trim.

Scope: plugin-only — no core changes. MHA/MoE FULL-graph patches are restricted to Qwen3.5-35B-A3B (config fingerprint); 397B and other Qwen3.5 variants keep the default plugin path. Standalone ATOM (VLLM_PLUGINS="", main zero patch) passes 4/4 on the same image, confirming the bug is in the vLLM plugin integration path, not ATOM core.

Changes (10 files, plugin-only)

File Purpose
atom/plugin/vllm/qwen35_attention_patch.py New — Qwen3.5-35B-A3B Attention.__new__ hook; routes MHA to legacy PagedAttention wrapper via bridge
atom/plugin/vllm/attention_bridge.py New (~62 lines) — slim unified_attention_with_output_base_for_plugin_mode bridge to vLLM native Attention
atom/plugin/vllm/qwen35_plugin_scope.py New — architecture + 35B-A3B fingerprint helpers (hidden=2048 / layers=40 / experts=256)
atom/plugin/vllm/register.py Register Qwen3.5 attention patch at plugin init
atom/plugin/config.py vllm_use_atom_attention flag for bridge RoPE vs native-attn path
atom/plugin/vllm/attention/layer_gdn.py Use verified vk-layout prefill + 0526-style GDN dispatch
atom/plugin/vllm/gdn_backend.py Remove FULL-cudagraph metadata compaction that corrupted ssm_state
atom/plugin/vllm/mori_patch.py 35B-A3B-only FULL-graph MoE trim (graph_bs*topk*dp); other models unchanged
tests/plugin/test_qwen35_plugin_scope.py New — unit tests that 35B-A3B matches and 397B is excluded
tests/plugin/test_vllm_kimi_k3.py Update GDN metadata tests (no compaction)

Not changed: atom/model_ops/base_attention.py, atom/model_ops/paged_attention.py, atom/models/qwen3_5.py — reverted to main.

Validation

Local probe harness (exp_atom_verify/probe_only.py, 4 greedy prompts) on rocm/atom-dev:vllm-latest (0822 image):

Config init infer Result
Before fix (plugin, FULL CG) — — p0/p1 corrupt after a few decode steps
After fix (plugin-only, FULL CG) ~653s ~105s 4/4 OK — no decode corruption
Standalone ATOM (main, zero patch) ~575s ~99s 4/4 OK — core unaffected
Settings enforce_eager=False, multimodal plugin, block-FP8

Customer perf vs 0520 baseline (compare_20260822_121300, GPU_MEM_UTIL=0.75):

Metric 0520 (customer) latest + fix Delta
probe infer 129.6s 105.7s −24s (~18% faster)
probe init 789s 652s −137s
4/4 probe OK OK —
customer 16/16 final OK ✓ ✓ —

ci:vllm (label run, SHA 5072fc8): passed after infra retries (aiter wheel curl timeouts on GLM/DeepSeek). Accuracy jobs green: GLM-5.2, DeepSeek-V4-Flash, MiniMax-M3, Kimi-K3, Qwen3.5-397B. Approval retriggered a new heavy-CI round.

Test plan

  • Qwen3.5-35B-A3B-FP8 probe 4/4 (FULL cudagraph, 0822 image + plugin-only patch)
  • Standalone ATOM main zero patch 4/4 (confirms core not needed)
  • Customer scenario perf vs vllm-v0.19.0-nightly_20260520
  • Scope 35B-A3B only (397B CI failed when the MHA bridge applied to all Qwen3.5 variants; fingerprint restrict + unit tests)
  • ci:vllm on 5072fc8 (label) — all accuracy jobs passed
  • Approval-triggered heavy CI (in progress)

Notes

  • Does not enable ATOM_QWEN35_TEXT_ONLY=1 or enforce_eager=True as workarounds.
  • GDN backend compaction removal applies to all GDN hybrid models on vLLM 0.27+ (bugfix aligned with upstream padding).
  • MHA bridge and MORI FULL-graph trim apply only to Qwen3.5-35B-A3B; 397B and other Qwen3.5 variants keep the existing plugin path.
  • Intentionally avoids restoring the full legacy plugin/attention.py module; only the bridge function needed by Qwen3.5 PagedAttention is added.

Restore the 0526-compatible MHA path for Qwen3.5 only, fix GDN prefill/dispatch
and metadata compaction on vLLM 0.27+, and scope MoE MORI trim so other models
keep the existing behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1985 --add-label <label>

zhangxinyuanliuhengyu and others added 3 commits August 22, 2026 15:20
Rebase onto main (online quant #1749, DSV4 fixes, etc.), resolve qwen3_5,
config, and gdn_backend conflicts keeping the cudagraph correctness fixes.
Replace the 2546-line plugin/attention.py restore with a ~60-line
plugin/vllm/attention_bridge.py used only by Qwen3.5 PagedAttention.

Co-authored-by: Cursor <cursoragent@cursor.com>
Format mori_patch for Black, fix ruff import/order/SIM102/RUF012/BLE001
on PR-touched files, and align Kimi K3 tests with removal of GDN FULL-graph
metadata compaction.

Co-authored-by: Cursor <cursoragent@cursor.com>
Move legacy PagedAttention routing into qwen35_attention_patch via
register hook and revert core model_ops/qwen3_5 changes. Standalone
ATOM on main needs no core patch for this probe.

Co-authored-by: Cursor <cursoragent@cursor.com>
397B CI regressed when the FULL-cudagraph MHA bridge applied to all Qwen3.5 variants; restrict it via config fingerprint while keeping the 35B customer fix.

Co-authored-by: Cursor <cursoragent@cursor.com>
@zhangxinyuanliuhengyu zhangxinyuanliuhengyu changed the title Fix Qwen3.5 block-FP8 correctness under vLLM FULL cudagraph [atom vllm]Fix Qwen3.5 block-FP8 correctness under vLLM FULL cudagraph Aug 26, 2026
@valarLip
valarLip merged commit 955d3e1 into main Aug 26, 2026
98 of 118 checks passed
@valarLip
valarLip deleted the fix/qwen35-vllm-blockfp8-cudagraph branch August 26, 2026 07:25
PerryZhang01 pushed a commit that referenced this pull request Aug 31, 2026
vLLM pads a FULL-cudagraph decode batch out to a captured graph size and
leaves the padded request rows in the GDN metadata. ATOM's GDN decode
kernel is request-indexed, so it walks those rows and folds them into
ssm_state. #1985 deleted the pass that compacted them away, to fix
Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to
35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an
identical copy in its own KDA builder
(AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata);
Qwen3.5-397B was left with nothing.

That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the
0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under
its 0.83 gate, on every branch that has run it since.

Put the pass back and give it the same 35B-A3B exemption the rest of
#1985 uses, so that fix is untouched. Measured on 4x gfx950 with the
runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter
nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the
harness's FULL_AND_PIECEWISE:

  as CI runs it today                     0.7817
  this change                             0.8544
  whole #1985 GDN revert (reference)      0.8431
  cudagraph_mode=PIECEWISE (reference)    0.8355

The 35B-A3B exemption is covered by unit tests but not by an end-to-end
35B block-FP8 run; that path keeps today's behavior either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Aug 31, 2026
The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how
#1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata
pass has no business knowing which model it is running under, and the
reason #1985 gave for removing the pass does not survive checking.

Why the pass is unconditionally needed. vLLM means to mark the padded
rows itself:

    non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID)

but num_decodes comes from split_decodes_and_prefills, whose fast path

    if max_query_len <= decode_threshold and ...:
        return num_reqs, 0, num_tokens, 0

fires for any pure-decode batch (max_query_len == 1) and returns
num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph
replay. The fill therefore starts at num_decodes == batch_size and is a
no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and
ATOM's request-indexed GDN decode kernel folds them into ssm_state. That
depends only on the decode batch being padded -- not on the model, its
quantization, or TP. It is the same reason Kimi-K3 carries the pass in
its own KDA builder.

Why #1985's reason does not hold. It removed the pass because it
"rewrote that metadata using num_actual_tokens (token-padded graph size)
and shrunk num_decodes". That was true of the original version
(2255001) and was fixed two versions earlier, in 441476e, which
switched to num_reqs with a comment saying exactly why.

Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image
assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544;
the two are within one stderr of each other, and both clear the 0.83
gate against 0.7817 before the fix).

The 35B-A3B block-FP8 path is now covered by the pass again. I could not
score it end to end -- this box has the checkpoint but no validated vLLM
plugin recipe for it, and my attempt floored at 0.05-0.07 under both
FULL and PIECEWISE, so it measured nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Sep 3, 2026
vLLM pads a FULL-cudagraph decode batch out to a captured graph size and
leaves the padded request rows in the GDN metadata. ATOM's GDN decode
kernel is request-indexed, so it walks those rows and folds them into
ssm_state. #1985 deleted the pass that compacted them away, to fix
Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to
35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an
identical copy in its own KDA builder
(AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata);
Qwen3.5-397B was left with nothing.

That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the
0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under
its 0.83 gate, on every branch that has run it since.

Put the pass back and give it the same 35B-A3B exemption the rest of
#1985 uses, so that fix is untouched. Measured on 4x gfx950 with the
runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter
nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the
harness's FULL_AND_PIECEWISE:

  as CI runs it today                     0.7817
  this change                             0.8544
  whole #1985 GDN revert (reference)      0.8431
  cudagraph_mode=PIECEWISE (reference)    0.8355

The 35B-A3B exemption is covered by unit tests but not by an end-to-end
35B block-FP8 run; that path keeps today's behavior either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01 pushed a commit that referenced this pull request Sep 3, 2026
The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how
#1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata
pass has no business knowing which model it is running under, and the
reason #1985 gave for removing the pass does not survive checking.

Why the pass is unconditionally needed. vLLM means to mark the padded
rows itself:

    non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID)

but num_decodes comes from split_decodes_and_prefills, whose fast path

    if max_query_len <= decode_threshold and ...:
        return num_reqs, 0, num_tokens, 0

fires for any pure-decode batch (max_query_len == 1) and returns
num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph
replay. The fill therefore starts at num_decodes == batch_size and is a
no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and
ATOM's request-indexed GDN decode kernel folds them into ssm_state. That
depends only on the decode batch being padded -- not on the model, its
quantization, or TP. It is the same reason Kimi-K3 carries the pass in
its own KDA builder.

Why #1985's reason does not hold. It removed the pass because it
"rewrote that metadata using num_actual_tokens (token-padded graph size)
and shrunk num_decodes". That was true of the original version
(2255001) and was fixed two versions earlier, in 441476e, which
switched to num_reqs with a comment saying exactly why.

Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image
assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544;
the two are within one stderr of each other, and both clear the 0.83
gate against 0.7817 before the fix).

The 35B-A3B block-FP8 path is now covered by the pass again. I could not
score it end to end -- this box has the checkpoint but no validated vLLM
plugin recipe for it, and my attempt floored at 0.05-0.07 under both
FULL and PIECEWISE, so it measured nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyohuangxin pushed a commit that referenced this pull request Sep 3, 2026
* Change atom-plugin CI runner to spur-runner-mi355x-8gpu

* Move plugin CI logs onto shared_nfs so Spur runners can write them.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Write plugin CI logs under the GitHub runner workspace.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Run vLLM and SGLang plugin CI directly on the GPU runner.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(v4): keep the profile forward off the decode running_tokens branch

Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All
four workers abort inside `determine_available_memory` -> `profile_run`:

    AttributeError: 'types.SimpleNamespace' object has no attribute
    'max_seqlen_q'
      atom/plugin/vllm/deepseek_v4_bridge.py:2164
      atom/utils/forward_context.py:296  running_tokens_from_bs

`force_dummy` builds `attn_metadata` as a bare namespace carrying only
`state` and `in_hipgraph` (#1839). That is deliberate: the real builder
allocates and copies tensors, which HIP forbids under stream capture.
The namespace has therefore never had a sequence length.

#2042 replaced `graph_bs=batch_size` at this call site with
`running_tokens_from_bs(...)`, which dereferences
`attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also
forces `is_prefill=False`, so the profile forward takes that branch
unconditionally -- it cannot not crash.

Pass the `None` the helper already documents as "count only", which
restores exactly the `batch_size` the replaced expression produced. The
fix goes at the call site, where `force_dummy` is known, rather than
loosening the helper: a real decode metadata that is missing
`max_seqlen_q` should still fail loudly instead of silently under-padding
the MoE all_gather.

Not a new-model or spec-decode issue -- it reproduces on plain
`DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's
`DeepSeek-V4-Pro TP8`, which shares this bridge.

Verified on 4x MI355X via the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry
"DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no
score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a
0.93 threshold, in 9 minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ci): give Kimi-K3 DCP8 enough budget to allocate a KV cache

`Accuracy (Kimi-K3 DSpark DCP8 TP8)` cannot start. The engine aborts
while sizing the KV cache:

    INFO gpu_worker.py:563] Available KV cache memory: -5.07 GiB
    ValueError: No available memory for the cache blocks.
                Try increasing `gpu_memory_utilization`

The job is red with no score at all, which reads like an accuracy or
spec-decode regression and is neither.

#2033 moved this entry to DCP8 and set `--gpu-memory-utilization` from
0.93 to 0.7. Kimi-K3 is ~187 GiB of weights per GPU at TP8; on a 288 GiB
MI355X a 0.7 budget is 201.6 GiB, which does not leave a KV cache behind
after weights and activations. The threshold to merely reach zero is
0.7 + 5.07/288 ~= 0.718, so 0.7 is not marginally low -- it is on the
wrong side of the window.

The window is narrow in both directions, which is why 0.7 looks like a
safe choice and is not: FULL cudagraph capture takes its scratch OUTSIDE
the vLLM budget, so a HIGHER utilization starves capture (0.85 dies with
`OutOfMemoryError` in `mla_decode_fwd`; 0.93 leaves so little headroom
that one rank stalls in allocation while the others leave the capture
context and its MoE all_reduce deadlocks). 0.75 is the value that
satisfies both ends.

Measured on 8x MI355X through the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, this matrix entry), with
every GPU verified idle (<1 GiB) before each launch:

    gpu-mem   KV cache      FULL capture   outcome
    0.70      -5.07 GiB     never reached  engine abort (reproduced twice,
                                           bit-identical)
    0.75      +9.33 GiB     48/48 ok       serves; gsm8k 5-shot 0.9454,
                                           DSpark acceptance 0.514

The +14.40 GiB delta is exactly 0.05 x 287.98 GiB, confirming the budget
is the only variable in play.

Acceptance 0.514 matches the ~52% recorded for DCP8 + DSpark N=7, so the
draft head is healthy. Note the run still lands under the 0.95 accuracy
threshold; that is a separate question from being able to start at all,
and is being characterized separately -- this commit only restores a
server that can allocate a KV cache.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): restore the GDN FULL-cudagraph decode metadata compaction

vLLM pads a FULL-cudagraph decode batch out to a captured graph size and
leaves the padded request rows in the GDN metadata. ATOM's GDN decode
kernel is request-indexed, so it walks those rows and folds them into
ssm_state. #1985 deleted the pass that compacted them away, to fix
Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to
35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an
identical copy in its own KDA builder
(AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata);
Qwen3.5-397B was left with nothing.

That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the
0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under
its 0.83 gate, on every branch that has run it since.

Put the pass back and give it the same 35B-A3B exemption the rest of
#1985 uses, so that fix is untouched. Measured on 4x gfx950 with the
runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter
nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the
harness's FULL_AND_PIECEWISE:

  as CI runs it today                     0.7817
  this change                             0.8544
  whole #1985 GDN revert (reference)      0.8431
  cudagraph_mode=PIECEWISE (reference)    0.8355

The 35B-A3B exemption is covered by unit tests but not by an end-to-end
35B block-FP8 run; that path keeps today's behavior either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): drop the model scope from the GDN metadata compaction

The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how
#1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata
pass has no business knowing which model it is running under, and the
reason #1985 gave for removing the pass does not survive checking.

Why the pass is unconditionally needed. vLLM means to mark the padded
rows itself:

    non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID)

but num_decodes comes from split_decodes_and_prefills, whose fast path

    if max_query_len <= decode_threshold and ...:
        return num_reqs, 0, num_tokens, 0

fires for any pure-decode batch (max_query_len == 1) and returns
num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph
replay. The fill therefore starts at num_decodes == batch_size and is a
no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and
ATOM's request-indexed GDN decode kernel folds them into ssm_state. That
depends only on the decode batch being padded -- not on the model, its
quantization, or TP. It is the same reason Kimi-K3 carries the pass in
its own KDA builder.

Why #1985's reason does not hold. It removed the pass because it
"rewrote that metadata using num_actual_tokens (token-padded graph size)
and shrunk num_decodes". That was true of the original version
(2255001) and was fixed two versions earlier, in 441476e, which
switched to num_reqs with a comment saying exactly why.

Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image
assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544;
the two are within one stderr of each other, and both clear the 0.83
gate against 0.7817 before the fix).

The 35B-A3B block-FP8 path is now covered by the pass again. I could not
score it end to end -- this box has the checkpoint but no validated vLLM
plugin recipe for it, and my attempt floored at 0.05-0.07 under both
FULL and PIECEWISE, so it measured nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): keep the MTP position-0 mask off the layer signature

Every vLLM-plugin MTP run has been dying in profile_run since the fused
FP8 prologue landed:

    TypeError: DeepSeekMultiTokenPredictorLayer.forward() takes from 5 to
    6 positional arguments but 7 were given

The plugin installed its position-0 input-embedding mask by rebinding
each draft layer's forward to a wrapper that re-called the original with
a hardcoded five-argument list. When the fused prologue gave the layer a
sixth argument (eh_input_quant), the wrapper kept advertising the old
arity -- and functools.wraps kept the original qualname, so the error
pointed at a model signature that was in fact correct.

The wrapper could not have worked on the fused path anyway: there the
embedding never materializes as a tensor the layer receives, so there is
nothing for a layer-level wrapper to mask.

Move the mask into the predictors, which own the embedding lookup on
both paths and cannot drift from their own layers. On the unfused path
it is the same torch.where as before. On the fused path the masked rows
are fed the sentinel id -1: the prologue kernel already emits a zero
embedding for any id outside [0, vocab_size), and RMSNorm(0) is 0, so
the result is bit-identical to zeroing inputs_embeds -- without
materializing the [num_tokens, hidden] tensor the fusion exists to
avoid.

The plugin now flips a flag instead of rewriting a signature, and raises
if an arch in _MTP_MASK_INPUT_ARCH has no predictor to flip it on,
rather than silently serving an unmasked draft.

The regression test lives in tests/ rather than tests/plugin/ because
run_unit_tests.sh skips the latter.

Fixes the GLM-5.2-MXFP4-MTP TP4 plugin accuracy job.

* ci: download plugin CI draft models like the target

Kimi-K3 DSpark failed because speculative-config pointed at a missing local draft path. Cache the draft the same way as the target and rewrite extra_args to the resolved directory.

Co-authored-by: Cursor <cursoragent@cursor.com>

* ci: SIGKILL leftover host/KFD VRAM holders after plugin docker cleanup

Plugin HIP smoke was failing on spur runners when VRAM stayed allocated by host PIDs after leftover containers were already gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: PerryZhang01 <Perry.Zhang@amd.com>
Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-v2-009.us-east2-a.compute.internal>
Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-012.us-east2-a.compute.internal>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants