Skip to content

vllm-atom: serve GLM-5.3-Flash on the plugin backend and gate it in plugin PR CI #3033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
PerryZhang01 wants to merge 8 commits into from
Closed

vllm-atom: serve GLM-5.3-Flash on the plugin backend and gate it in plugin PR CI #3033

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b9d5297
vllm-atom: serve GLM-5.3-Flash on the plugin backend
zgplvyou Nov 5, 2026
60b588f
ci(vllm-plugin): add a GLM-5.3-Flash kpool cell to the PR accuracy ma…
zgplvyou Nov 5, 2026
71f82f8
ci(vllm-plugin): drop the GLM-5.2-MXFP4-MTP cell from the PR accuracy…
zgplvyou Nov 5, 2026
217d896
vllm-atom: keep the GLM-5.3-Flash plugin changes out of the native model
zgplvyou Nov 5, 2026
a90e5a7
vllm-atom: GLM-5.3-Flash plugin review fixes
zgplvyou Nov 5, 2026
36593ec
vllm-atom: test GLM-5.3-Flash prefill chunks shorter than index_kpool
zgplvyou Nov 6, 2026
eb07d7e
vllm-atom: expand the kpool tail for prefill rows shorter than a pool
zgplvyou Nov 6, 2026
4c33493
ci: run the GLM-5.3-Flash plugin cell on fp8 KV
zgplvyou Nov 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions .github/workflows/atom-vllm-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,16 +142,6 @@ jobs:
lm_eval_num_fewshot: 20
accuracy_test_threshold: 0.93
runner: linux-atom-pitt-mi350x-8
- display_name: "GLM-5.2-MXFP4-MTP TP4"
model_name: "GLM-5.2-MXFP4-MTP"
model_path: "amd/GLM-5.2-MXFP4"
extra_args: "--trust-remote-code --tensor-parallel-size 4 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.9 --additional-config '{\"online_quant_config\": {\"global_quant_config\": \"ptpc_fp8\", \"exclude_layer\": [\"lm_head\", \"model.embed_tokens\", \"*.mlp.gate\", \"model.layers.[0-9].mlp.*expert*\", \"model.layers.[1-6][0-9].mlp.*expert*\", \"model.layers.7[0-7].mlp.*expert*\"]}}' --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 3}'"
# vLLM keeps GlmMoeDsaForCausalLM on V1 on ROCm unless asked; this
# cell runs it, and its MTP draft, on Model Runner V2.
env_vars: "AITER_QUICK_REDUCE_QUANTIZATION=INT4\nAITER_USE_FLYDSL_MOE_SORTING=1\nVLLM_USE_V2_MODEL_RUNNER=1"
lm_eval_num_fewshot: 20
accuracy_test_threshold: 0.91
runner: linux-atom-pitt-mi350x-8
- display_name: "MiniMax-M3-MXFP4 EAGLE3 TP4"

@valar-cop valar-cop Nov 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this drops the only V2_MODEL_RUNNER=1 coverage for GlmMoeDsa. move the flag to the nightly cell or say V2 is no longer covered.

@PerryZhang01 PerryZhang01 Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stated in the PR body: the nightly GLM-5.2-MXFP4 MTP TP4 cell runs on V1, so the PR gate no longer runs GlmMoeDsa on V2. DeepSeek-V4-Flash keeps the gate's V2 coverage.

model_name: "MiniMax-M3-MXFP4-EAGLE3"
model_path: "amd/MiniMax-M3-MXFP4"
Expand Down Expand Up @@ -201,6 +191,23 @@ jobs:
# runs; these floors sit near 70% of that.
mtp_per_pos_threshold: "0.55,0.40,0.25,0.18,0.10"
runner: linux-atom-pitt-mi350x-8
- display_name: "GLM-5.3-Flash kpool TP4"
model_name: "GLM-5.3-Flash-kpool"
model_path: "zai-org/GLM-5.3-Flash"
# Chat 16-shot: every prompt runs 2763-3591 tokens, past
# index_topk=2048, so the pooled (kpool) indexer decides what the
# model attends to. At 3-shot it would select every token and the
# cell would only show the pooled writes did no harm.
# `--kv-cache-dtype bfloat16` overrides the harness's fp8: bf16 is
# the configuration validated here and in the native cell.
client_command: "lm_eval --model local-chat-completions --apply_chat_template --fewshot_as_multiturn --model_args model=${MODEL_PATH},base_url=http://127.0.0.1:${VLLM_PORT}/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,max_gen_toks=512 --tasks gsm8k --num_fewshot ${LM_EVAL_NUM_FEWSHOT} --batch_size 65 --output_path ${OUTPUT_PATH}"
extra_args: "--kv-cache-dtype bfloat16 --tensor-parallel-size 4 --max-model-len 8192 --max-num-batched-tokens 8192 --max-num-seqs 32 --gpu-memory-utilization 0.85"
lm_eval_num_fewshot: 16
# Measured 0.9697 / 0.9666 flexible-extract over two runs of this
# exact serve command (native ATOM: 0.9666); 0.94 matches the
# native GLM-5.3-Flash-kpool-16shot gate.
accuracy_test_threshold: 0.94
runner: linux-atom-pitt-mi350x-8
- display_name: "Qwen3.5-397B-A17B-MXFP4 TP4"
model_name: "Qwen3.5-397B-A17B-MXFP4"
model_path: "amd/Qwen3.5-397B-A17B-MXFP4"
Expand Down
Loading
Loading