Skip to content

[Model][Qwen4Exp] Run routed experts on DeepGEMM MegaMoE (MXFP8 or MXFP4) - #64315

Merged
khluu merged 10 commits into
vllm-project:mainfrom
abmfy:qwen4exp-megamoe
Nov 9, 2026
Merged

khluu merged 10 commits into
vllm-project:mainfrom
abmfy:qwen4exp-megamoe

Conversation

@abmfy

@abmfy abmfy commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Note

This description was edited by Claude (an AI assistant) with authorization from Thien Tran (gau-nernst). The previous version is in the edit history.

Overview

Runs Qwen4Exp (Qwen3.8-Flash-Next) routed experts on the DeepGEMM MegaMoE backend (--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}', built on DeepGemmMegaMoEExperts from #61538). Expert parallelism is required.

The routed experts load in the checkpoint's format when MegaMoE can run it, and are requantized otherwise:

Checkpoint experts Default MegaMoE format Forced with --quantization / --quantization-config
BF16 (official Qwen/Qwen3.8-Flash-Next, fused gate_up_proj / down_proj) BF16, loaded as stored mxfp8 or mxfp4
128x128 block FP8 (official Qwen/Qwen3.8-Flash-Next-FP8) MXFP8, requantized with a warning mxfp8 or mxfp4
MXFP8 (ModelOpt) MXFP8, loaded as stored mxfp4
MXFP4 (Quark) MXFP4, loaded as stored mxfp8

Claims

  • Qwen4Exp serves on deep_gemm_mega_moe with DP + EP on Blackwell, from the official BF16 and FP8 checkpoints. MXFP8 and MXFP4 checkpoints load directly; no public one exists yet, so that path is covered by unit tests only (format detection is tested for ModelOpt MXFP8; Quark MXFP4 goes through vLLM's existing Quark config lookup).
  • Accuracy matches the default MoE backend on GSM8K (table below). MXFP8 and MXFP4 MegaMoE also pass non-inferiority against stock FP8 on AIME25, AIME26, GPQA-Diamond and MMMU-Pro.
  • B300, FP8 checkpoint, vs the default MoE backend on the same commit:
    • TP1 prefill throughput: +12-13% (MXFP8), +15-16% (MXFP4).
    • DEP4 decode output throughput: +13-17% (MXFP8), +23-25% (MXFP4); mean TPOT -10-13% / -18-20%.
  • Works with layerwise reload and chunked weight updates that deliver every expert shard (see Limitations).

Validation

Unit tests

pytest tests/models/qwen4_exp/test_mega_moe.py: 10 passed on GB200. They run in the existing qwen4-exp-hc-kernels-b200 Buildkite step. The tests cover:

  • format selection from the checkpoint and the online quantization config, including ignore, targets and forced MXFP8/MXFP4;
  • loading per-expert block-FP8, MXFP8 and MXFP4 tensors and the official checkpoint's fused BF16 tensors into the kernel layout on one EP rank of two, bitwise equal to the generic DeepGemmMegaMoEExperts.weight_loader fed tensors requantized with DeepGEMM's own casts;
  • weight and scale arriving in separate load_weights calls, and the post-load error on an incomplete pair (requantized formats);
  • layerwise reload replacing the experts in the original storage.

The MegaMoE kernels themselves are covered by tests/kernels/moe/test_deep_gemm_mega_moe.py. CI does not run Qwen4ExpMegaMoeBlock's forward or the decoder's backend selection; the GSM8K runs below cover them end to end.

GSM8K

4x GB200 node, DP4 + EP, MTP3 speculative decoding, 5-shot, greedy, thinking off, full test set (1319 questions), one run per arm:

Checkpoint MoE backend Experts flexible / strict
BF16 default BF16 96.82 / 96.82
BF16 MegaMoE BF16, as stored 96.74 / 96.82
FP8 default block FP8 97.19 / 97.27
FP8 MegaMoE MXFP8, requantized 96.66 / 96.74

Four runs of the FP8 MegaMoE arm on different revisions of this branch scored 96.59-96.89 flexible. The requantized weights are identical across those revisions, so this range is run-to-run noise.

Reasoning evals and throughput (FP8 checkpoint, B300)

These were measured on an earlier revision of this PR. The FP8 -> MXFP8/MXFP4 requantization is numerically unchanged since then: same DeepGEMM casts, applied row by row.

Reasoning evals with thinking on, temperature=1.0, top_p=0.95, top_k=20, max_tokens=98304, MTP3. Stock FP8 is TP4 + EP on the default MoE backend; MegaMoE is DP4 + EP. Rule: paired difference vs stock with a bootstrap 95% CI; non-inferior if the lower bound is >= -2 pt (AIME) or -1 pt (GPQA-D, MMMU-Pro).

Benchmark Stock FP8 MegaMoE MXFP8 Δ vs stock [95% CI] MegaMoE MXFP4 Δ vs stock [95% CI]
AIME25 99.69 99.69 +0.00 [-0.52, +0.52] 99.32 -0.36 [-1.04, +0.16]
AIME26 97.19 97.50 +0.31 [-0.52, +1.15] 96.77 -0.42 [-1.88, +0.73]
GPQA-Diamond 90.81 91.26 +0.44 [-0.19, +1.10] 90.74 -0.08 [-0.84, +0.66]
MMMU-Pro (flexible) 76.82 77.02 +0.20 [-0.52, +0.92] 76.73 -0.09 [-0.70, +0.51]

vllm bench serve --dataset-name random --ignore-eos, --max-num-seqs 256 --gpu-memory-utilization 0.9 --no-enable-prefix-caching, no speculative decoding; each point run twice (runs differed by <= 3%).

Prefill, TP1, 1 output token Stock tok/s MXFP8 MXFP4
8192 x 256 prompts, concurrency 64 45,188 51,009 (+12.9%) 52,441 (+16.1%)
32768 x 64 prompts, concurrency 16 43,252 48,633 (+12.4%) 49,865 (+15.3%)
Decode, DP4 + EP, 2048 in / 1024 out Stock out tok/s MXFP8 MXFP4
concurrency 128 8,348 9,456 (+13.3%) 10,447 (+25.1%)
concurrency 256 12,453 14,045 (+12.8%) 15,269 (+22.6%)
concurrency 512 17,268 20,123 (+16.5%) 21,356 (+23.7%)

Weight loading

FP8 checkpoint -> MXFP8 on GB200: Loading weights took 48 s per rank at DP4 + EP, and 148 s at TP1. TP1 is the worst case: one rank requantizes all 512 experts of 48 layers, one expert projection at a time. Batching a projection's experts loaded in 120 s at TP1, but needed a chunking bound and a deferred flush; the simpler per-expert path was preferred.

Details

Format selection. There is no new flag. mega_moe_expert_formats reads the checkpoint's expert format (Fp8Config, or get_checkpoint_quant_keys for ModelOpt and Quark), then applies the online quantization config per layer. moe or targets forces MXFP8 or MXFP4. Layers matched by ignore keep the default. Requantized layers show in the online quantization summary. The MTP drafter's config does not carry the online config, so its experts take the default format.

Loading. Experts in a MegaMoE format are written straight into the kernel layout through each parameter's weight_loader, as for DeepSeek V4, Kimi K3 and MiniMax M3. Other experts are requantized one expert projection at a time, as soon as its weight and scale have both arrived: dequantize to FP32, then DeepGEMM's per_token_cast_to_fp8 / per_token_cast_to_fp4 with UE8M0 scales over 1x32 groups. The results go through the same weight_loader, so layerwise reload (reload_weights, NCCL/IPC weight transfer) can buffer and replay them. A chunked update may split an expert's weight from its scale; the half pair waits for the next load_weights call, and process_weights_after_loading fails on any pair still incomplete.

Shared expert. Qwen4Exp gates the shared expert per token with sigmoid(shared_expert_gate(x)), which MegaMoE's fused shared expert cannot apply, so the shared expert stays unfused. It runs on an auxiliary stream while the router and top-k run (maybe_execute_in_parallel, which runs both on one stream under breakable CUDA graph capture), and joins before the MegaMoE launch, which needs every SM. VLLM_DISABLE_SHARED_EXPERTS_STREAM turns the overlap off.

Other changes. The DeepGEMM MegaMoE warmup covers Qwen4Exp blocks, and its cache key includes the routed-expert format.

Docs. None, as for the other DeepGEMM MegaMoE integrations (GLM-5.3-Flash, Kimi K3, MiniMax M3, Step5): the docs do not cover the deep_gemm_mega_moe backend for any model.

Limitations.

  • TP > 1 is rejected (it needs sequence-parallel inputs on the HC path); use DP + EP. Non-linear --expert-placement-strategy is rejected.
  • NVFP4 is not wired up. At Qwen4Exp's intermediate size (640, not a multiple of 256), the pinned DeepGEMM nvfp4_mega_moe kernel issues ceil(640 / 256) = 3 TMA stages per L2 task while the MMA warp consumes floor = 2. Large launches hang and small ones return wrong output.
  • Explicit activation precisions in the online config are ignored: MegaMoE fixes them per kernel (MXFP8 dispatch for MX experts, BF16 for BF16).
  • Mixing MXFP8 and MXFP4 layers allocates one symmetric buffer per MMA type, about 1-2.6 GiB per GPU each for these shapes. Pooling them is a follow-up in the generic MegaMoE layer.
  • EPLB plumbing comes from DeepGemmMegaMoEExperts but was not exercised here.
  • A layerwise reload that sends a layer's weights without any of their scales writes nothing to that layer: vLLM restores its previous experts (logging Failed to load weights), and the unpaired tensors stay buffered. For natively loaded MXFP8/MXFP4 experts, a missing scale is caught only when no scale of that parameter arrives at all, as for the other MegaMoE models.
  • Moving DeepGEMM MegaMoE behind the MoE oracle, as [ROCm][MoE] Move AITER MegaMoEV2 To Oracle #63229 does for AITER MegaMoE, would let it reuse the generic online requantization. That is a cross-model refactor left for later.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--64315.org.readthedocs.build/en/64315/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build qwen Related to Qwen models labels Oct 28, 2026
@mergify

mergify Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @abmfy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Comment on lines +127 to +238
def flush_mega_moe_requantize(model: nn.Module) -> None:
"""Requantize the complete expert shards buffered by ``load_weights``.

Half pairs stay buffered: a chunked weight update may pass an expert's
weight and its scale in different calls.
"""
for module in model.modules():
if isinstance(module, Qwen4ExpMegaMoEExperts):
module.flush_requantize()


class _FinishRequantize(QuantizeMethodBase):
"""Flushes the experts; fails on shards missing their weight or scale."""

requires_device_loading = False
supports_pre_processed_weights = True

def create_weights(self, layer: nn.Module, *args, **kwargs) -> None:
raise NotImplementedError("MegaMoE experts allocate weights at construction")

def apply(self, layer: nn.Module, *args, **kwargs) -> torch.Tensor:
raise NotImplementedError("MegaMoE experts run through their forward")

def process_weights_after_loading(self, layer: nn.Module) -> None:
assert isinstance(layer, Qwen4ExpMegaMoEExperts)
layer.flush_requantize(strict=True)


class Qwen4ExpMegaMoEExperts(DeepGemmMegaMoEExperts):
"""MegaMoE experts loaded from per-expert block-FP8 checkpoint tensors.

Each EP rank buffers the complete (weight, scale) pairs of its own experts
and requantizes them per projection in a few batched calls.
"""

def __init__(self, vllm_config: VllmConfig, **kwargs) -> None:
super().__init__(vllm_config, activation=MegaMoEActivation(), **kwargs)
self.quant_method = _FinishRequantize()
# Parameters may sit on meta (weight-cache IPC loading, layerwise
# reload), so stage and requantize on the current GPU.
self._requantize_device = torch.device(
current_platform.device_type, torch.accelerator.current_device_index()
)
self._pending_shards: dict[tuple[int, str], dict[str, torch.Tensor]] = {}
# Complete (expert, weight, block scale) shards per shard id.
self._ready_shards: dict[str, list[tuple[int, torch.Tensor, torch.Tensor]]] = {}
# Logical experts this rank holds (redundant replicas share one).
self._num_owned_experts = len(
{
p % self.num_logical_experts
for p in range(self.experts_start_idx, self.experts_end_idx)
}
)

def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
"""Load ``{expert}.{gate,up,down}_proj.{weight,weight_scale_inv}``."""
loaded: set[str] = set()
for name, tensor in weights:
expert, proj, kind = name.split(".")
expert_id, shard_id = int(expert), _CHECKPOINT_SHARDS[proj]
prefix = "w2" if shard_id == "w2" else "w13"
loaded.update((f"{prefix}_weight", f"{prefix}_weight_scale"))
if not self._map_global_expert_id(expert_id):
continue
pending = self._pending_shards.setdefault((expert_id, shard_id), {})
# Stage on the GPU as it arrives: no CPU-side torch ops, which stall
# when several engines load on one host.
pending[kind] = tensor.to(self._requantize_device)
if len(pending) < 2:
continue
del self._pending_shards[(expert_id, shard_id)]
ready = self._ready_shards.setdefault(shard_id, [])
ready.append((expert_id, pending["weight"], pending["weight_scale_inv"]))
if len(ready) == self._num_owned_experts:
self._requantize_ready(shard_id)
return loaded

def flush_requantize(self, strict: bool = False) -> None:
for shard_id in list(self._ready_shards):
self._requantize_ready(shard_id)
if strict and self._pending_shards:
raise ValueError(
f"{self.prefix}: expert shards without both weight and scale: "
f"{sorted(self._pending_shards)[:4]}"
)

def _requantize_ready(self, shard_id: str) -> None:
ready = self._ready_shards.pop(shard_id)
rows = ready[0][1].shape[0]
if any(w.shape != ready[0][1].shape for _, w, _ in ready):
raise ValueError(f"{self.prefix}: {shard_id} shards differ in shape.")
prefix = "w2" if shard_id == "w2" else "w13"
chunk = max(1, _REQUANTIZE_CHUNK_ELEMENTS // ready[0][1].numel())
for start in range(0, len(ready), chunk):
batch = ready[start : start + chunk]
weight, scale = requantize_fp8_block(
torch.cat([w for _, w, _ in batch]),
torch.cat([s for _, _, s in batch]),
self.weight_format,
)
for i, (expert_id, _, _) in enumerate(batch):
for suffix, value in (("weight", weight), ("weight_scale", scale)):
param_name = f"{prefix}_{suffix}"
# The parameter's own loader: layerwise reload wraps it.
param = getattr(self, param_name)
param.weight_loader(
param,
value[i * rows : (i + 1) * rows],
param_name,
shard_id=shard_id,
expert_id=expert_id,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you check #64378 #60276 if you could instead use the model-agnostic infra available at vllm/model_executor/layers/quantization/online/?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to land this PR soon so I stick to the existing per-model DeepGEMM integration. We can do a refactor for all models in a future PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I realized MegaMOE on cuda side does not use this online quantization infra. It's fine

)


class Qwen4ExpMegaMoeBlock(nn.Module):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance you could use oracle + experts implementation (similar to #63229) if this can be re-used for other models?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be done later - existing code in vllm does not use that

vllm/vllm/config/kernel.py

Lines 163 to 166 in 28dd9c3

# "Native" here meaning that the backend does not use the fused MoE path.
# Currently only relevant for the vLLM provided DeepGEMM MegaMoE.,
# TODO: migrate to the fused MoE path ultimately.
NATIVE_MEGA_MOE_BACKENDS = frozenset({"deep_gemm_mega_moe"})

@mergify

mergify Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @abmfy.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

abmfy and others added 8 commits November 9, 2026 06:24
…FP4)

With --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' and expert
parallelism, Qwen4Exp's routed experts run as one DeepGEMM MegaMoE launch
(fused EP dispatch, expert GEMMs and combine) instead of the FusedMoE path.

The official checkpoint stores experts as 128x128 block FP8, which MegaMoE
does not take. Each EP rank requantizes the experts it owns as they load,
to MXFP8 (1x32 UE8M0 scales) by default, or to MXFP4 with
--quantization-config '{"moe": "mxfp4"}'; ignore/targets patterns of the
online quantization config pick formats per layer, and requantized layers
show in the online quantization summary. The MTP drafter's config does not
carry --quantization-config, so its experts stay MXFP8.

Requantization rounds as DeepGEMM's own casts, runs on the GPU and is
batched over a projection's local experts. Shards are written through each
parameter's weight_loader, so layerwise reload and chunked weight updates
(which may split an expert's weight from its scale) work; a shard still
missing its pair after loading fails in process_weights_after_loading.

The sigmoid-gated shared expert stays unfused and runs on an auxiliary
stream next to the router (VLLM_DISABLE_SHARED_EXPERTS_STREAM turns it
off). TP>1 is not supported yet (use DP+EP). The DeepGEMM MegaMoE warmup
covers Qwen4Exp blocks and keys its cache by the routed-expert format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bowen Wang <bowen@inferact.ai>
…ntization

The Qwen4Exp MegaMoE loader accepted only 128x128 block-FP8 checkpoints.
It now reads the routed experts in every format MegaMoE runs, and
requantizes when a format is forced:

- BF16 (the official checkpoint's fused gate_up_proj/down_proj, or
  per-expert tensors), MXFP8 (ModelOpt) and MXFP4 (Quark) experts load
  straight into the kernel layout. BF16 runs the bf16_mega_moe kernel.
- --quantization mxfp8|mxfp4 or --quantization-config moe/targets force
  a format; the experts are then dequantized from their checkpoint
  format and requantized with DeepGEMM's casts, as before for block FP8.
- Block-FP8 experts, which MegaMoE cannot run, still default to MXFP8,
  now with a warning that names the flag to choose the format.
- Checkpoint formats are read with get_checkpoint_quant_keys; unexpected
  per-expert tensors fail with an explicit error.

Tests cover every (checkpoint format, MegaMoE format) pair: loading into
the kernel layout, requantization, layerwise reload, and the kernel
against a dequantized reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
- Dequantize MX checkpoints in torch; only the cast needs DeepGEMM.
- Drop the BF16 kernel probe, a dead online target and the docs changes.
- Run the tests in the existing B200 Qwen4 Exp kernels job.
- Test one representative case per path instead of every format pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
…fter_loading

Drop the module-level flush called from Qwen4ExpModel/Qwen4ExpMTP
load_weights: every load path (initial, layerwise and direct reload)
already runs the experts' process_weights_after_loading, which now
requantizes whatever is still buffered, as Step5's MegaMoE does.

Trim the tests: the load test checks requantization against DeepGEMM's
own casts, which subsumes the requantize test; the kernels are covered
by tests/kernels/moe/test_deep_gemm_mega_moe.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Requantize each expert shard as soon as its weight and scale have both
arrived instead of batching a projection's owned experts. Staging stays
at one shard, which removes the batching, the chunk bound and the
deferred requantization; process_weights_after_loading only checks that
no shard is missing its weight or scale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Drop the module-level E2M1 table, which a model built under a device or
dtype context could create on the GPU at import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
…ntize into requantize

The deep_gemm module re-exports its utils, so the import_module detour is
not needed. Dequantization has a single caller; merge it into a private
_requantize and clarify the _CheckLoaded and MTP docstrings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
@gau-nernst
gau-nernst marked this pull request as ready for review November 9, 2026 06:25
@gau-nernst
gau-nernst requested a review from a team November 9, 2026 06:25

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@gau-nernst

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101001 for commit d5be2a513eaa.

- Overlap the shared expert with routing through maybe_execute_in_parallel,
  which also falls back to one stream under breakable CUDA graph capture.
- Report only the expert parameters a load actually wrote, so the loader's
  missing-parameter check still catches a checkpoint without scales.
- Test a targets override in format selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
@gau-nernst

Copy link
Copy Markdown
Contributor

/ci cancel

@gau-nernst

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Requested cancellation of 1 CI build for qwen4exp-megamoe: #101001.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #101034 for commit f42ca669d5fe.

ℹ️ This PR is 7 commits behind upstream main. Merged upstream main into this branch as f42ca669d5fe before starting CI. Pull the branch before pushing more changes.

@khluu khluu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI stamp

@gau-nernst
gau-nernst enabled auto-merge (squash) November 9, 2026 08:44
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 9, 2026
@gau-nernst

Copy link
Copy Markdown
Contributor

/ci retry

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #101034.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build documentation Improvements or additions to documentation qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants