Skip to content

fix(vllm): forward-port Qwen3 MXFP8 rollout stabilization to main - #4932

Open
humairafirdowse18 wants to merge 3 commits into
mainfrom
codex/forward-port-4479-main
Open

humairafirdowse18 wants to merge 3 commits into
mainfrom
codex/forward-port-4479-main

Conversation

@humairafirdowse18

@humairafirdowse18 humairafirdowse18 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Forward-port #4479 from r0.8.0 to main after the automated three-way cherry-pick failed.

This change:

  • Initializes dummy-loaded MXFP8 E8M0 scale tensors to the finite neutral encoding 127.
  • Handles both linear and MoE MXFP8 scale initialization.
  • Preserves real checkpoint/refit scales after initial model construction.
  • Keeps the first 3 and last 12 Qwen3-32B layers in BF16.
  • Retains the median token_mult_prob_error < 1.1 CI gate while removing the overly sensitive single-step gate.
  • Moves MXFP8 recipe-setting tests into tests/unit/test_recipe_settings.py.

Why

vLLM's dummy loader does not initialize integer scale tensors. MXFP8 E8M0 scale buffers could therefore contain arbitrary bytes, including 255, which represents NaN. Those invalid scales caused unstable rollout probabilities.

Initializing only dummy-loaded scales to the E8M0 encoding for 1.0 prevents NaNs without overwriting scales loaded from checkpoints or later policy refits.

The BF16 boundary layers reduce Qwen3-32B rollout drift while keeping 49 of its 64 transformer layers in MXFP8.

Conflict resolution

The forward-port preserves newer main coverage for:

  • MXFP8 refit optimizations.
  • NCCL-reshard recipe validation.
  • Runtime scale replacement during repeated refits.

The obsolete top-level tests/test_mxfp8_rollout_recipes.py remains deleted, and its latest main coverage is incorporated into tests/unit/test_recipe_settings.py.

Validation

  • ruff check: passed
  • ruff format --check: passed
  • Python compilation checks: passed
  • MXFP8 recipe tests: 28 passed
  • 5 NCCL-reshard checks were dependency-gated in the lightweight local environment because transformers was unavailable
  • git diff --check: passed

The original 10-step workload validation, W&B evidence, review discussion, and CI results are available in #4479.

Before this PR is ready for review

  • Read and followed the contributor guidelines.
  • Preserved and adapted the tests from the original merged PR.
  • Full CI against main is pending.
  • Documentation changes are not required because this is a forward-port of an already documented and reviewed fix.

Original change

Signed-off-by: humairafirdowse18 <humairafirdo@nvidia.com>
(cherry picked from commit b67d99b)
Signed-off-by: humairafirdowse18 <humairafirdo@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@humairafirdowse18 humairafirdowse18 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 27, 2026
@humairafirdowse18

Copy link
Copy Markdown
Contributor Author

/ok to test 6decfda

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers — CI is red at 6decfda on 1 failure caused by this PR.

Thanks for forward-porting the Qwen3 MXFP8 rollout stabilization to main. Carrying main's refit-optimization and NCCL-reshard recipe tests into tests/unit/test_recipe_settings.py keeps that coverage intact.

Scope: forward-port of #4479 onto main, at 6decfda

Checked:

  • Port fidelity — the fp8.py change is byte-identical to the #4479 merge commit
  • Test migration — all 16 tests from main's tests/test_mxfp8_rollout_recipes.py land in tests/unit/test_recipe_settings.py with the same bodies
  • Latest main — #2517's later fp8.py change touches only the block-FP8 MoE path, and the branch merges cleanly

Remaining comments are on:

  • The new dummy-scale MoE test, which predates the shape asserts main has in create_weights_mxfp8_moe

monkeypatch.setattr(
vllm_parameter, "get_tensor_model_parallel_world_size", lambda: 1
)
layer = torch.nn.Module()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_create_mxfp8_moe_initializes_dummy_scales fails on main with AttributeError: 'Module' object has no attribute 'intermediate_size_per_partition'.

On main, create_weights_mxfp8_moe asserts the layer's intermediate_size_per_partition and hidden_size (added by #3294, absent on r0.8.0), and line 1209 passes a bare module.

Suggest fix:

Suggested change
layer = torch.nn.Module()
layer = torch.nn.Module()
layer.hidden_size = 32
layer.intermediate_size_per_partition = 32

@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants