Repository navigation
fix(vllm): forward-port Qwen3 MXFP8 rollout stabilization to main - #4932
humairafirdowse18 wants to merge 3 commits into
Conversation
Signed-off-by: humairafirdowse18 <humairafirdo@nvidia.com> (cherry picked from commit b67d99b) Signed-off-by: humairafirdowse18 <humairafirdo@nvidia.com>
|
/ok to test 6decfda |
yuki-97
left a comment
There was a problem hiding this comment.
Merge blockers — CI is red at 6decfda on 1 failure caused by this PR.
Thanks for forward-porting the Qwen3 MXFP8 rollout stabilization to main. Carrying main's refit-optimization and NCCL-reshard recipe tests into tests/unit/test_recipe_settings.py keeps that coverage intact.
Scope: forward-port of #4479 onto main, at 6decfda
Checked:
- Port fidelity — the
fp8.pychange is byte-identical to the #4479 merge commit - Test migration — all 16 tests from main's
tests/test_mxfp8_rollout_recipes.pyland intests/unit/test_recipe_settings.pywith the same bodies - Latest main — #2517's later
fp8.pychange touches only the block-FP8 MoE path, and the branch merges cleanly
Remaining comments are on:
- The new dummy-scale MoE test, which predates the shape asserts main has in
create_weights_mxfp8_moe
| monkeypatch.setattr( | ||
| vllm_parameter, "get_tensor_model_parallel_world_size", lambda: 1 | ||
| ) | ||
| layer = torch.nn.Module() |
There was a problem hiding this comment.
test_create_mxfp8_moe_initializes_dummy_scales fails on main with AttributeError: 'Module' object has no attribute 'intermediate_size_per_partition'.
On main, create_weights_mxfp8_moe asserts the layer's intermediate_size_per_partition and hidden_size (added by #3294, absent on r0.8.0), and line 1209 passes a bare module.
Suggest fix:
| layer = torch.nn.Module() | |
| layer = torch.nn.Module() | |
| layer.hidden_size = 32 | |
| layer.intermediate_size_per_partition = 32 |
Summary
Forward-port #4479 from
r0.8.0tomainafter the automated three-way cherry-pick failed.This change:
127.token_mult_prob_error < 1.1CI gate while removing the overly sensitive single-step gate.tests/unit/test_recipe_settings.py.Why
vLLM's dummy loader does not initialize integer scale tensors. MXFP8 E8M0 scale buffers could therefore contain arbitrary bytes, including
255, which represents NaN. Those invalid scales caused unstable rollout probabilities.Initializing only dummy-loaded scales to the E8M0 encoding for
1.0prevents NaNs without overwriting scales loaded from checkpoints or later policy refits.The BF16 boundary layers reduce Qwen3-32B rollout drift while keeping 49 of its 64 transformer layers in MXFP8.
Conflict resolution
The forward-port preserves newer
maincoverage for:The obsolete top-level
tests/test_mxfp8_rollout_recipes.pyremains deleted, and its latestmaincoverage is incorporated intotests/unit/test_recipe_settings.py.Validation
ruff check: passedruff format --check: passedtransformerswas unavailablegit diff --check: passedThe original 10-step workload validation, W&B evidence, review discussion, and CI results are available in #4479.
Before this PR is ready for review
mainis pending.Original change
b67d99b7fc7abbb7365a830ddf13ca89d738e404