[liusy58]

Overview

Enable serialized ModelOpt per-tensor and per-channel FP8 linear weights to be reused by the weight-cache daemon with CUTLASS. Related to #56049; depends on #63672 for restoring CUTLASS's logical output width.

Incremental review: liusy58/vllm@3571116...7f75449 (the GitHub diff against main includes the prerequisite).

Claims

• Enable CUDA IPC for ModelOpt FP8 and FP8_PER_CHANNEL_PER_TOKEN linear layers using CUTLASS and the default format scheme.
• Restore the transposed-weight layout metadata without transposing, requantizing, padding, or mutating the daemon's already processed weights/scales a second time.
• Keep other ModelOpt FP8 backends and block FP8 unsupported; existing FP4/MXFP8 capability rules are unchanged.

Validation

Current head 7f75449765425d759565d3a770d829070d909f99 includes main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e. October 29 agent-run validation after resolving the inherited online-FP8 import conflict with upstream #60163. The merge retains both CUTLASS and Humming imports, the upstream processed-dimension metadata, and upstream rejection of the unsupported Humming per-token/channel path. It adds no Humming IPC capability.

Check	Exact-head result
modelopt-layers	117 passed, 5 deselected, 14 warnings in 85.71s (0:01:25)
modelopt-model	1 passed, 70 deselected, 18 warnings in 78.89s (0:01:18)
Full feature-file and commit hooks	Passed, including mypy and ruff

Runtime: NVIDIA B300, PyTorch 2.13.0+cu130, process-local CUDA 13.4 compatibility libraries. The two stable-libtorch extensions are from the official CUDA 13.0 wheel for main 914d61fc538a84e8487cb16fe8241a70f6f4bf9e, checked against wheel RECORD hashes and linked only into task worktrees. The shared venv and host driver were unchanged. This does not validate the full upstream PyTorch 2.14 dependency matrix.

Full-model checks use TP1, BF16, eager mode, CUTLASS and TRITON_ATTN, with the existing batch-invariant exact greedy-output comparisons. Each completed case checks ordinary load, missing-daemon fallback, IPC and a second IPC load; both IPC loads forbid fallback. These are output-equivalence smoke checks, not accuracy or performance benchmarks. No multi-GPU, ROCm or other-backend result is claimed.

ModelOpt full-model tests use their checked-in configurations without checkpoint/budget overrides. Per-channel/per-token checkpoint: CedricHwang/qwen2.5-0.5b-modelopt-fp8-pc-pt, pinned d9c87375a9b950b1c7965a1110ac6ca455929726.

Commands for this revision

All commands set HF_HUB_OFFLINE=1, PYTHONPATH to the corresponding worktree and LD_LIBRARY_PATH=/home/liusy/codex-runtime/cuda-compat-13-4/usr/local/cuda-13.4/compat.

CUDA_VISIBLE_DEVICES=5  .venv/bin/python -m pytest tests/quantization/test_modelopt.py -k 'not checkpoint_setup and not nested_mixed_checkpoint and not mixed_dspark_checkpoint' -q
CUDA_VISIBLE_DEVICES=5  .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k modelopt_fp8 -q

GitHub checks for the new head are pending at publication. No Buildkite GPU-suite pass or completed human review is claimed. AI assistance and original feature baselines are recorded below.

Earlier validation and original feature baselines (previous heads)

NVIDIA B300, BF16, CUTLASS linear backend. The real-model test uses CedricHwang/qwen2.5-0.5b-modelopt-fp8-pc-pt, pinned to d9c87375a9b950b1c7965a1110ac6ca455929726, TP1, eager mode, TRITON_ATTN and VLLM_BATCH_INVARIANT=1.

Check	Before implementation, base 93a266aaa2 plus new tests	With implementation
Real checkpoint, ordinary loading → missing-daemon fallback → IPC → second IPC	1 failed, 48.62s: daemon rejects ModelOptLinearMethod	1 passed, 74.98s; exact output equality for both prompts × 16 generated tokens across all four loads
Fused linear layers, tensor/channel scales × aligned/padded output widths, two attachments each	4 failed, 10.28s: pre-processed weights rejected	All 4 pass; exact outputs, shared pointers and tensor contents preserved
Unsupported Torch/Triton backend rejection	—	Both pass
Existing ModelOpt suite excluding five checkpoint-setup/integration cases; includes all six new layer cases	—	113 passed, 5 deselected, 72.13s
Changed-file and commit hooks	—	Passed, including mypy and ruff

Both real IPC loads set fallback=False. The deterministic generation comparison is a smoke evaluation, not a broad accuracy benchmark. Full-model coverage is per-channel/per-token; static per-tensor FP8 is covered by real CUDA layers with unequal fused-projection scales. No performance, multi-GPU, other-platform, or FP8-MoE claim is made.

CUDA_VISIBLE_DEVICES=2 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/model_executor/model_loader/test_weight_cache.py -k modelopt_fp8 -q
CUDA_VISIBLE_DEVICES=2 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/quantization/test_modelopt.py -k 'preprocessed_shared_weights or preprocessed_unsupported_backend' -q
CUDA_VISIBLE_DEVICES=2 HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest tests/quantization/test_modelopt.py -k 'not checkpoint_setup and not nested_mixed_checkpoint and not mixed_dspark_checkpoint' -q

HF_HUB_OFFLINE=1 assumes the pinned checkpoint and OPT-125m configuration are cached; omit it for the initial download. Before implementation the first command reported:

UnsupportedQuantForIPCError: layer model.layers.0.self_attn.qkv_proj:
ModelOptLinearMethod does not support loading from pre-processed weights.
1 failed, 70 deselected, 16 warnings in 48.62s

The four new shared-weight layer cases reported RuntimeError: layer <root>: ModelOptLinearMethod does not support pre-processed weights. With the implementation, all six new cases passed in 12.00s; the full selected regression run above was repeated after test formatting/type-check fixes. Runs emitted existing torch deprecation warnings. Initial setup attempts encountered a missing offline revision reference and a changing-free-memory profiling assertion; the pinned, isolated baseline above then reproduced the intended rejection.

Details

The daemon currently rejects every ModelOpt FP8 linear method even when CUTLASS can consume the processed tensor layout. Opt in only the two plain FP8 weight schemes with the actual selected CUTLASS kernel and no format wrapper. The engine restores processed_weight_dims=(0, 1) and returns before post-load weight transformations. CUTLASS logical width is initialized from configuration by #63672, so padded shared weights still produce the original output width.

The existing L4 preload CI step now selects the real ModelOpt IPC test and the six layer cases, with dependencies on the changed ModelOpt source and tests. The preload feature guide documents the formats, matching daemon/engine kernel configuration, and exclusions. Local validation used B300; L4 CI has not been run locally. No new scheduler/model-runner path or steady-state GPU synchronization.

Duplicate checks on October 26 covered #56049 and its comments, issue-linked PRs, and open searches for ModelOpt FP8 preload/pre-processed weights. #63672 covers online quantization and NVFP4, #63673 covers compressed-tensors FP8, and #62036 covers W4A16 NVFP4 layerwise reload; none implements this serialized ModelOpt FP8 IPC path. This branch includes main 373614f66d.

AI assistance: Codex implemented and ran the recorded validation at the submitter's request. This is a draft for human review; no completed human approval or human-run tests are asserted.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Narrow capability opt-in and metadata restoration; existing loading and kernel interfaces reused.

• Testing and Validation: Real checkpoint before/after, exact real-layer tests, regression results and CI resource limits recorded above.

• Code Quality and Style: Changed-file hooks passed; rendered preload docs and CI selection updated.

• Pull Request Contents: Root cause, prerequisite, duplicate checks, commands, evidence, limitations and AI disclosure included.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[mergify]

Documentation preview: https://vllm--63754.org.readthedocs.build/en/63754/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liusy58.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liusy58.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Isotr0py]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99580 for commit fad81c78cd30.

ℹ️ This PR is 4 commits behind upstream main. Merged upstream main into this branch as fad81c78cd30 before starting CI. Pull the branch before pushing more changes.