Use CUDA 12.4 as default for release and nightly wheels

[mgoin]

I think it is time to match pytorch's default cuda version. We should still keep wheels built with 12.1 around.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[tlrmchlsmth]

Agree! there are several pieces we pick up going from 12.1 -> 12.4 (Lovelace fp8 kernels, 2:4 sparse kernels, some cuda graph stuff)

[tlrmchlsmth]

We should still keep wheels built with 12.1 around.

+1 to this as well

[youkaichao]

both "Build wheel - CUDA 12.4" and "Build wheel - CUDA 12.1" build with cuda 12.4?

[khluu]

^ this should stay 12.1.0

[mgoin]

Isn't this the release image? I have not changed the "Build wheel - CUDA 12.1" case. Why shouldn't this also be 12.4?

[youkaichao]

please note that, in .buildkite/upload-wheels.sh , we do not upload cuda 11.8 wheels, but will upload cuda 12 wheels.

if you build for both 12.1 and 12.4, make sure only one version is uploaded to avoid upload race condition.

[khluu]

Triggered a test run here: https://buildkite.com/vllm/release/builds/2643

and maybe it's time we stop building cu118 wheels?

[khluu]

^ this should stay 12.1.0

[mgoin]

if you build for both 12.1 and 12.4, make sure only one version is uploaded to avoid upload race condition.

I thought that wheels built that don't match the MAIN_CUDA_VERSION in setup.py will have a +cuXXX added to their version/wheel name

vllm/setup.py

Line 49 in 54cacf0

MAIN_CUDA_VERSION = "12.1"

So I think would prevent wheel collision. It would be nice to have explicit wheel versions like this to make it more transparent what is available

[youkaichao]

@mgoin the upload wheel only checks if cu118 is present 🤕

vllm/.buildkite/upload-wheels.sh

Line 50 in b5b57e3

if [[ $normal_wheel == *"cu118"* ]]; then

[simon-mo]

Looks like PyTorch has 11.8, 12.1, and 12.4. So to confirm, we will publish 12.4 to PyPI, still build for 12.1 and 11.8 for nightly, then push them to artifacts?

[mgoin]

@khluu could you help give this a test again?

[simon-mo]

Build here https://buildkite.com/vllm/release/builds/3324/table?jid=01954347-30ab-4007-b8aa-016419854d17

[mgoin]

Validated with cutlass sparsity support, since we need 12.2 CUDA at least on Hopper

vllm/csrc/sparse/cutlass/sparse_scaled_mm_entry.cu

Lines 8 to 10 in b382a7f

	bool cutlass_sparse_scaled_mm_supported(int64_t cuda_device_capability) {
	// sparse CUTLASS kernels need at least
	// CUDA 12.2 and SM90 (Hopper)

uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/a6022b380c82ea785233baaf37f10d3a0a55d009
python -c "from vllm.model_executor.layers.quantization.utils.w8a8_utils import sparse_cutlass_supported; print(sparse_cutlass_supported())"
True

uv pip install -U vllm==0.7.3
python -c "from vllm.model_executor.layers.quantization.utils.w8a8_utils import sparse_cutlass_supported; print(sparse_cutlass_supported())"
False