Async vllm

[clefourrier]

Adds the option to use the new AsyncVLLM from vllm v1. It supports DP + PP/TP, but not setting the batch size, and deploys an independent async VLLM model which manages requests on its own through the async engine.

Thanks to the kind people at vllm vllm-project/vllm#17385, I realised we actually have to use a single event loop for async models, so this is what it does - it's also very fast now.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[clefourrier]

Linked to #670

[ScottHoang]

awesome !

[clefourrier]

Or @lewtun if you want to take a look?

[clefourrier]

Btw, interestingly I suspect that our current other async model calls (like the tgi ones) were only successful because we were only implementing one type of async loop - we might want to unify them in a later PR

[lewtun]

Really nice feature @clefourrier ! The logic LGTM and I'm curious if you've benchmarked both the speed up and whether there is approximate parity with the synchronous version for a small model on a few benchmarks?

I didn't quite understand the constraint about batch size = 1 because AFAIK vllm doesn't expose that: normally you just pass a list of prompts and let the engine's continuous batching handle the scheduling.

[clefourrier]

Re batch size = 1, batch size of more is simply not supported yet in the generate method of the AsyncLLM model (unless I'm reading this comment wrong ^^)

[lewtun]

Re batch size = 1, batch size of more is simply not supported yet in the generate method of the AsyncLLM model (unless I'm reading this comment wrong ^^)

Ah I see, so in the async version one cannot pass a list of prompts. It would be interesting to benchmark one of the pass@1 evals like AIME24 with a DeepSeek-Distill model to get a sense for how much of a speed difference this makes (mostly asking to see if we adopt it in open-r1

[clefourrier]

On it, so DP2 with ray vs with async?

[NathanHB]

great PR ! Once tests are done it's ready :)
We can keep the cleanup for adadpter models in this PR, those are small atomic changes so i think it's fine

[clefourrier]

Ok so I found a fun thing - I assumed that the .generate method actually sent back the total generation but... it does not - you need to iterate on it to get a generation token per token. (I was getting 0 all day and empty text on my generative evals).
So do not merge, I still need to do some time estimates now ^^"

[clefourrier]

There's an issue on the pass @ metrics that I need to investigate as they are failing

[lewtun]

On it, so DP2 with ray vs with async?

That would be a good test! Even better would be DP=8 if you can get a free node :)

[clefourrier]

Spent my day on this, I suspect I'm either missing something extremely trivial in how the sampling is done in the async vllm or how I should gather results - I need to start working on stg else so feel free to take a look if it's urgetn