Support AsyncPipeline for RESTful API

[toilaluan]

Are you planning to support this feature?
I'm wanna use FastGen in my app but it's not currently support RESTful API asynchronously
vLLM support it's very well: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/api_server.py

I've also use Ray to deploy a server use MIIPipeline with dynamic batching but the performance is far behind vLLM default settings.

[mrwyattii]

@toilaluan we actually have RESTful API capabilities, but we have not fully tested them (these pieces were brought over from MII-Legacy). I suspect enabling it will error out currently.

I can bring this feature back to life later this week or early next week when I find time!

[toilaluan]

@mrwyattii I've tested it, but currently it serve requests sequentially. Hope you can do soon, thank for your work 🔥

[mrwyattii]

Please see the example we have for enabling the REST API: https://github.com/microsoft/DeepSpeed-MII#restful-api

You will need to install from source until we do our next release: pip install git+https://github.com/microsoft/DeepSpeed-MII.git

[dongxiaolong]

Please see the example we have for enabling the REST API: https://github.com/microsoft/DeepSpeed-MII#restful-api

You will need to install from source until we do our next release: pip install git+https://github.com/microsoft/DeepSpeed-MII.git

An openai compatible api would be much easier to use.
such as https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py

[mrwyattii]

@dongxiaolong are you referring to being able to pass "role" and "content" as the input? Could you please open an issue and add the Enhancement label? Thanks!