[taisazero]

I added a few messages to instruct folk to install fschat and accelerate (due to a nested dependency).
I also added native FastAPI access token authentication to vllm's openai server and vanilla server.

[XBeg9]

Any ideas when this will be merged?

[simon-mo]

Hi @taisazero, thank you for your PR. Can you help modify the code to simplify it:

• Do not change the demo api server. Only change OpenAI compatible server. We would like to keep the demo server simple. OpenAI server have more features for production.
• Use starlette or fastapi middleware for token authentication instead. If we can share the same way that OpenAI's client library uses key that would be even better.
• fschat message is not needed as we are trying to moving towards huggingface chat template.

Thank you again!

[taisazero]

@simon-mo I cleaned up the other TODO's. For using middleware do you mean something like this link to remove the need for Depends? I already am using the fastapi- APIKey middleware: https://github.com/taisazero/vllm/blob/main/vllm/entrypoints/openai/api_server.py#L68

[simon-mo]

Yes. I believe instead of depends, using @app.middleware is a more composable and less intrusive approach: https://fastapi.tiangolo.com/tutorial/middleware/

Additionally, let's make it so the the authentication behaves the following:

• By default, no auth
• When the env var or command argument is supplied, check auth

If you can add a snippet to the documentation, that would be even better! Thank you

[simon-mo]

@taisazero please let me know if you need any help in the described items!

[taisazero]

I'll get to it -- just have a lot of deadlines right now 😭

[oushu1zhangxiangxuan1]

Awesome! Long for this!

[Slawikus]

It would be awesome to have this feature merged. Thanks for working on it @taisazero