[Feature Request]: Support LLMs, embeddings & reranking models served through vLLM

[K-Mistele]

Is there an existing issue for the same feature request?

• I have checked the existing issues.

Is your feature request related to a problem?

No response

Describe the feature you'd like

It would be great if RAGflow could support LLMs served through vLLM's OpenAI-compatible APi server, and if RAGflow could also support embedding models and re-ranking models served through vLLM.

Describe implementation you've considered

vLLM is the most popular open-source framework for high-performance, GPU-accelerated serving of LLMs and other AI models, and it would be great to have support for it.

vLLM has a well-documented API for LLM serving and embeddings (which is compatible with OpenAI's format), and a custom API for serving reranking models:

• Chat Completions API documentation
• Embeddings API documentation
• Rerank API documentation, named "score"

Documentation, adoption, use case

No response

Additional information

No response

[KevinHuSh]

What about this?

[K-Mistele]

vLLM's rerank API implementation is different than the OpenAI-compatible API option's configuration unfortunately. Maybe I will just introduce a PR into vLLM to change the interface; although I'm curious where the implementation that you have for the OpenAI-compatible rerank API comes from since OpenAI doesn't actually have a rerank API.

[KevinHuSh]

Got it. We're gona support it latter.

[yukon12345]

Got it. We're gona support it latter.

I earnestly hope that full compatibility with vLLM can be achieved. LLM run by ollama are not very suitable for multi-user use.

[K-Mistele]

Got it. We're gona support it latter.

I earnestly hope that full compatibility with vLLM can be achieved. LLM run by ollama are not very suitable for multi-user use.

This is actually supported now. vLLM already has OpenAI-compatible completions, chat completions, and empeddings endpoints, and the PR I linked above (vllm-project/vllm#12376) merged in a Cohere- and Jina AI-compatible API endpoint.
Since OpenAI doesn't have a re-rank API, RagFlow's "OpenAI-compatible rerank" is actually just a Jina/Cohere-compatible implementation.

So, you can use the "OpenAI compatible" option which picking chat, vision, embedding, and rerank models in the dashbaord, and specify the URL to whatever vLLM servers you are using to host these models.

If ragflow supports jina AI / cohere AI -compatible rerank endpoints, you can also configure these to point at vLLM that's serving a cross-encoder or embedding model and it will work.

[yukon12345]

Got it. We're gona support it latter.

I earnestly hope that full compatibility with vLLM can be achieved. LLM run by ollama are not very suitable for multi-user use.

This is actually supported now. vLLM already has OpenAI-compatible completions, chat completions, and empeddings endpoints, and the PR I linked above (vllm-project/vllm#12376) merged in a Cohere- and Jina AI-compatible API endpoint. Since OpenAI doesn't have a re-rank API, RagFlow's "OpenAI-compatible rerank" is actually just a Jina/Cohere-compatible implementation.

So, you can use the "OpenAI compatible" option which picking chat, vision, embedding, and rerank models in the dashbaord, and specify the URL to whatever vLLM servers you are using to host these models.

If ragflow supports jina AI / cohere AI -compatible rerank endpoints, you can also configure these to point at vLLM that's serving a cross-encoder or embedding model and it will work.

Thank you very much for your detailed reply.You are so nice。

[dromeuf]

Work well for me with :

(vllm_env) dromeuf@MAIA:~$ vllm serve BAAI/bge-m3 --served-model-name bge-m3 --dtype auto --task embedding --host 0.0.0.0 --port 33434 --api-key vLLM

(vllm_env) dromeuf@MAIA:~$ vllm serve BAAI/bge-reranker-v2-m3 --served-model-name bge-reranker-v2-m3 --dtype auto --task score --host 0.0.0.0 --port 33435 --api-key vLLM

and RAGFlow OpenAI-API-Compatible :

embedding , bge-m3 , http://host.docker.internal:33434 , vLLM , 8192

rerank , bge-reranker-v2-m3 , http://host.docker.internal:33435 , vLLM, 8192

Kind regards, David.

[K-Mistele]

Work well for me with :

(vllm_env) dromeuf@MAIA:~$ vllm serve BAAI/bge-m3 --served-model-name bge-m3 --dtype auto --task embedding --host 0.0.0.0 --port 33434 --api-key vLLM

(vllm_env) dromeuf@MAIA:~$ vllm serve BAAI/bge-reranker-v2-m3 --served-model-name bge-reranker-v2-m3 --dtype auto --task score --host 0.0.0.0 --port 33435 --api-key vLLM

and RAGFlow OpenAI-API-Compatible :

embedding , bge-m3 , http://host.docker.internal:33434 , vLLM , 8192

rerank , bge-reranker-v2-m3 , http://host.docker.internal:33435 , vLLM, 8192

Kind regards, David.

Yes, it should work now because of my PR above :)
Check out the vLLM docs here too @yukon12345 I forgot to share earlier: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#re-rank-api

[dromeuf]

Yes, it should work now because of my PR above :) Check out the vLLM docs here too @yukon12345 I forgot to share earlier: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#re-rank-api

thanks for this request. Do you use the -override-pooler-config parameters for all models ? or just when you have a doubt ?

--override-pooler-config '{"pooling_type": "MEAN"}'

[K-Mistele]

Yes, it should work now because of my PR above :) Check out the vLLM docs here too @yukon12345 I forgot to share earlier: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#re-rank-api

thanks for this request. Do you use the -override-pooler-config parameters for all models ? or just when you have a doubt ?

--override-pooler-config '{"pooling_type": "MEAN"}'

I've never used this; it shouldn't be necessary.