openai自定义协议配置多个模型动态转换没有生效 || openai custom protocol configuration multiple model dynamic conversion does not take effect #3189
Description
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via ASRC(Alibaba Security Response Center) where the issue will be triaged appropriately.
- I have searched the issues of this repository and believe that this is not a duplicate.
Ⅰ. Issue Description
配置openai自定义服务，目标模型没有被切换。
Ⅱ. Describe what happened
我们是自己在Azure上部署的openAI模型。在一个订阅下部署了3个模型，同样的模型，3个名字，希望通过higress的配置，能让用户的使用均衡的打到3个模型上，负载均衡。
因为Azure的一个订阅对外域名是同一个，因此我在【AI服务提供者管理】里面配了3个服务提供者，其baseUrl和apiKey都是一样的。
然后在【AI路由管理】里面配置了1个路由。里面【目标AI服务】分配加入了上面配置的3个服务提供者，并且每个服务提供者对应的模型都配置的不一样，对应我们自己部署的3个模型。
实际调用后发现，确实会均分打到3个服务提供者上（日志能看出来），但是模型没有被替换。
Ⅲ. Describe what you expected to happen
同样的逻辑，配置火山引擎和阿里百炼的平台上的模型就可以被自动切换。
但是配置openai协议自定义的服务，就不行。
端口是/openai/v1/responses
Ⅳ. How to reproduce it (as minimally and precisely as possible)
- azure开一个订阅
- 部署2个模型，名称不同
- 按照我上面的描述配置好
- 使用curl命令调用higress，会发现目标服务确实负载均衡，但是相应的模型没有被替换。
Ⅴ. Anything else we need to know?
It is recommended to provided Higress runtime logs and configurations for us to investigate your issue, especially for controller and gateway components.
Please checkout following documents on how to obtain these data.
Ⅵ. Environment:
- Higress version: 2.1.9
- OS: k8s
- Others:
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via ASRC(Alibaba Security Response Center) where the issue will be triaged appropriately.
- I have searched the issues of this repository and believe that this is not a duplicate.
Ⅰ. Issue Description
Configure openai custom service, the target model is not switched.
Ⅱ. Describe what happened
We deployed the openAI model on Azure ourselves. Three models are deployed under one subscription. The same model has three names. It is hoped that through the configuration of higress, the user's usage can be balanced to the three models and the load will be balanced.
Because the external domain name of a subscription in Azure is the same, I configured three service providers in [AI Service Provider Management], and their baseUrl and apiKey are the same.
Then configure a route in [AI Route Management]. The [Target AI Service] allocation includes the three service providers configured above, and the models corresponding to each service provider are configured differently, corresponding to the three models we deployed ourselves.
After the actual call, it was found that the calls were indeed distributed equally to three service providers (as can be seen in the log), but the model was not replaced.
Ⅲ. Describe what you expected to happen
With the same logic, models configured on the Huoshan Engine and Alibaba Bailian platforms can be automatically switched.
But configuring openai protocol-customized services does not work.
The port is /openai/v1/responses
Ⅳ. How to reproduce it (as minimally and precisely as possible)
- Open a subscription in azure
- Deploy 2 models with different names
- Configure it as I described above
- Use the curl command to call higress. You will find that the target service is indeed load balanced, but the corresponding model has not been replaced.
Ⅴ. Anything else we need to know?
It is recommended to provided Higress runtime logs and configurations for us to investigate your issue, especially for controller and gateway components.
Please checkout following documents on how to obtain these data.
Ⅵ. Environment:
- Higress version: 2.1.9
- OS: k8s
- Others: