URLRewrite to llm backend is not applying correctly

[jimmoore22]

kgateway version

v2.1.0-main

Kubernetes Version

v1.31.1

Describe the bug

URLRewrite is not working when using an LLM Backend type. I have tried this with both the gemini and openai providers but neither is working.

I am trying to connect to the gemini openai compatible API (https://generativelanguage.googleapis.com/v1beta/chat/completions) using the openai provider with an httproute providing the URL rewrite. I used both ReplacePrefixMatch and ReplaceFullPath but neither works.

When checking the envoy logs I can see from the headers that the path is being set correctly as per the httproute but it does not actually send to the correct URL.

backend looks as below

spec:
  ai:
    llm:
      hostOverride:
        host: generativelanguage.googleapis.com
        port: 443
      provider:
        openai:
          authToken:
            kind: SecretRef
            secretRef:
              name: google
          model: gemini-2.5-pro-preview-05-06
  type: AI

httproute looks as below

filters:
- type: URLRewrite
  urlRewrite:
    path:
      replaceFullPath: /v1beta/chat/completions
      type: ReplaceFullPath
matches:
- path:
    type: PathPrefix
    value: /openai

From the envoy logs I can see the following

':path', '/v1beta/chat/completions'                                                                                                                                                                                                               
'x-envoy-original-path', '/openai'

I can see the connection to the backend is fine

[2025-07-16 09:57:58.372][29][debug][connection] [external/envoy/source/common/network/connection_impl.cc:1092] [Tags: "ConnectionId":"13"] connecting to 74.125.193.95:443                                                                       │ 
[2025-07-16 09:57:58.372][29][debug][connection] [external/envoy/source/common/network/connection_impl.cc:1111] [Tags: "ConnectionId":"13"] connection in progress                                                                                │ 
[2025-07-16 09:57:58.374][29][debug][connection] [external/envoy/source/common/network/connection_impl.cc:810] [Tags: "ConnectionId":"13"] connected

But I am getting a 404 response code.

If I remove the hostOverride and point at the base openAI backend it works fine. Even if I change the urlRewrite rule to be /banana it still works.

This looks like the urlRewrite is not being honoured and as such I am unable to connect to the gemini openai compatible backend as the path to that is /v1beta/chat/completions as opposed to the standard openai path of /v1/chat/completions.

I have also tried to use pathOverride in the backend configuration but that gives the same result.

Expected Behavior

The gateway should honour the urlRewrite path that we see in the envoy logs.

Steps to reproduce the bug

Create a backend

spec:
  ai:
    llm:
      provider:
        openai:
          authToken:
            kind: SecretRef
            secretRef:
              name: openai
          model: gpt-4.1-mini
  type: AI

Create a httproute with the URLRewrite

    - type: URLRewrite
      urlRewrite:
        path:
          replaceFullPath: /banana
          type: ReplaceFullPath
    matches:
    - path:
        type: PathPrefix
        value: /openai

Curl to the gateway listener on /openai

curl -vi "http://<gateway>:8080/openai" -H "Content-Type: application/json" -d '{
    "model": "gpt-4o-mini",
    "messages": [
        {"role": "user", "content": "Explain to me how AI works"}
    ]
    }'

If the URLRewrite is being honoured you should get a 404 due to /banana not existing at the destination. If the query works then URLRewrite is not being honoured.

Additional Environment Detail

No response

Additional Context

No response

[timflannagan]

FYI @npolshakova

[npolshakova]

Hi @jimmoore22 👋 So urlRewrite works a little different for the AI backend types because the request goes through the ai extensions extproc server. This should be fixed in the latest kgateway build by #11282.

You should be able to use the pathOverride like so:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  labels:
    app: kgateway
  name: openai-override
  namespace: gwtest
spec:
  type: AI
  ai:
    llm:
      hostOverride:
        host: test-ai-provider.ai-test.svc.cluster.local
        port: 443
      pathOverride:
        fullPath: "/api/v1/chat/completions"
      authHeaderOverride:
        prefix: "custom-prefix"
        headerName: "custom-header"
      provider:
        openai:
          authToken:
            kind: "SecretRef"
            secretRef:
              name: openai-override-secret

[jimmoore22]

Thank you for the clarification @npolshakova

When I update my config to match

spec:
  ai:
    llm:
      hostOverride:
        host: generativelanguage.googleapis.com
        port: 443
      pathOverride:
        fullPath: "/v1beta/chat/completions"

I get a 404 from the backend. If I point the backend at an nginx proxy I have deployed locally doing a URLRewrite to /v1beta/chat/completions it is working fine.

Am I misusing the pathOverride?

Thanks,

Jim

[npolshakova]

Hi Jim, are you using openai for the provider type? If you're using a host override, you may need to set https://github.com/kgateway-dev/kgateway/blob/main/api/v1alpha1/backend_types.go#L234, since the path is built like this:

kgateway/internal/kgateway/extensions2/plugins/backend/ai/ai_model_cluster.go

Line 490 in 3696922

if llm.PathOverride != nil {

[jimmoore22]

Hi,

Still get the same 404 error when setting insecureSkipVerify: true

I am using the openai provider type as below:

spec:
  ai:
    llm:
      hostOverride:
        host: generativelanguage.googleapis.com
        insecureSkipVerify: true
        port: 443
      pathOverride:
        fullPath: /v1beta/chat/completions
      provider:
        openai:
          authToken:
            kind: SecretRef
            secretRef:
              name: secret
          model: gemini-2.5-pro-preview-05-06
  type: AI