Envoy Gateway Initial

[kerthcet]

What would you like to be added:

1. Integrate with envoy ai gateway https://github.com/envoyproxy/ai-gateway with advanced features with GenAI services.
2. Support to dispatch requests to corresponding inference services, for example, a request looks like

curl -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek-r3",
        "messages": [
            {
                "role": "system",
                "content": "Hi."
            }
        ]
    }' \
    $GATEWAY_URL/v1/chat/completions

The model name deepseek-r3 refers to the playground/inference service name, the request will route to the corresponding service(by default, we'll create a service named <sersvice>-lb routing to the real workloads).

See ai-gateway document for resources we need to create: https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/examples/basic/basic.yaml.

Note that, the api is following openAI schema: https://platform.openai.com/docs/api-reference/chat/create

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kerthcet]

/milestone v0.2.0
/assign @pacoxu

[pacoxu]

/assign

[pacoxu]

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: envoy-ai-gateway-basic
  namespace: default
spec:
  schema:
    name: OpenAI
  targetRefs:
    - name: envoy-ai-gateway-basic
      kind: Gateway
      group: gateway.networking.k8s.io
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: gpt-4o-mini
      backendRefs:
        - name: envoy-ai-gateway-basic-openai
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: us.meta.llama3-2-1b-instruct-v1:0
      backendRefs:
        - name: envoy-ai-gateway-basic-aws
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: some-cool-self-hosted-model
      backendRefs:
        - name: envoy-ai-gateway-basic-testupstream
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: qwen2-0.5b
      backendRefs:
        - name: envoy-ai-gateway-llmaz-model-1
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: envoy-ai-gateway-llmaz-model-1
  namespace: default
spec:
  schema:
    name: OpenAI
  backendRef:
    name: qwen2-0--5b-lb
    kind: Service
    port: 8080

I added something like above.

[root@paco ~]# curl -H "Content-Type: application/json"     -d '{
        "model": "qwen2-0.5b",
        "messages": [
            {
                "role": "system",
                "content": "Hi."
            }
        ]
    }'     $GATEWAY_URL/v1/chat/completions  | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   286  100   121  100   165  11000  15000 --:--:-- --:--:-- --:--:-- 26000
{
  "error": {
    "message": "model \"qwen2-0.5b\" not found, try pulling it first",
    "type": "api_error",
    "param": null,
    "code": null
  }
}

This is caused by that llmaz default use /v1/completions while envoy ai gateway use openAI /v1/chat/completions.

[root@paco ~]# curl http://localhost:18080/v1/completions -H "Content-Type: application/json" -d '{
    "model": "qwen2:0.5b",
    "prompt": "San Francisco is a",
    "max_tokens": 10,
    "temperature": 0
}' | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   406  100   297  100   109     84     31  0:00:03  0:00:03 --:--:--   116
{
  "id": "cmpl-577",
  "object": "text_completion",
  "created": 1744878906,
  "model": "qwen2:0.5b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "text": "San Francisco, the capital of the United States,",
      "index": 0,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 10,
    "total_tokens": 22
  }
}

Other info

[root@paco ~]# curl  $GATEWAY_URL/v1/models | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   419  100   419    0     0  59857      0 --:--:-- --:--:-- --:--:-- 59857
{
  "data": [
    {
      "id": "gpt-4o-mini",
      "created": 1744880950,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "us.meta.llama3-2-1b-instruct-v1:0",
      "created": 1744880950,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "some-cool-self-hosted-model",
      "created": 1744880950,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "qwen2-0.5b",
      "created": 1744880950,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    }
  ],
  "object": "list"
}

[pacoxu]

https://github.com/envoyproxy/ai-gateway/blob/616f37a6d582c0457b4f34477d464330b18be484/cmd/extproc/mainlib/main.go#L124-L125

[kerthcet]

This is caused by that llmaz default use /v1/completions while envoy ai gateway use openAI /v1/chat/completions

I think ollama doesn't support this, let's try with llama.cpp locally.

[kerthcet]

My configuration can't work, I created an issue here: envoyproxy/ai-gateway#579