Add envoy ai gateway

[pacoxu]

What this PR does / why we need it

Action Item:

1. support envoy gateway and envoy ai gateway installation
2. for playground change, update AIGatewayRoute/envoy-ai-gateway-basic and update AIServiceBackend/playgroundName.

Which issue(s) this PR fixes

Fixes #339

Special notes for your reviewer

Does this PR introduce a user-facing change?

[googs1025]

Do we necessarily have to use hardcoding? in name namespace🤔

[kerthcet]

In the future, we should combine the gateway configuration with Playground for easy start. However, right now, envoy ai gateway is still alpha, let's have the users to configure the envoy configurations themselves. What I mean here is we want't have any envoy configurations in our code base, what we need is just a Documentation about how to use the AI gateway. WDYT?

The consideration here is because envoy has a lot of configurations, like the weight, token limits, if we just hard code them, it doesn't make any sense, all the stuff should be exported one day, as I mentioned, with Playground, but not today.

[kerthcet]

List my example here, however, not work.

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: qwen2-0--5b
spec:
  familyName: qwen2
  source:
    modelHub:
      modelID: Qwen/Qwen2-0.5B-Instruct-GGUF
      filename: qwen2-0_5b-instruct-q5_k_m.gguf
---
apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: qwen2-0--5b
spec:
  replicas: 1
  modelClaim:
    modelName: qwen2-0--5b
  backendRuntimeConfig:
    backendName: llamacpp
    configName: default
    args:
      - -fa # use flash attention
---
apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: qwen2--5-coder
spec:
  familyName: qwen2
  source:
    modelHub:
      modelID: Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF
      filename: qwen2.5-coder-0.5b-instruct-q2_k.gguf
---
apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: qwen2--5-coder
spec:
  replicas: 1
  modelClaim:
    modelName: qwen2--5-coder
  backendRuntimeConfig:
    backendName: llamacpp
    configName: default
    args:
      - -fa # use flash attention
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: default-envoy-ai-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default-envoy-ai-gateway
  namespace: default
spec:
  gatewayClassName: default-envoy-ai-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: default-envoy-ai-gateway
  namespace: default
spec:
  schema:
    name: OpenAI
  targetRefs:
    - name: default-envoy-ai-gateway
      kind: Gateway
      group: gateway.networking.k8s.io
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: qwen2-0--5b
      backendRefs:
        - name: qwen2-0--5b
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: qwen2--5-coder
      backendRefs:
        - name: qwen2--5-coder
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: qwen2-0--5b
  namespace: default
spec:
  schema:
    name: OpenAI
  backendRef:
    name: qwen2-0--5b-lb
    kind: Service
    port: 8080
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: qwen2--5-coder
  namespace: default
spec:
  schema:
    name: OpenAI
  backendRef:
    name: qwen2--5-coder-lb
    kind: Service
    port: 8080
---

[kerthcet]

It works ... I don't know why.

[kerthcet]

Once this is ready, I'll write a post to open-webui as kubernetes integration.

[kerthcet]

also another adopter for envoy ai gateway.

[pacoxu]

Almost ready for review:

• I need to do a manual test, as I do not write an e2e test.

This may be a todo item.

[pacoxu]

Without this, there will be a compiler error, IIRC.

[kerthcet]

Please submit with another PR. It's irrelevant.

[kerthcet]

Will --force-conflicts cause other problems? Should be careful here, prefer to keep the server-side only.

[kerthcet]

Let's not include these files in the helm chart I think, let's make them an example instead. Because most of them should be user-defined.

[kerthcet]

I don't think we can orchestrate the envoy ai gateway now, because we didn't expose any configurations from playground yet, for example, because the llm serving is usually very slow, we need to define the timeout in the AIServiceBackend, or most of the time, the request will be timeout.

So my suggestion is let's deploy these configurations manually, we'll provide an example for users to follow, only envoy-ai-gateway is mature, we'll add some fields to playground for quick integration, similar like we do here. So what we need is:

• an example
• helm dependence and enabled by default, if users want to disable the components, they should append the disable args after the install cmds.

[pacoxu]

Use #360 instead.

This PR tried to create/update the ai gateway resource according to current playground. However, we may add some extra attributes for playground or new resource later to do that.

/close