update bbr quickstart guide with latest functionality

[nirrozenbaum]

What type of PR is this?
/kind documentation

What this PR does / why we need it:
update serving multiple inference pool quickstart guide when using the code from main.

Which issue(s) this PR fixes:

This PR completes the work on multi inference pool management through bbr.
Fixes #1812

Does this PR introduce a user-facing change?:

None

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	6d6e6af
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6967efa3fef939000842265b
😎 Deploy Preview	https://deploy-preview-2150--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[nirrozenbaum]

/hold self verifying before opening for review.

[nirrozenbaum]

/unhold

[nirrozenbaum]

cc: @howardjohn please pay attention that continuing our conversation on #1908 kgateway was temporarily removed, since it doesn't support bbr's new functionality. it would be great if we can add kgateway back with support for the new functionality.

[howardjohn]

Please don't remove our implementation. We can support the native and ext-proc approach, so there is no need for removal -- maybe tweaks.

I don't see any new functionality that cannot be done natively, IIUC. I believe the idea is there is now a mapping of adapter to model, instead of a simple body --> header.

This can be trivially done by the proxy as well with a minor tweak:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: bbr
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  traffic:
    phase: PreRouting
    transformation:
      request:
        set:
        - name: X-Gateway-Model-Name
          value: |
            {
              "food-review-1": "meta-llama/Llama-3.1-8B-Instruct",
              "meta-llama/Llama-3.1-8B-Instruct": "meta-llama/Llama-3.1-8B-Instruct",
            }[json(request.body).model]

[nirrozenbaum]

Please don't remove our implementation. We can support the native and ext-proc approach, so there is no need for removal -- maybe tweaks.

I don't see any new functionality that cannot be done natively, IIUC. I believe the idea is there is now a mapping of adapter to model, instead of a simple body --> header.

This can be trivially done by the proxy as well with a minor tweak:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: bbr
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  traffic:
    phase: PreRouting
    transformation:
      request:
        set:
        - name: X-Gateway-Model-Name
          value: |
            {
              "food-review-1": "meta-llama/Llama-3.1-8B-Instruct",
              "meta-llama/Llama-3.1-8B-Instruct": "meta-llama/Llama-3.1-8B-Instruct",
            }[json(request.body).model]

sounds good.
will make these additions and ping you for review on that section.
holding to prevent merging without kgateway.
/hold

[nirrozenbaum]

@howardjohn updated agentgatewaypolicy as you suggested.
please let me know if this looks ok.

/unhold

[howardjohn]

thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: howardjohn, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [nirrozenbaum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kfswain]

/lgtm