fix infinite loop in profile picker and switch predictor based routing to on by default with a header to disable

[BenjaminBraunDev]

Small bugfix for latency routing profile picker (it wasn't returning an empty profile map when sending a request with predictor routing header set to false and the profiles all run).

This PR both corrects that logic and also turns predictor based routing to on by default (when deploying with latency based routing flag in helm charts) and replaces the old flag with a flag that turns it off when included instead.

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	ad7455a
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69335154a2cec000082b81a3
😎 Deploy Preview	https://deploy-preview-1929--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[shmuelk]

If a Boolean header is not found, why should the default value be true?

Please revert this change.

[BenjaminBraunDev]

@kfswain @ahg-g @kaushikmitr and I discussed this and since this is all behind the helm deployment flag to begin with, it makes the most sense for this header to specifically be used to disable the predictive routing, it should be on by default.

[BenjaminBraunDev]

Updated header to a clear "off" flag and updated documentation.

[shmuelk]

In general, the header utility functions should be removed from both the profile modified in this PR and in SLO Aware plugin package. They should be added as functions on the LLMRequest struct.

[BenjaminBraunDev]

In general, the header utility functions should be removed from both the profile modified in this PR and in SLO Aware plugin package. They should be added as functions on the LLMRequest struct.

This was mainly done to keep all SLO routing change isolated to its associated plugins and modify as little non-plugin code as possible.

[BenjaminBraunDev]

@ahg-g Could you help get this PR approved? It's part of what's necessary for vertex to do AB testing with latency prediction.

@kaushikmitr has a PR out for getting approval for the plugin package, but this affects a few files outside the package.

[ahg-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, BenjaminBraunDev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[shmuelk]

I don't understand why there is a header to disable the SLO based scheduling. Just don't use the plugin in the configuration.

Also the header name is strange. x-prediction-based-scheduling-off seems strange. It should have been x-disable-prediction-based-scheduling

[ahg-g]

We should remove the header when we graduate this plugin, for the most part it is to allow A/B testing and quickly turning it off if things are not working as expected.

[shmuelk]

We should remove the header when we graduate this plugin, for the most part it is to allow A/B testing and quickly turning it off if things are not working as expected.

A better way would be to add a profile handler that picks profiles based on a header. It would solve your A/B testing issue here as well as any other similar tests in a much more general fashion.