Update InferencePool helm chart to use FailOpen as default

[RyanRosario]

What type of PR is this?
/kind cleanup
/kind documentation

What this PR does / why we need it:

Encourages use of FailOpen as the default failure mode in InferencePool in the helm chart

Which issue(s) this PR fixes:
Fixes #

Does this PR introduce a user-facing change?:

It sets the default failure mode to FailOpen in the helm chart and strongly encourages use of it as the default.

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	d0d5a5c
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/699e05a8724fe50008d39aa9
😎 Deploy Preview	https://deploy-preview-2365--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

Hi @RyanRosario. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[RyanRosario]

Tested with:

Test:

helm template test-pool config/charts/inferencepool --set inferencePool.modelServers.matchLabels.app=test helm template test-pool config/charts/inferencepool --set inferencePool.modelServers.matchLabels.app=test | grep failureMode -C 3

Proof that default is FailOpen:

helm template test-pool config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=test | grep failureMode -C 3

  endpointPickerRef:
    name: test-pool-epp
    port:
      number: 9002
    failureMode: FailOpen

Proof that users can still override:

helm template test-pool config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=test \
  --set inferenceExtension.failureMode=FailClose | grep failureMode -C 3


  endpointPickerRef:
    name: test-pool-epp
    port:
      number: 9002
    failureMode: FailClose

(also ran make vefify)

[RyanRosario]

@kfswain Ready for review

[RyanRosario]

Below is the verification of FailOpen now returning 200 whereas FailClose returns 503 when EPP is broken.

There were a few challenges here (order of operations):

1. Ordering conditions due to delays in the load balancer provisioning.
2. helm upgrade kept re-rendering the EPP deployment back to a healthy condition.

These are the steps I used to verify:

Setup

IGW_LATEST_RELEASE=$(curl -s https://api.github.com/repos/kubernetes-sigs/gateway-api-inference-extension/releases | jq -r '.[] | select(.prerelease == false) | .tag_name' | sort -V | tail -n1)
export IGW_CHART_VERSION=$IGW_LATEST_RELEASE
export GATEWAY_PROVIDER=gke
echo "Using IGW release: $IGW_LATEST_RELEASE"

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/${IGW_LATEST_RELEASE}/config/manifests/vllm/sim-deployment.yaml

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${IGW_LATEST_RELEASE}/manifests.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/${IGW_LATEST_RELEASE}/config/manifests/gateway/gke/gateway.yaml

Additional Experimental Setup

./toggle_failure_mode.sh Close

helm template vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION | grep -i fail

helm install vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION

(Apply HealthCheckPolicy so that failures aren't just because of missing health checks)
cat <<EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: inference-healthcheck
spec:
  default:
    config:
      httpHealthCheck:
        portSpecification: USE_SERVING_PORT
        requestPath: /v1/models
      type: HTTP
  targetRef:
    group: inference.networking.k8s.io
    kind: InferencePool
    name: vllm-qwen3-32b
EOF

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/${IGW_LATEST_RELEASE}/config/manifests/inferenceobjective.yaml


IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
PORT=80
echo "Gateway: ${IP}:${PORT}"

Good EPP, FailClose (already set previously and old default)

**Expectation: ** 200 OK

./toggle_failure_mode.sh Close

helm template vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION | grep -i fail

kubectl get inferencepool vllm-qwen3-32b -o yaml | grep -i fail

curl -i ${IP}:${PORT}/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "food-review-1",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0
  }'

Received HTTP/1.1 200 OK eventually

Good EPP, FailOpen (new default)

**Expectation: ** 200 OK

./toggle_failure_mode.sh Open

helm template vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION | grep -i fail

helm upgrade vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION

kubectl get inferencepool vllm-qwen3-32b -o yaml | grep -i fail

curl -i ${IP}:${PORT}/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "food-review-1",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0
  }'

Received HTTP/1.1 200 OK.

Break the EPP

kubectl set image deploy/vllm-qwen3-32b-epp epp=ghcr.io/does-not-exist/fake-epp:v0.0.0

Bad EPP, Fail Open

Expectation: Something other than 500/503

./toggle_failure_mode.sh Open

curl -i ${IP}:${PORT}/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "food-review-1",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0
  }'

Received HTTP/1.1 200 OK.

Bad EPP, FailClose (and break EPP again)

Expectation: Failure (500 or 503)

./toggle_failure_mode.sh Close

helm template vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION | grep -i fail

helm upgrade vllm-qwen3-32b \
  config/charts/inferencepool \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=$GATEWAY_PROVIDER \
  --set experimentalHttpRoute.enabled=true \
  --version $IGW_CHART_VERSION

kubectl get inferencepool vllm-qwen3-32b -o yaml | grep -i fail

kubectl set image deploy/vllm-qwen3-32b-epp epp=ghcr.io/does-not-exist/fake-epp:v0.0.0

curl -i ${IP}:${PORT}/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "food-review-1",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0
  }'

Received HTTP/1.1 503 Service Unavailable.

Ancillary Test

I was concerned that these tests do not load the defaults that I specified; however, the only changes I made to the files was in the Helm chart itself and each time Helm picked up the new FailureMode.

Ancillary Script

Purpose: Quickly switch between FailOpen and FailClose

#!/bin/bash
set -e

# Check argument
if [ -z "$1" ]; then
  echo "Usage: $0 [Open|Close]"
  exit 1
fi

MODE=$(echo "$1" | tr '[:lower:]' '[:upper:]')

if [ "$MODE" == "OPEN" ]; then
    TARGET="FailOpen"
    SOURCE="FailClose"
elif [ "$MODE" == "CLOSE" ]; then
    TARGET="FailClose"
    SOURCE="FailOpen"
else
    echo "Invalid argument. Use 'Open' or 'Close'."
    exit 1
fi

VALUES_FILE="config/charts/inferencepool/values.yaml"
TEMPLATE_FILE="config/charts/inferencepool/templates/inferencepool.yaml"

# Replace SOURCE with TARGET in the files
# This ensures that if we want Open, any FailClose becomes FailOpen.
# If we want Close, any FailOpen becomes FailClose.

echo "Setting global failureMode to $TARGET..."

sed -i "s/$SOURCE/$TARGET/g" "$VALUES_FILE"
sed -i "s/$SOURCE/$TARGET/g" "$TEMPLATE_FILE"

echo "Updated $VALUES_FILE and $TEMPLATE_FILE to use $TARGET"

Additionally, the timeouts in httproute.yaml needs to be commented out as apparently it's experimental. I chose not to commit this as it may break things and it was only used for this experiment.

timeouts:
    request: 300s

[ahg-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, RyanRosario

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment