Azure OpenAI returns RST_STREAM when Envoy uses HTTP/2

[nutslove]

Summary

I'm running Envoy AI Gateway with multiple LLM backends (AWS Bedrock, GCP Vertex AI, Azure OpenAI). Bedrock and Vertex AI work fine, but every request to Azure OpenAI fails with protocol error (502). After investigating, it appears Envoy negotiates HTTP/2 via TLS ALPN with Azure, but Azure immediately responds with RST_STREAM.

I found a workaround using EnvoyPatchPolicy to force HTTP/1.1, but I'd like to understand:

1. Why does this happen only with Azure OpenAI?
2. Is forcing HTTP/1.1 via EnvoyPatchPolicy the recommended approach, or is there a better way?
3. Are there any plans / existing options I might have missed?

Any insight from the maintainers or the community would be greatly appreciated 🙏

Environment

• Envoy AI Gateway: v0.5.0
• Envoy Gateway: v1.7.2
• Platform: AWS EKS
• Backend: Azure OpenAI (*.openai.azure.com, Japan East region)
• Auth: Entra ID Service Principal (client credentials flow)

Other backends on the same gateway work without any issue:

• AWS Bedrock ✅
• GCP Vertex AI ✅
• Azure OpenAI ❌

What I observed

Error response

{"type":"error","error":{"type":"OpenAIBackendError","code":"502","message":"upstream connect error or disconnect/reset before headers. reset reason: protocol error"}}

Envoy access log

{
  "response_code": 502,
  "response_code_details": "upstream_reset_before_response_started{protocol_error}",
  "response_flags": "UPE",
  "upstream_cluster": "httproute/.../rule/N",
  "upstream_host": "<azure-ip>:443"
}

Envoy admin stats for the cluster

upstream_cx_http2_total: 2
upstream_cx_http1_total: 0
http2.rx_reset: 2
upstream_cx_destroy_remote: 2
upstream_rq_tx_reset: 2
ssl.handshake: 2
ssl.versions.TLSv1.2: 2
upstream_cx_protocol_error: 0

My interpretation: TLS handshake succeeds, HTTP/2 connection is established, but every stream is RST'd by Azure right after the request is sent. Please correct me if I'm reading this wrong.

Direct curl from the same network

To rule out network or auth issues, I tried calling Azure OpenAI directly from the same network as the Envoy Pod:

# HTTP/1.1 — 200 OK
curl --http1.1 https://<resource>.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview \
  -H "api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hi"}]}'

# HTTP/2 — also 200 OK
curl --http2 https://<resource>.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview \
  -H "api-key: $KEY" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hi"}]}'

Both succeed. So the issue seems specific to Envoy's HTTP/2 framing against Azure OpenAI's frontend, not Azure OpenAI itself rejecting all HTTP/2.

My current workaround

I applied EnvoyPatchPolicy to force HTTP/1.1 on the Azure cluster:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: azure-http1
  namespace: envoy-gateway-system
spec:
  type: JSONPatch
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: <gateway-name>
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
      name: httproute/<ns>/<route>/rule/<N>
      operation:
        op: add
        path: "/typed_extension_protocol_options"
        value:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
            explicit_http_config:
              http_protocol_options: {}

After applying:

• upstream_cx_http1_total increases ✅
• upstream_rq_total succeeds with 200 ✅

It works, but it feels fragile because:

• Cluster name httproute/.../rule/N depends on AIGatewayRoute rule order
• Adding/removing/reordering rules shifts the N, silently breaking the patch
• Each new Azure model deployment requires another patch entry

Questions

1. Root cause: Is this a known issue with Azure OpenAI's HTTP/2 frontend, or with Envoy's HTTP/2 implementation, or some interaction between them? curl with --http2 works fine, so I'm puzzled why Envoy specifically triggers RST_STREAM.

2. Recommended approach: Is EnvoyPatchPolicy the recommended way to force HTTP/1.1, or is there a cleaner method I'm missing? For example, is there an existing field in Backend, BackendTLSPolicy, or BackendSecurityPolicy that I overlooked?

3. Long-term: Are there plans to support specifying upstream HTTP version natively in Envoy AI Gateway / Envoy Gateway, especially for known-problematic backends like Azure OpenAI?

4. Other reports: Has anyone else hit this with Azure OpenAI? Wondering if this is environment-specific (region, deployment type, etc.) or universal.

Thanks in advance for any guidance!

[ehrnst]

this sounds like a similar thing we suddenly encountered from one day to another. However our issue was connection reset before headers 503
We had to add useClientProtocol: true to our backendTrafficPolicy. Might not be related, but we were running for a couple of months in production before we got issues.
https://gateway.envoyproxy.io/docs/api/extension_types/