Support for configurable tls in remote jwks store

[dmitri-d]

Description

This PR depends on changes in #13011. The PR introduces support for setting of tls options for connections to remote jwks sources in jwks store. This is accomplished by setting a backendRef on a AgentgatewayPolicy, finding a matching BackendTLSPolicy, and using it to configure tls options.

If no BackendTLSPolicy is present, an http client with default tls options is used (including setting InsecureSkipVerify to false).

Implemenation Notes

• Only k8s service and static agent backends can be pointed to in backendRefs. We should probably validate that jwksUri and the service/backend are consistent.
• Limited set of tls options can be configured:

• system or custom CA certs to use
• enabling/disabling InsecureSkipVerify. I don't think we support this anywhere else, could be confusing.

Change Type

/kind feature

Changelog

Support setting of tls options in connections to remote jwks sources.

Additional Notes

Example of a policy using remote jwks, configured via static backend:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: route-policy
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: route-secure
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
      - issuer: https://kgateway.dev
        jwks:
          remote:
            jwksPath: org-one/keys
            backendRef:
              group: agentgateway.dev
              kind: AgentgatewayBackend
              name: dummy-idp-1
              port: 8443
      - issuer: https://kgateway.dev
        jwks:
          remote:
            jwksPath: org-two/keys
            backendRef:
              group: agentgateway.dev
              kind: AgentgatewayBackend
              name: dummy-idp
              port: 8443
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: dummy-idp
spec:
  policies:
    tls:
      insecureSkipVerify: All
  static:
    host: dummy-idp.default
    port: 8443

Example of a policy using remote jwks, configured via k8s service:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: route-policy
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: route-secure
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
      - issuer: https://kgateway.dev
        jwks:
          remote:
            jwksPath: org-one/keys
            backendRef:
              group: ""
              kind: Service
              name: dummy-idp
              port: 8443
---
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: idp-policy
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: dummy-idp
  backend:
    tls:
      caCertificateRefs:
      - name: ca
---
apiVersion: v1
kind: Service
metadata:
  name: dummy-idp
  namespace: default
  labels:
    app: dummy-idp
spec:
  ports:
    - name: http
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: dummy-idp

[Copilot]

Pull request overview

This PR introduces support for configurable TLS options when connecting to remote JWKS sources in the JWKS store. It allows users to specify custom TLS configurations via BackendTLSPolicy by setting a backendRef on an AgentgatewayPolicy.

Key changes include:

• Refactored JWKS store architecture to use event-driven updates via channels instead of queue-based polling
• Added support for custom TLS configurations per JWKS source through BackendTLSPolicy integration
• Separated concerns into dedicated controllers: policy controller for JWKS source changes and ConfigMap controller for persistence
• Improved security by removing default InsecureSkipVerify: true from HTTP clients

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
internal/kgateway/jwks/jwks_store.go	Refactored to use channel-based updates; added mapping between ConfigMap names and JWKS URIs
internal/kgateway/jwks/jwks_fetcher.go	Added TLS configuration support per keyset; simplified keyset management API; removed default insecure TLS
internal/kgateway/jwks/jwks_cache.go	Added thread-safe methods with mutex locks; removed comparison logic from add operation
internal/kgateway/jwks/config_map_syncer.go	Simplified by removing write logic and KRT collection; now only handles loading ConfigMaps
internal/kgateway/agentjwksstore/policy_controller.go	New controller that watches policies and creates JWKS sources with TLS configs from BackendTLSPolicy
internal/kgateway/agentjwksstore/cm_controller.go	New controller that persists JWKS to ConfigMaps using event queue pattern
internal/kgateway/controller/start.go	Updated to initialize new controllers in correct order
internal/kgateway/jwks/jwks_fetcher_test.go	Updated tests to use new defaultJwksClient field name
api/v1alpha1/agentgateway_policy_types.go	Made BackendRef a pointer and JwksUri required; removed mutual exclusivity validation
install/helm/kgateway-crds/templates/gateway.kgateway.dev_agentgatewaypolicies.yaml	Updated CRD to reflect API changes
api/v1alpha1/zz_generated.deepcopy.go	Generated code for pointer-based BackendRef deep copy

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

[lgadban]

nit: would be nice to have a trivial example here on the format, e.g. /path/here.

we could also throw validation to e.g. enforce : is not included, etc.

[lgadban]

nit: we can add some color here explaining that the backendRef determines where and how to establish a connection and allows policy to be attached etc.

[dmitri-d]

will do

[lgadban]

just curious, do we not have an index like this elsewhere in the agw codebase?

[dmitri-d]

Not that I could find.

[lgadban]

this TODO is a bit confusing, is this saying we may want to support AgwPolicy being attached to the Backend being referenced?

If so I think we would want to do that in the future to have a consistent approach.

[dmitri-d]

I'll remove the TODO, that's been implemented.

[lgadban]

// no port, as policy targetRef may not have it

is there an implication here?
i.e. can AgwPolicy not sectionName specific ports?

[dmitri-d]

an issue here is that sectionName is optional and we don't know apriori if it's present in the policy's targetRef; so we either ignore it completely (an issue if there are multiple policies targeting the same service but different ports), or do multiple searches (with the port set first, then without the port).

[lgadban]

// check if a

is this just leftover cruft?

[dmitri-d]

indeed it is.

[apexlnc]

Feedback on required backendRef for remote JWKS

Following up on the discussion about keeping backendRef required - wanted to share our experience with this change.

Use case: JWT authentication with public IdP JWKS endpoints (GitHub Actions, GitLab CI)

Before:

jwks:
  remote:
    uri: https://token.actions.githubusercontent.com/.well-known/jwks
    cacheDuration: 30m

After:

# Need to create a Backend for a public HTTPS endpoint
backends:
  github-jwks:
    static:
      host: token.actions.githubusercontent.com
      port: 443
    policies:
      tls:
        sni: token.actions.githubusercontent.com

# Then reference it
jwks:
  remote:
    jwksPath: .well-known/jwks
    backendRef:
      name: github-jwks
      group: agentgateway.dev
      kind: AgentgatewayBackend
    cacheDuration: 30m

For public HTTPS endpoints with standard TLS (common for IdPs like GitHub, GitLab, Auth0, Okta, etc.), the Backend object adds a bit of extra configuration when custom TLS settings aren't needed.

Thought: Would it be possible to make backendRef optional and fall back to the previous uri behavior when not specified? That would keep things simple for common cases while still enabling custom TLS configuration when needed.

Thanks for considering!

[lgadban]

@apexlnc per your comment above @dmitri-d opened #13186

As described in the issue there may be larger work necessary to get this working in a consistent manner across the various agentgateway policies.