Extra minimal OCG API #3952

kflynn · 2025-07-23T21:34:07Z

This is a GEP for an extra minimal OCG API, intended not to be production-ready but to permit experimentation.

/kind gep

NONE

karthikbox · 2025-07-23T23:02:55Z

geps/gep-3951/index.md

+- The trust bundle
+  in the Gateway resource
+  will define the CA certificate(s)
+  that the OCG
+  should accept as trusted
+  when validating connections
+  from meshed peers.
+
+- The trust bundle
+  in the Mesh resource
+  will define the CA certificate(s)
+  that the mesh
+  should accept as trusted
+  when validating connections
+  from the OCG.


can we add a bit more info around why we chose to have GW trust bundle and mesh and the Mesh trust bundle in the GW resource?

while i agree that this model to be simpler, but IMO it's worth mentioning the alternatives.
i added some info around this in https://github.com/kubernetes-sigs/gateway-api/pull/3941/files#diff-4a7d8011b2ad7222ce2d13ee98f49443d6eb56518625438daa62c10e94d9f772R279-R489
specifically proposals 1 and 2.

with the proposed model, is the mesh trust bundle duplicated every single Gateway resource?
(is there a common gateway config somewhere?)

The proposed model is that each Gateway gets its own trust bundle. We may want to consider having a default in the GatewayClass, but this is the extra minimal API so it's not there yet.

And I agree about alternatives -- I had it in my head that in many cases I should move them into GEP-3792 itself, but thinking about it while the sun is up, that seems silly. 🙂 Will update.

karthikbox · 2025-07-23T23:11:23Z

geps/gep-3951/index.md

+    trustBundle:
+      name: mesh-trust-bundle
+      namespace: mesh-namespace
+      # Key in Configmap; defaults to "ca-bundle.crt"
+      bundleKey: ca-bundle.crt


i personally think clusterTrustBundle is a good fit for this. can we mention that we intend to support clusterTrustBundle in the future? or do u see some fundamental problem with it?

+1, I'd much rather start with ClusterTrustBundle as the recommendation where available, and ConfigMap as an optional backfill where it's not. I know that's not great now, but by the time this API is stable/GA, I'm guessing ClusterTrustBundle will be much more widely available.

karthikbox · 2025-07-23T23:16:19Z

geps/gep-3951/index.md

+      matchLabels:
+        mesh: one-mesh-to-mesh-them-all


just to confirm - is this selecting the Mesh resource or the Routes or Namespaces?
i think it's the latter but the one-mesh-to-mesh-them-all is confusing me because u have that set on the Mesh resource in the Mesh GEP.

also, can u consider calling out some alternative mechanisms?
https://github.com/kubernetes-sigs/gateway-api/pull/3941/files#diff-4a7d8011b2ad7222ce2d13ee98f49443d6eb56518625438daa62c10e94d9f772R638-R730.
doesn't have to be this but i think it's useful to document the alternatives.

If lines 142-144 are still the state, it looks like this selects Routes. Left another comment on those lines. And also we had a different comment thread on the original PR.

My strong preference is to go with namespace, and potentially provide opt-out for services.

karthikbox · 2025-07-23T23:21:05Z

geps/gep-3951/index.md

+The extra-minimal API
+solves this problem
+by adding a label selector
+to the Gateway resource
+that indicates which Routes
+are meshed.
+When the OCG connects
+to any Route
+that either directly matches this selector,
+or is in a namespace that matches this selector,
+it MUST use mTLS
+with a certificate
+that is ultimately signed
+by a CA certificate
+in the Mesh resource's `trustBundle`,
+and the OCG MUST validate
+that the peer presents a certificate
+that is ultimately signed
+by a CA certificate
+in the Gateway resource's `trustBundle`.


couple of open questions (copied from my #3941) :

How to configure the Gateway's identity certificate and private key?

This GEP defines how the OCG and the mesh should be configured to trust each
other by exchanging CA bundles. However, it does not standardize how an
administrator configures the specific client certificate and private key that
the OCG uses to identify itself to the mesh.

Currently, this is left as an implementation detail, likely handled via a
provider-specific CRD referenced from the GatewayClass or through an out-of-band
mechanism. The open question is: Should a future version of this GEP standardize
this configuration to ensure a consistent user experience? This could involve
adding a new identityCertificateRef field to the Gateway spec.

Should use cases where mesh workloads disable mTLS be supported?

This GEP focuses on meshes where mTLS is strictly enforced for
communication. However, some service meshes support a "DISABLE" mode where mTLS
can be disabled for certain workloads. This raises the question: How should an
OCG behave when a target workload is discovered as "meshed" but does not require
or accept an mTLS connection?

Should a future version of this GEP standardize this configuration to ensure a consistent user experience?

I'm not certain there's much value in standardizing the inner mechanism here as long as the high-level interface is sufficient for implementations to configure mTLS.

How should an OCG behave when a target workload is discovered as "meshed" but does not require
or accept an mTLS connection?

This might have some overlap with #3876

k8s-ci-robot · 2025-07-25T21:26:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kflynn
Once this PR has been reviewed and has the lgtm label, please assign danwinship for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Flynn <[email protected]>

k8s-ci-robot · 2025-07-26T14:08:44Z

@kflynn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gateway-api-verify	`d17ffa6`	link	true	`/test pull-gateway-api-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

LiorLieberman · 2025-08-06T20:53:39Z

geps/gep-3951/index.md

+[GEP-3792] defines the rationale
+for allowing out-of-cluster Gateways (OCGs)
+to participate in a
+GAMMA-compliant in-cluster service mesh,
+and the problems that must be solved
+to allow them to do so.
+This GEP defines
+an extremely minimal API
+to permit experimentation
+with OCGs and
+in-cluster mTLS meshes.


This is odd formatting? is this a new formatter or something?

Refs discussion at #3950 (comment), I personally find it harder to read even if it does make edit diffs more clear.

LiorLieberman · 2025-08-06T21:05:30Z

geps/gep-3951/index.md

+- a `labelSelector` field
+  that indicates which Routes
+  are meshed.


I think we had similar comment thread on the original PR. #3894 (comment)

I am unclear why we need to select routes, namespace seems like it would cover 90% of the cases, and we can opt-in OR opt-out services.

LiorLieberman · 2025-08-06T21:06:56Z

geps/gep-3951/index.md

+      matchLabels:
+        mesh: one-mesh-to-mesh-them-all


If lines 142-144 are still the state, it looks like this selects Routes. Left another comment on those lines. And also we had a different comment thread on the original PR.

My strong preference is to go with namespace, and potentially provide opt-out for services.

k8s-ci-robot · 2025-09-17T18:19:47Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mikemorris

Finally got around to reading through this, left a few high-level notes.

mikemorris · 2025-10-13T20:04:52Z

geps/gep-3951/index.md

+...
+spec:
+  ...
+  ocg:


Suggested change

ocg:

trustFederation:

ocg is a dense acronym that would have zero familiarity to most users, would suggest spelling out a field name and using more familiar language. Additionally, naming this more generically might unlock additional use cases under the broader "trust domain federation" umbrella, such as transitioning a mesh over to a new root, migrating a mesh deployment to a new cluster, or extending trust to other off-cluster resources, such as "mesh expansion" for connectivity with VMs using their own PKI root (and I'm intentionally not even citing "mesh federation" yet).

https://istio.io/latest/docs/tasks/security/authorization/authz-td-migration/

https://spiffe.io/docs/latest/architecture/federation/readme/

mikemorris · 2025-10-13T20:12:43Z

geps/gep-3951/index.md

+#### Additions to the Gateway Resource
+
+The Gateway resource
+gains a `mesh` stanza


This again feels too narrowly-scoped, and likely more controversial than splitting the trust federation capability into some way to specify a bundle under https://gateway-api.sigs.k8s.io/reference/spec/#gatewaytlsconfig, and handling the mesh scoping entirely within the mesh resource through something like a namespace selector as suggested by @LiorLieberman (with some consideration for how to handle exclusion, likely via matchExpressions).

(Also the Gateway trust bundle scope feels relatively duplicative of #91?)

mikemorris · 2025-10-13T20:21:31Z

geps/gep-3951/index.md

+The `namespace` field is required
+in the Mesh resource,
+but may be omitted
+in the Gateway resource


Probably better to model these as two different Go types similar to the existing WithNamespace patterns to avoid UX confusion.

mikemorris · 2025-10-13T20:27:05Z

geps/gep-3951/index.md

+In practice, this isn't
+actually a question of _workloads_
+but of _Routes_:


In simple cases this holds, but I think it's actually inaccurate. A single route could absolutely split traffic between a "meshed" backend in one namespace and an "unmeshed" backend in another namespace.

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/gep PRs related to Gateway Enhancement Proposal(GEP) cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 23, 2025

k8s-ci-robot requested review from gcs278 and mikemorris July 23, 2025 21:34

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 23, 2025

karthikbox reviewed Jul 23, 2025

View reviewed changes

kflynn mentioned this pull request Jul 25, 2025

Mesh resource definition #3950

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2025

kflynn added 3 commits July 26, 2025 09:53

Extra minimal OCG API

8f2b22c

Signed-off-by: Flynn <[email protected]>

Add RFC8174 definitions

503fed6

Signed-off-by: Flynn <[email protected]>

mkdocs entry

d17ffa6

Signed-off-by: Flynn <[email protected]>

kflynn force-pushed the flynn/gep-3951 branch from 23bd508 to d17ffa6 Compare July 26, 2025 13:54

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2025

LiorLieberman reviewed Aug 6, 2025

View reviewed changes

LiorLieberman mentioned this pull request Aug 6, 2025

API for Off-Cluster Gateways (superseeds #3894) #3941

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2025

mikemorris mentioned this pull request Oct 13, 2025

Move GEP-3792 to Memorandum with API in other GEPs #3894

Open

mikemorris reviewed Oct 13, 2025

View reviewed changes

Extra minimal OCG API #3952

Are you sure you want to change the base?

Extra minimal OCG API #3952

Conversation

kflynn commented Jul 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robscott Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 25, 2025

Uh oh!

k8s-ci-robot commented Jul 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Sep 17, 2025

Uh oh!

mikemorris left a comment

Choose a reason for hiding this comment

Uh oh!

mikemorris Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

robscott Jul 24, 2025 •

edited

Loading

mikemorris Oct 13, 2025 •

edited

Loading