Define plugin interface and consumer library #17

qiujian16 · 2025-05-21T08:24:32Z

No description provided.

RainbowMango · 2025-05-21T08:33:12Z

Where can I find the background of this?

qiujian16 · 2025-05-21T08:51:39Z

see the thread here https://kubernetes.slack.com/archives/C09R1PJR3/p1747786073248159

Signed-off-by: Jian Qiu <[email protected]>

ryanzhang-oss · 2025-05-22T01:43:28Z

tools/plugins/spiffe/plugin.go

+	return "spiffe"
+}
+
+func (p *Plugin) Credential(_ *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error) {


this doesn't even use the clusterProfile?

ryanzhang-oss · 2025-05-22T01:43:53Z

tools/plugins/ocm/plugin.go

+}
+
+func (p *Plugin) Credential(cluster *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error) {
+	tokenSecret, err := p.client.CoreV1().Secrets(cluster.Name).Get(context.Background(), "token", v1.GetOptions{})


this just use a secret of the same name as the clusterProfile...

michaelawyu · 2025-05-23T13:47:41Z

Thanks, qiujian16!

For the federated auth flow, the current interface could totally work; cluster-specific info. can be kept in the profile object, and application-specific info. would be discovered by the plugin itself in whichever means the plugin itself finds most appropriate.

I am not 100% sure if we would like to enforce some form of linking mechanism that would allow the plugin to access application-specific info. in some way, e.g., via (implicitly) linked config maps, but that could be implemented in further PRs I assume.

ryanzhang-oss · 2025-05-29T23:13:36Z

tools/plugins/interface.go

+	Name() string
+
+	// Credential returns the authentication information to connect to the cluster.
+	Credential(cluster *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error)


as discussed off line, we might need an extra field to contain application specific info so that different application get different identity and possibly different hub api URL too.

zhujian7 · 2025-06-03T10:29:22Z

tools/plugins/spiffe/plugin.go

+
+func (p *Plugin) Credential(_ *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error) {
+	svid, err := p.client.FetchJWTSVID(context.Background(), &workload.JWTSVIDRequest{
+		Audience: []string{"kube"},


The audience should be passed via a param or some other ways.

Signed-off-by: Jian Qiu <[email protected]>

corentone

do you intend to keep the spiffee as a built-in provider? My feeling is that we should focus on the exec and not have any built-in to avoid having a two tiered approach for now.

We can definitely do it later, but I think it will be detrimental to the plugin ecosystem if some of considered built-in.

corentone · 2025-06-04T20:18:52Z

tools/config/config.go

+	clusterEndpoint, err := getEndpointFromClusterProfile(cluster)
+	if err != nil {
+		return nil, err
+	}
+
+	config := &rest.Config{
+		Host: clusterEndpoint,
+		TLSClientConfig: rest.TLSClientConfig{
+			CAData: getCABundleFromClusterProfile(cluster),
+		},
+	}


I think we should have a method for this entirely.
Build config from CP.

corentone · 2025-06-04T20:19:19Z

tools/config/config.go

+	a := newAuthenticator(plugin, cluster)
+	transportConfig, err := config.TransportConfig()
+	if err := a.UpdateTransportConfig(transportConfig); err != nil {
+		return nil, err
+	}
+


I'd merge all this as "apply plugin for credentials" on the config (so that it's all put together)

corentone · 2025-06-04T20:19:42Z

tools/config/config.go

+func init() {
+	spiffePlugin, err := spiffe.NewPlugin()
+	if err == nil {
+		pluginMap[spiffePlugin.Name()] = spiffePlugin
+	} else {
+		utilruntime.HandleError(err)
+	}
+}
+


Could you isolate the "using library" part of obtaining credentials? (that way you would have a main that sets up the plugin, has a clusterprofile example and asks the library to construct credentials)

corentone · 2025-06-04T20:21:42Z

tools/plugins/spiffe/plugin.go

@@ -0,0 +1,84 @@
+package spiffe


could you make this plugin exec'ed ? So his package would become a main and we can leverage the client-go code to call it.

corentone · 2025-06-04T20:23:45Z

tools/plugins/spiffe/plugin.go

+	return pluginName
+}
+
+func (p *Plugin) Credential(cluster *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error) {


with the latest from the creds kep, you would actually get a "Cluster" object here.
https://github.com/kubernetes/enhancements/pull/5338/files#diff-6dcc0abd0aea71ca2e11e2a5b38ae81f50a227ae1c2c116d058b4de8abd35d20R236

I cannot make Cluster in the CRD because of this line https://github.com/kubernetes/kubernetes/blob/a34c07971b610eb33908a743eadb4c61beeecc50/staging/src/k8s.io/client-go/pkg/apis/clientauthentication/v1/types.go#L92...
seems like a bug to me.

corentone · 2025-06-04T20:25:17Z

tools/config/config.go

+	return res, nil
+}
+
+type Authenticator struct {


is there no existing authenticator struct you can leverage and build on top?

We could use https://github.com/kubernetes/kubernetes/blob/544e7a63e2cb6646d19e6a17bd665b07f39843d1/staging/src/k8s.io/client-go/plugin/pkg/client/auth/exec/exec.go#L244 as long as we agree that exec is the only way.

corentone · 2025-06-04T20:26:40Z

tools/plugins/exec/plugin.go

+	return p.name
+}
+
+func (p *Plugin) Credential(cluster *v1alpha1.ClusterProfile) (*clientauthentication.ExecCredential, error) {


is there code in client-go we can leverage to avoid having to implement this ourselves?

michaelawyu · 2025-06-05T01:58:45Z

tools/config/config.go

+	PluginPropertyKey            = "plugin.multicluster.k8s.io"
+)
+
+var pluginMap = map[string]plugins.PluginInterface{}


Hi qiujian16! Since this variable is not exposed, for exec plugins I assume that people wouldn't be able to use the BuildConfigFromClusterProfile, if I understand it correctly?

michaelawyu · 2025-06-05T02:04:10Z

tools/config/config.go

+	return []byte(caString)
+}
+
+func getPluginFromClusterProfile(cluster *v1alpha1.ClusterProfile) (plugins.PluginInterface, error) {


Hi qiujian16! Just a quick Q on this part: it seems that the API alright includes a provider name part (and the exec plugin uses it) which seems to suggest multiple plugins available at a time, but the code here (including the Build function) would query the plugin name property key (i.e., only one active plugin at a time) and retrieve the desirable plugin.

Signed-off-by: Jian Qiu <[email protected]>

corentone

I love how simple it is; thank you for putting it together.
The only thing missing is provider configuration that would need to be done once for the controller on start and I think we can have a working prototype!

corentone · 2025-06-10T17:42:05Z

apis/v1alpha1/clusterprofile_types.go

+	Config ProviderConfig `json:"config"`
+}
+
+type ProviderConfig struct {


was there a blocker to use clientauthenticationapi.Cluster ?

same question, they look the same?

corentone · 2025-06-10T17:44:37Z

tools/config/config.go

+}
+
+// BuildConfigFromClusterProfile is to build the rest.Config to init the client.
+func BuildConfigFromClusterProfile(cluster *v1alpha1.ClusterProfile, providerName string) (*rest.Config, error) {


I like this proto!

I think we will need an additional method.
func GetOptimalPluginForCP(CP, pluginConfig) pluginName

Something that would allow a user to find the right plugin based on what they've configured.

also again, I do love how simple it made this function by reusing existing exec components. thats a win for maintainability :) (and the immense featureset we get on proxy, TLS config etc...)

any example of what a "provider" is in real world case? I am a bit confused on what it actually is.

A provider type name would be GKE.

So for example, inside of the ClusterProfile, there will be a map with the key GKE. Something like:

status: credentials: - name: GKE config: server: http://apiserver_endpoint:port

In the controller configuration (likely a flag or configmap?), there will be something like:

GKE: ExecConfig{ command:"/usr/bin/gke-gcloud-auth-plugin", apiVersion: "client.authentication.k8s.io/v1beta1", provideClusterInfo: true } MyProviderType: ExecConfig{ command:"/usr/bin/mybinary", provideClusterInfo: true }

You can easily see that the binary is /usr/bin/gke-gcloud-auth-plugin and it will receive infor from the cluster thats in the config part of the CP. That's it.
You can also easily imagine that if there were two providers to support, it's just a matter of adding configuration into the controller to support them, for example the "MyProviderType" above. And the ClusterProfile would have a credentials request for that kind of provider (my example only has GKE, but it technically could have a different one as well).

corentone · 2025-06-10T17:56:36Z

tools/config/config.go

+
+	// If it is not exec Config, we have to find a way to fallback to a different path.
+	execConfig := &clientcmdapi.ExecConfig{}
+	err = json.Unmarshal(provider.Config.Raw, execConfig)


This exec config will likely not come from the CP; it will come from the configuration of the provider type for the controller.

when I look at this struct https://github.com/kubernetes/client-go/blob/master/tools/clientcmd/api/types.go#L208

type ExecConfig struct { // Command to execute. Command string `json:"command"` // Arguments to pass to the command when executing it. // +optional Args []string `json:"args"` // Env defines additional environment variables to expose to the process. These // are unioned with the host's environment, as well as variables client-go uses // to pass argument to the plugin. // +optional Env []ExecEnvVar `json:"env"` // Preferred input version of the ExecInfo. The returned ExecCredentials MUST use // the same encoding version as the input. APIVersion string `json:"apiVersion,omitempty"` // This text is shown to the user when the executable doesn't seem to be // present. For example, `brew install foo-cli` might be a good InstallHint for // foo-cli on Mac OS systems. InstallHint string `json:"installHint,omitempty"` // ProvideClusterInfo determines whether or not to provide cluster information, // which could potentially contain very large CA data, to this exec plugin as a // part of the KUBERNETES_EXEC_INFO environment variable. By default, it is set // to false. Package k8s.io/client-go/tools/auth/exec provides helper methods for // reading this environment variable. ProvideClusterInfo bool `json:"provideClusterInfo"` // Config holds additional config data that is specific to the exec // plugin with regards to the cluster being authenticated to. // // This data is sourced from the clientcmd Cluster object's extensions[exec] field: // // clusters: // - name: my-cluster // cluster: // ... // extensions: // - name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config // extension: // audience: 06e3fbd18de8 # arbitrary config // // In some environments, the user config may be exactly the same across many clusters // (i.e. call this exec plugin) minus some details that are specific to each cluster // such as the audience. This field allows the per cluster config to be directly // specified with the cluster info. Using this field to store secret data is not // recommended as one of the prime benefits of exec plugins is that no secrets need // to be stored directly in the kubeconfig. // +k8s:conversion-gen=false Config runtime.Object `json:"-"` // InteractiveMode determines this plugin's relationship with standard input. Valid // values are "Never" (this exec plugin never uses standard input), "IfAvailable" (this // exec plugin wants to use standard input if it is available), or "Always" (this exec // plugin requires standard input to function). See ExecInteractiveMode values for more // details. // // If APIVersion is client.authentication.k8s.io/v1alpha1 or // client.authentication.k8s.io/v1beta1, then this field is optional and defaults // to "IfAvailable" when unset. Otherwise, this field is required. // +optional InteractiveMode ExecInteractiveMode `json:"interactiveMode,omitempty"` // StdinUnavailable indicates whether the exec authenticator can pass standard // input through to this exec plugin. For example, a higher level entity might be using // standard input for something else and therefore it would not be safe for the exec // plugin to use standard input. This is kept here in order to keep all of the exec configuration // together, but it is never serialized. // +k8s:conversion-gen=false StdinUnavailable bool `json:"-"` // StdinUnavailableMessage is an optional message to be displayed when the exec authenticator // cannot successfully run this exec plugin because it needs to use standard input and // StdinUnavailable is true. For example, a process that is already using standard input to // read user instructions might set this to "used by my-program to read user instructions". // +k8s:conversion-gen=false StdinUnavailableMessage string `json:"-"` }

All those configs can be shared for all the CPs of a similar provider type.
instead of passing providerName, we should pass a struct with both the name and the exec config:

type Provider struct { Name string ExecConfig *clientcmdapi.ExecConfig }

Update code following the KEP

Signed-off-by: Jian Qiu <[email protected]>

mikeshng

/lgtm

k8s-ci-robot · 2025-07-02T13:27:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikeshng, qiujian16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mikeshng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 21, 2025

k8s-ci-robot requested a review from mikeshng May 21, 2025 08:24

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 21, 2025

k8s-ci-robot requested a review from ryanzhang-oss May 21, 2025 08:24

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 21, 2025

qiujian16 force-pushed the credential branch from 58ce073 to 902f745 Compare May 21, 2025 09:02

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 21, 2025

qiujian16 force-pushed the credential branch 2 times, most recently from fb49e15 to dab2b3b Compare May 21, 2025 09:08

Define plugin interface and consumer library

a71bb6e

Signed-off-by: Jian Qiu <[email protected]>

qiujian16 force-pushed the credential branch from dab2b3b to a71bb6e Compare May 21, 2025 09:10

ryanzhang-oss reviewed May 22, 2025

View reviewed changes

ryanzhang-oss reviewed May 29, 2025

View reviewed changes

mikeshng mentioned this pull request Jun 2, 2025

KEP 5339 - ClusterProfile Credentials external providers kubernetes/enhancements#5338

Open

zhujian7 reviewed Jun 3, 2025

View reviewed changes

Add exec config in the api status

b017892

Signed-off-by: Jian Qiu <[email protected]>

corentone reviewed Jun 4, 2025

View reviewed changes

michaelawyu reviewed Jun 5, 2025

View reviewed changes

Only support exec

c8ee052

Signed-off-by: Jian Qiu <[email protected]>

corentone reviewed Jun 10, 2025

View reviewed changes

everpeace mentioned this pull request Jun 19, 2025

🌱 Cluster Inventory(ClusterProfile) API Provider kubernetes-sigs/multicluster-runtime#48

Open

corentone and others added 3 commits June 19, 2025 23:21

Update code following the KEP

0740baa

Add an example

22d642b

Merge pull request #1 from corentone/credential

1be9936

Update code following the KEP

k8s-ci-robot removed the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 24, 2025

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 24, 2025

Rename pkg and add unit test

78f9633

Signed-off-by: Jian Qiu <[email protected]>

qiujian16 force-pushed the credential branch from a646a10 to 78f9633 Compare June 24, 2025 08:35

Add integration tests

212a451

Signed-off-by: Jian Qiu <[email protected]>

qiujian16 marked this pull request as ready for review July 2, 2025 02:18

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 2, 2025

k8s-ci-robot requested review from dixudx and ryanzhang-oss July 2, 2025 02:19

mikeshng approved these changes Jul 2, 2025

View reviewed changes

k8s-ci-robot assigned mikeshng Jul 2, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 2, 2025

k8s-ci-robot merged commit 0f613c6 into kubernetes-sigs:main Jul 2, 2025
2 checks passed

mikeshng removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 2, 2025

Define plugin interface and consumer library #17

Define plugin interface and consumer library #17

Uh oh!

Conversation

qiujian16 commented May 21, 2025

Uh oh!

RainbowMango commented May 21, 2025

Uh oh!

qiujian16 commented May 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelawyu commented May 23, 2025

Uh oh!

ryanzhang-oss May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corentone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelawyu Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corentone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikeshng left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

ryanzhang-oss May 29, 2025 •

edited

Loading

michaelawyu Jun 5, 2025 •

edited

Loading