-
Notifications
You must be signed in to change notification settings - Fork 1.2k
📖Adding ComponentConfig proposal #818
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,320 @@ | ||
# ComponentConfig Controller Runtime Support | ||
Author: @christopherhein | ||
|
||
Last Updated on: 03/02/2020 | ||
|
||
## Table of Contents | ||
|
||
<!--ts--> | ||
* [ComponentConfig Controller Runtime Support](#componentconfig-controller-runtime-support) | ||
* [Table of Contents](#table-of-contents) | ||
* [Summary](#summary) | ||
* [Motivation](#motivation) | ||
* [Links to Open Issues](#links-to-open-issues) | ||
* [Goals](#goals) | ||
* [Non-Goals/Future Work](#non-goalsfuture-work) | ||
* [Proposal](#proposal) | ||
* [ComponentConfig Load Order](#componentconfig-load-order) | ||
* [Embeddable ComponentConfig Type](#embeddable-componentconfig-type) | ||
* [Default ComponentConfig Type](#default-componentconfig-type) | ||
* [Using Flags w/ ComponentConfig](#using-flags-w-componentconfig) | ||
* [Kubebuilder Scaffolding Example](#kubebuilder-scaffolding-example) | ||
* [User Stories](#user-stories) | ||
* [Controller Author with controller-runtime and default type](#controller-author-with-controller-runtime-and-default-type) | ||
* [Controller Author with controller-runtime and custom type](#controller-author-with-controller-runtime-and-custom-type) | ||
* [Controller Author with kubebuilder (tbd proposal for kubebuilder)](#controller-author-with-kubebuilder-tbd-proposal-for-kubebuilder) | ||
* [Controller User without modifications to config](#controller-user-without-modifications-to-config) | ||
* [Controller User with modifications to config](#controller-user-with-modifications-to-config) | ||
* [Risks and Mitigations](#risks-and-mitigations) | ||
* [Alternatives](#alternatives) | ||
* [Implementation History](#implementation-history) | ||
|
||
<!--te--> | ||
|
||
## Summary | ||
|
||
Currently controllers that use `controller-runtime` need to configure the `ctrl.Manager` by using flags or hardcoding values into the initialization methods. Core Kubernetes has started to move away from using flags as a mechanism for configuring components and standardized on [`ComponentConfig` or Versioned Component Configuration Files](https://docs.google.com/document/d/1FdaEJUEh091qf5B98HM6_8MS764iXrxxigNIdwHYW9c/edit). This proposal is to bring `ComponentConfig` to `controller-runtime` to allow controller authors to make `go` types backed by `apimachinery` to unmarshal and configure the `ctrl.Manager{}` reducing the flags and allowing code based tools to easily configure controllers instead of requiring them to mutate CLI args. | ||
|
||
## Motivation | ||
|
||
This change is important because: | ||
- it will help make it easier for controllers to be configured by other machine processes | ||
- it will reduce the required flags required to start a controller | ||
- allow for configuration types which aren't natively supported by flags | ||
- allow using and upgrading older configurations avoiding breaking changes in flags | ||
|
||
### Links to Open Issues | ||
|
||
- [#518 Provide a ComponentConfig to tweak the Manager](https://github.com/kubernetes-sigs/controller-runtime/issues/518) | ||
- [#207 Reduce command line flag boilerplate](https://github.com/kubernetes-sigs/controller-runtime/issues/207) | ||
- [#722 Implement ComponentConfig by default & stop using (most) flags](https://github.com/kubernetes-sigs/kubebuilder/issues/722) | ||
|
||
### Goals | ||
|
||
- Provide an interface for pulling configuration data out of exposed `ComponentConfig` types (see below for implementation) | ||
- Provide a new `ctrl.NewFromComponentConfig()` function for initializing a manager | ||
- Provide an embeddable `ControllerConfiguration` type for easily authoring `ComponentConfig` types | ||
- Provide an `DefaultControllerConfig` to make the switching easier for clients | ||
|
||
### Non-Goals/Future Work | ||
|
||
- `kubebuilder` implementation and design in another PR | ||
- Changing the default `controller-runtime` implementation | ||
- Dynamically reloading `ComponentConfig` object | ||
- Providing `flags` interface and overrides | ||
|
||
## Proposal | ||
|
||
The `ctrl.Manager` _SHOULD_ support loading configurations from `ComponentConfig` like objects. | ||
An interface for that object with getters for the specific configuration parameters is created to bridge existing patterns. | ||
|
||
Without breaking the current `ctrl.NewManager` which uses an exported `ctrl.Options{}` the `manager.go` can expose a new func, `NewFromComponentConfig()` this would be able to loop through the getters to populate an internal `ctrl.Options{}` and pass that into `New()`. | ||
|
||
```golang | ||
//pkg/manager/manager.go | ||
|
||
// ManagerConfiguration defines what the ComponentConfig object for ControllerRuntime needs to support | ||
type ManagerConfiguration interface { | ||
GetSyncPeriod() *time.Duration | ||
|
||
GetLeaderElection() bool | ||
GetLeaderElectionNamespace() string | ||
GetLeaderElectionID() string | ||
|
||
GetLeaseDuration() *time.Duration | ||
GetRenewDeadline() *time.Duration | ||
GetRetryPeriod() *time.Duration | ||
|
||
GetNamespace() string | ||
GetMetricsBindAddress() string | ||
GetHealthProbeBindAddress() string | ||
|
||
GetReadinessEndpointName() string | ||
GetLivenessEndpointName() string | ||
|
||
GetPort() int | ||
GetHost() string | ||
|
||
GetCertDir() string | ||
} | ||
|
||
func NewFromComponentConfig(config *rest.Config, scheme *runtime.Scheme, filename string, managerconfig ManagerConfiguration) (Manager, error) { | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
codecs := serializer.NewCodecFactory(scheme) | ||
if err := decodeComponentConfigFileInto(codecs, filename, managerconfig); err != nil { | ||
|
||
} | ||
options := Options{} | ||
|
||
if scheme != nil { | ||
options.Scheme = scheme | ||
} | ||
|
||
// Loop through getters | ||
if managerconfig.GetLeaderElection() { | ||
options.LeaderElection = managerconfig.GetLeaderElection() | ||
} | ||
// ... | ||
|
||
return New(config, options) | ||
} | ||
``` | ||
|
||
#### ComponentConfig Load Order | ||
|
||
 | ||
|
||
#### Embeddable ComponentConfig Type | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
To make this easier for Controller authors `controller-runtime` can expose a set of `config.ControllerConfiguration` type that can be embedded similar to the way that `k8s.io/apimachinery/pkg/apis/meta/v1` works for `TypeMeta` and `ObjectMeta` these could live in `pkg/api/config/v1alpha1/types.go`. See the `DefaultComponentConfig` type below for and example implementation. | ||
|
||
```golang | ||
// pkg/api/config/v1alpha1/types.go | ||
package v1alpha1 | ||
|
||
import ( | ||
"time" | ||
|
||
configv1alpha1 "k8s.io/component-base/config/v1alpha1" | ||
) | ||
|
||
// ControllerConfiguration defines the embedded RuntimeConfiguration for controller-runtime clients. | ||
type ControllerConfiguration struct { | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Namespace string `json:"namespace,omitempty"` | ||
|
||
SyncPeriod *time.Duration `json:"syncPeriod,omitempty"` | ||
|
||
LeaderElection configv1alpha1.LeaderElectionConfiguration `json:"leaderElection,omitempty"` | ||
|
||
MetricsBindAddress string `json:"metricsBindAddress,omitempty"` | ||
|
||
Health ControllerConfigurationHealth `json:"health,omitempty"` | ||
|
||
Port *int `json:"port,omitempty"` | ||
Host string `json:"host,omitempty"` | ||
|
||
CertDir string `json:"certDir,omitempty"` | ||
} | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
// ControllerConfigurationHealth defines the health configs | ||
type ControllerConfigurationHealth struct { | ||
HealthProbeBindAddress string `json:"healthProbeBindAddress,omitempty"` | ||
|
||
ReadinessEndpointName string `json:"readinessEndpointName,omitempty"` | ||
LivenessEndpointName string `json:"livenessEndpointName,omitempty"` | ||
} | ||
``` | ||
|
||
|
||
|
||
#### Default ComponentConfig Type | ||
|
||
To enable `controller-runtime` to have a default `ComponentConfig` struct which can be used instead of requiring each controller or extension to build it's own `ComponentConfig` type, we can create a `DefaultControllerConfiguration` type which can exist in `pkg/api/config/v1alpha1/types.go`. This will allow the controller authors to use this before needing to implement their own type with additional configs. | ||
|
||
```golang | ||
// pkg/api/config/v1alpha1/types.go | ||
package v1alpha1 | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1" | ||
) | ||
|
||
// DefaultControllerConfiguration is the Schema for the DefaultControllerConfigurations API | ||
type DefaultControllerConfiguration struct { | ||
metav1.TypeMeta `json:",inline"` | ||
|
||
Spec configv1alpha1.ControllerConfiguration `json:"spec,omitempty"` | ||
} | ||
``` | ||
|
||
This would allow a controller author to use this struct with any config that supports the json/yaml structure. For example a controller author could define their `Kind` as `FoobarControllerConfiguration` and have it defined as the following. | ||
|
||
```yaml | ||
# config.yaml | ||
apiVersion: config.somedomain.io/v1alpha1 | ||
kind: FoobarControllerConfiguration | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
spec: | ||
port: 9443 | ||
metricsBindAddress: ":8080" | ||
leaderElection: | ||
leaderElect: false | ||
``` | ||
|
||
Given the following config and `DefaultControllerConfiguration` we'd be able to initialize the controller using the following. | ||
|
||
|
||
```golang | ||
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, &defaultv1alpha1.DefaultControllerConfiguration{}) | ||
if err != nil { | ||
// ... | ||
} | ||
``` | ||
|
||
The above example uses `configname` which is the name of the file to load the configuration from and uses `scheme` to get the specific serializer, eg `serializer.NewCodecFactory(scheme)`. This will allow the configuration to be unmarshalled into the `runtime.Object` type and passed into the | ||
`ctrl.NewManagerFromComponentConfig()` as a `ManagerConfiguration` interface. | ||
|
||
#### Using Flags w/ ComponentConfig | ||
|
||
Since this design still requires setting up the initial `ComponentConfig` type and passing in a pointer to `ctrl.NewFromComponentConfig()` if you want to allow for the use of flags, your controller can use any of the different flagging interfaces. eg [`flag`](https://golang.org/pkg/flag/), [`pflag`](https://godoc.org/github.com/spf13/pflag), [`flagnum`](https://godoc.org/github.com/luci/luci-go/common/flag/flagenum) and set values on the `ComponentConfig` type prior to passing the pointer into the `ctrl.NewFromComponentConfig()`, example below. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just FYI for backwards compatibility flags usually need to take precedence over config (unless you can force users to use exclusively flags or config). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good callout, since we're decoding in the same object that the flags could have already been set on there isn't a way I'm aware of to not override existing values while decoding. Have any suggestions? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. save off the flags and apply them after the decode? (merge the options structs together more or less)? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yea, that probably makes the most sense. I personally don't think this needs to be documented in the design doc, do you? Feeling okay with the rest of this @shawn-hurley ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Instead of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Totally a possibility. I'd originally started with that path and ended up combining the funcs to make the new API quicker to implement. @shawn-hurley @mtaufen @DirectXMan12 @stealthybox Do you have a preference? The options would be:
or
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another thought would be for the ComponentConfig-populating function to accept a pointer to a func main() {
opts := manager.Options{}
_ = flag.StringVar(&opts.Namespace, "namespace", "", "Namespace to watch.")
ctrl.UpdateOptionsFromComponentConfig(&opts)
// Parse flags after populating from ComponentConfig so that flags
// take precedence
_ = flag.Parse()
_ = manager.New(ctrl.GetConfigOrDie(), opts)
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We'd need to pass similar params into each func, EG parsing out the func main() {
opts := manager.Options{}
_ = flag.StringVar(&opts.Namespace, "namespace", "", "Namespace to watch.")
_ = flag.StringVar(&filename, "config", "f", "ConmponentConfig type configuration.")
_ = ctrl.UpdateOptionsFromComponentConfig(scheme, filename, &componentconfig, &opts)
// Parse flags after populating from ComponentConfig so that flags
// take precedence
_ = flag.Parse()
_ = manager.New(ctrl.GetConfigOrDie(), opts)
} I think both are fair flows and we could achieve the same goals with a composite function that returns the manager or returns the options. Does anyone have strong opinions against this? @stealthybox @johnsonj @joelanford @djzager @shawn-hurley @mtaufen |
||
|
||
```golang | ||
leaderElect := true | ||
|
||
config := &defaultv1alpha1.DefaultControllerConfiguration{ | ||
Spec: configv1alpha1.ControllerConfiguration{ | ||
LeaderElection: configv1alpha1.LeaderElectionConfiguration{ | ||
LeaderElect: &leaderElect, | ||
}, | ||
}, | ||
} | ||
mgr, err := ctrl.NewManagerFromComponentConfig(ctrl.GetConfigOrDie(), scheme, configname, config) | ||
if err != nil { | ||
// ... | ||
} | ||
``` | ||
|
||
#### Kubebuilder Scaffolding Example | ||
|
||
Within expanded in a separate design _(link once created)_ this will allow controller authors to generate a type that implements the `ManagerConfiguration` interface. The following is a sample of what this looks like: | ||
|
||
```golang | ||
package config | ||
|
||
import ( | ||
"time" | ||
|
||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
configv1alpha1 "sigs.k8s.io/controller-runtime/pkg/apis/config/v1alpha1" | ||
) | ||
|
||
type ControllerNameConfigurationSpec struct { | ||
christopherhein marked this conversation as resolved.
Show resolved
Hide resolved
|
||
configv1alpha1.ControllerConfiguration `json:",inline"` | ||
} | ||
|
||
type ControllerNameConfiguration struct { | ||
metav1.TypeMeta | ||
|
||
Spec ControllerNameConfigurationSpec `json:"spec"` | ||
} | ||
``` | ||
|
||
Usage of this custom `ComponentConfig` type would require then changing the `ctrl.NewFromComponentConfig()` to use the new struct vs the `DefaultControllerConfiguration`. | ||
|
||
## User Stories | ||
|
||
### Controller Author with `controller-runtime` and default type | ||
|
||
- Mount `ConfigMap` | ||
- Initialize `ctrl.Manager` with `NewFromComponentConfig` with config name and `DefaultControllerConfiguration` type | ||
- Build custom controller as usual | ||
|
||
### Controller Author with `controller-runtime` and custom type | ||
|
||
- Implement `ComponentConfig` type | ||
- Embed `ControllerConfiguration` type | ||
- Mount `ConfigMap` | ||
- Initialize `ctrl.Manager` with `NewFromComponentConfig` with config name and `ComponentConfig` type | ||
- Build custom controller as usual | ||
|
||
### Controller Author with `kubebuilder` (tbd proposal for `kubebuilder`) | ||
|
||
- Initialize `kubebuilder` project using `--component-config-name=XYZConfiguration` | ||
- Build custom controller as usual | ||
|
||
### Controller User without modifications to config | ||
|
||
_Provided that the controller provides manifests_ | ||
|
||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
### Controller User with modifications to config | ||
|
||
- _Following from previous example without changes_ | ||
- Create a new `ConfigMap` for changes | ||
- Modify the `controller-runtime` pod to use the new `ConfigMap` | ||
- Apply the controller to the cluster | ||
- Deploy custom resources | ||
|
||
|
||
## Risks and Mitigations | ||
|
||
- Given that this isn't changing the core Manager initialization for `controller-runtime` it's fairly low risk | ||
|
||
## Alternatives | ||
|
||
* `NewFromComponentConfig()` could load the object from disk based on the file name and hydrate the `ComponentConfig` type. | ||
|
||
## Implementation History | ||
|
||
- [x] 02/19/2020: Proposed idea in an issue or [community meeting] | ||
- [x] 02/24/2020: Proposal submitted to `controller-runtime` | ||
- [x] 03/02/2020: Updated with default `DefaultControllerConfiguration` | ||
- [x] 03/04/2020: Updated with embeddable `RuntimeConfig` | ||
- [x] 03/10/2020: Updated embeddable name to `ControllerConfiguration` | ||
|
||
|
||
<!-- Links --> | ||
[community meeting]: https://docs.google.com/document/d/1Ih-2cgg1bUrLwLVTB9tADlPcVdgnuMNBGbUl4D-0TIk |
Uh oh!
There was an error while loading. Please reload this page.