Custom controller framework

[mhausenblas]

This design issue is about how custom controllers are written in ASO. The goal is to select a widely used framework that has a low barrier to contributing to ASO. Usage of requirements key words as per RFC 2119.

Scope

In scope for this issue is the selection of the custom controller framework, that is, a Go-based framework supporting the creation of custom controllers used in ASO. Out of scope for this issue is the question how the business logic (managing AWS services) in the custom controllers is implemented, see #5 for this.

Input

Relevant candidates:

• Kubebuilder
• Operator SDK

Input from roadmap issue aws/containers-roadmap#456:

• @daviddyball in AWS Controllers for Kubernetes (ACK) aws/containers-roadmap#456 (comment)

Requirements

1. The custom controller framework MUST support Go.
2. The custom controller framework SHOULD be well-documented, well-maintained, and allow for easy on-boarding.

[daviddyball]

I'd vote for kubebuilder. My company moved from operator-sdk to kubebuilder very early on and it was just a smoother process. We went from one barely working operator to 4 fully functioning controllers (one of them our internal implementation of ASO as a controller 😛)

I believe it is a younger project to operator-sdk, but it actually feels more stable and implements things in a "Kubernetes" way, whereas the Operator SDK defines it's own jargon and concepts that don't translate to the Kubernetes world very well IMHO.

kubebuilder is also a K8s SIG project, so support is pretty solid and well supported.

[cpoole]

+1 for kubebuilder my experience mirrors @daviddyball

kubebuilder feels much more kube native and I expect will see significantly more support and growth in the next year

[jaypipes]

My vote is obviously with kubebuilder as well. The trick for ASO is putting some scaffolding in place where we can a) generate the kubebuilder commands for each exposes AWS service API (basically, we need a "generator generator" :) and b) keep the CRDs and controller implementation for the various exposed AWS service APIs up to date as either upstream aws-sdk-go or the AWS APIs themselves change.

[pires]

kubebuilder has some rough edges, and so does controller-runtime (CT) which kubebuilder relies on. That said, it is a project backed by a couple sig-apimachinery people, so I trust it to gain more attention (and love) as its usage grows.
As an example, CT/kubebuilder are not yet prime-time for CRD v1.

[negz]

Hi folks! I'm the tech lead for https://github.com/crossplane/stack-aws - a project very similar to what it sounds like you're planning with the Service Operator. We went with controller-runtime and have been quite happy with our choice. Rather than take a pure code generation/scaffolding approach we built a series of generic reconcilers that reconcile a "shape" (or duck type) of Kubernetes custom resource. These are packaged in crossplane-runtime, and could be reused by a project like the Service Operator if you so desired.

The general pattern is that there's a "reconciler scaffold" that handles all the parts of reconciling a Kubernetes CR with an AWS API that are mostly the same from API to API. This reconciler wraps an ExternalClient, such that the task of the entity implementing each controller is narrowed to satisfying some CRUD methods;

type ExternalClient interface {
	// Observe the external resource the supplied Managed resource represents,
	// if any. Observe implementations must not modify the external resource,
	// but may update the supplied Managed resource to reflect the state of the
	// external resource.
	Observe(ctx context.Context, mg resource.Managed) (ExternalObservation, error)

	// Create an external resource per the specifications of the supplied
	// Managed resource. Called when Observe reports that the associated
	// external resource does not exist.
	Create(ctx context.Context, mg resource.Managed) (ExternalCreation, error)

	// Update the external resource represented by the supplied Managed
	// resource, if necessary. Called unless Observe reports that the
	// associated external resource is up to date.
	Update(ctx context.Context, mg resource.Managed) (ExternalUpdate, error)

	// Delete the external resource upon deletion of its associated Managed
	// resource. Called when the managed resource has been deleted.
	Delete(ctx context.Context, mg resource.Managed) error
}

Our ElastiCache ReplicationGroup controller is a good example of a reconciler for a moderately complex AWS API that uses this pattern.

[negz]

@jaypipes PS the thoughts on code generation you captured in #15 matches our thinking around the subject almost exactly. Would you be interested in collaborating on something both projects could use?

[jaypipes]

@jaypipes PS the thoughts on code generation you captured in #15 matches our thinking around the subject almost exactly. Would you be interested in collaborating on something both projects could use?

@negz so sorry for being so uncommunicative about this! Not sure if you've been following along with our progress on the mvp branch, but we've been slogging along slowly but surely. Last Friday I pushed up a pull request that adds a good chunk of the controller implementation generation. Would be awesome to get your comments on it. I can certainly see our teams collaborating on a bunch of common runtime stuff, for the record!

[negz]

Nice, thanks @jaypipes! I'll take a look this week. Would you be interested in setting up a call at some point to swap notes?

[jaypipes]

absolutely @negz I would be more than happy to do that. My schedule next week is pretty open. Catch me on Slack and we'll schedule a time. :)

[jaypipes]

Closing this out. We ended up going with a custom code generator + controller-gen tooling.