2.0: Consider daemonization

[squeed]

Background

The exec-based API can be awkward for a few reasons:

• Many plugins need a daemon anyways
• Writing binaries to disk is a security risk
• Executing binaries in a containerized environment is fragile (not everything can / should be statically linked)

Considerations

• Not all CNI runtimes are daemons, so this would need to be optional (or we continue supporting 1.0 indefinitely)
• Golang and namespaces is still ripe for bugs
• Should we use gRPC?
• If a plugin is used for more than one "network", does that mean multiple processes, or should plugins support "multitenancy"?

Advantages

• Daemonized plugins could support additional lifecycle / methods
  • garbage collection
  • initialization
• Easier to deploy in as a container

Bidirectionality?

Would we want plugins to be able to send events to the runtime? Or should we still model this as a request-response protocol?

[robberphex]

The CNI is an interface. So in CNI, we could just define an interface and some methods, and plugin/runtime interaction. The CNI server could be implemented by others(containerd I think).

I think CNI should be defined via gRPC, to be called by CRI, or others. For plugins, call via envvar is good, I think we can still use it in 2.0.

[danwinship]

Executing binaries in a containerized environment is fragile (not everything can / should be statically linked)

Citation required? It seems to me that the problems with dynamic linking are entirely in the opposite direction; if you copy a dynamically-linked binary from your container to the root disk, it may not be able to find the shared libraries it needs any more.

If you're talking about openssl FIPS compliance stuff specifically, I don't think that's a problem that can be solved at the CNI level; if a particular distribution (eg OCP) or cluster-infrastructure provider (eg EKS) needs to require specific things of all CNI plugins running in that environment, then either they need to build all the CNI plugins themselves, or they need to bless specific container images containing tested/approved CNI plugins. I don't see any way of getting around that.

Golang and namespaces is still ripe for bugs

To the best of my knowledge there is currently exactly one known bug, which is that if a goroutine running in a namespace spawns off a new goroutine, it is not guaranteed that the new goroutine will remain in the same namespace. Which seems like something golang needs to provide some fix for (though it will be complicated since sometimes you do want the current semantics).

Not all CNI runtimes are daemons, so this would need to be optional (or we continue supporting 1.0 indefinitely)
Easier to deploy in as a container

It would be great if CNI 2.0 required all plugins to be run from containers, even if they aren't daemons, and did not require containers to make any modifications to the root disk.

[squeed]

Executing binaries in a containerized environment is fragile ...

Citation required? It seems to me that the problems with dynamic linking are entirely in the opposite direction; if you copy a dynamically-linked binary from your container to the root disk, it may not be able to find the shared libraries it needs any more.

Yes, good point. I wrote it backwards. s/executing/leaking-from-a-container/

Golang and namespaces is still ripe for bugs

To the best of my knowledge there is currently exactly one known bug ...

Agreed, that is my understanding as well. I still don't love doing something that is, at best, unofficially supported by the runtime.

Not all CNI runtimes are daemons, so this would need to be optional (or we continue supporting 1.0 indefinitely)
Easier to deploy in as a container

It would be great if CNI 2.0 required all plugins to be run from containers, even if they aren't daemons, and did not require containers to make any modifications to the root disk.

If we switch to gRPC, then the "interface" is a socket file, making no statement as to how plugins are executed (and thus enabling containerization). However, if we eschew gRPC and stick with an execution-based protocol, it would be interesting if the CNI flow included, somehow, the container runtime engine executing more containers (i.e. plugins) on behalf of CNI...

Of course, many plugins need to store some state between invocations, so there is always that concern. But that is somewhat incidental.

[kfox1111]

To the best of my knowledge there is currently exactly one known bug ...

Agreed, that is my understanding as well. I still don't love doing something that is, at best, unofficially supported by the runtime.

Got a link for posterity?

[kfox1111]

Advantages

• Daemonized plugins could support additional lifecycle / methods

  • garbage collection
  • initialization

• Easier to deploy in as a container

I'd add * more consistent/predictable resource utilization
You can reserve it in the container and its always allocated to the daemon rather then try and reserve it on the host and watch the hosts memory fluctuate.

[danwinship]

It would be great if CNI 2.0 required all plugins to be run from containers, even if they aren't daemons, and did not require containers to make any modifications to the root disk.

(I meant "did not require plugins to make any modifications")

If we switch to gRPC, then the "interface" is a socket file, making no statement as to how plugins are executed (and thus enabling containerization). However, if we eschew gRPC and stick with an execution-based protocol, it would be interesting if the CNI flow included, somehow, the container runtime engine executing more containers (i.e. plugins) on behalf of CNI...

Yes, that's what I meant. I mean, obviously the runtime knows how to run a binary out of a container, so...

Of course, many plugins need to store some state between invocations, so there is always that concern. But that is somewhat incidental.

I'm fine with plugins being allowed to write to the root disk, I just don't think CNI should require plugins to write to the root disk. ie, it should not require you to copy out a binary and it should not require you to write out a config file.

[danwinship]

To the best of my knowledge there is currently exactly one known bug ...

Agreed, that is my understanding as well. I still don't love doing something that is, at best, unofficially supported by the runtime.

Got a link for posterity?

I don't have a link but I can explain the problem.

There's a golang routine (runtime.LockOSThread()) that lets you say "I'm doing something weird with this thread, and this goroutine needs to stay on that thread until I'm done". So github.com/containernetworking/plugins/pkg/ns uses that, so when you do netns.Do(func() { ... }) it calls runtime.LockOSThread() and then syscall.Setns() and then runs your function.

Long ago, people realized that this caused bugs, because although LockOSThread prevented the current goroutine from being moved to another thread, it didn't prevent other goroutines from being scheduled to this thread if your goroutine got blocked. So this caused awful problems where a goroutine's network namespace would randomly change. This eventually got fixed.

But more recently people discovered that the opposite problem exists; if you spawn off a new goroutine from your locked thread, that goroutine won't stay on the same thread. In particular, if you do netns.Do(func() { net.Dial("tcp", "dual-stack-host.example.com") }), then net.Dial will spawn goroutines to try IPv4 and IPv6 connections in parallel, but those goroutines will end up running in the main netns, not the one you were trying to use.

And this is not an unambiguous bug like the previous problem, because goroutines are ubiquitous and there are lots of circumstances where you wouldn't want the goroutine to inherit the special behavior from the original thread. So any fix for this is likely to require API changes of some sort (?) and it's not clear any fix is coming any time soon.

[danwinship]

(And you can't just say "well don't use the 'unsafe' functions" because there's no way to know from the outside which functions are unsafe. It's entirely possible that plugins that work today would break in the future if additional goroutines were added to functions in "exec" or "github.com/vishvananda/netlink".)

(Ugh.)

[nevermore-muyi]

Add an advantage: in my environment, i deploy 2000 nodes and 40,000 pods.

1. cni request need get pod info and network-attachment-definitions
2. cni request faile always
3. cni request will retry and fail again
4. cycle...

So, in sometime may many nodes retry cni request and send request to kube-apiserver, and get 429 response from kube-apiserver. If deployed in daemonization, we can limit every node's request by client-go's throttles and reduce kube-apiserver's pressure.

[yuval-k]

how would plugin delegation work? i.e. something like ptp calling host-local for IPAM.
will the daemonized plugins invoke make an RPC call to the cni runtime to invoke plugins?

[danwinship]

It's entirely possible that plugins that work today would break in the future if additional goroutines were added to functions in "exec" or "github.com/vishvananda/netlink"

OK, actually "exec" guarantees it won't do this:

If the calling goroutine has locked the operating system thread with runtime.LockOSThread and modified any inheritable OS-level thread state (for example, Linux or Plan 9 name spaces), the new process will inherit the caller's thread state.

[danwinship]

So what if we didn't think about this in terms of "daemonization" and "gRPC", but just thought about it as "providing a network transport for CNI requests in addition to the existing exec-based transport".

Basically, section 2 of the spec would be rewritten to say that there are two ways of invoking a CNI plugin:

1. Execute a binary with certain environment environment variables set, pass it some JSON data via stdin, and read its JSON response from its stdout.
2. Connect to a remote server (via some means), pass it the same JSON and environment-variable data as in the exec case (encoded in some form), and wait for the response from the server.

And then, if, say, you have a chain consisting of a network-based plugin and an exec-based plugin, the semantics are still totally clear: you make a network call to the network-based plugin, get back the result, then exec the exec-based plugin, passing it the result from the network plugin as its input.

And because the network-based protocol is just a trivial translation of the exec-based protocol, we could provide an (exec-based) "cnishim" plugin in containernetworking/plugins that is invoked like any other exec-based plugin, which uses the information in the CNI conf to find the server to connect to, makes a network-based CNI request, and then serializes the results back to exec form to return to its caller. So then, people who want to use the network-based transport in their plugin will write out a CNI config that specifies "type": "cnishim", "socket": "...", and newer runtimes would see that and just connect directly to the socket, but older runtimes would hand off requests to the cnishim binary, which would make the connection for them.

• Writing binaries to disk is a security risk

• Executing binaries [copied out of] a containerized environment is fragile (not everything can / should be statically linked)

This solves both of those in the long term (though in the short term, to ensure compatibility with older systems, cnishim users would have to copy the cnishim binary out of their container image if there wasn't already a copy of it in /opt/cni/bin).

• Not all CNI runtimes are daemons, so this would need to be optional (or we continue supporting 1.0 indefinitely)

it's optional

• Golang and namespaces is still ripe for bugs

The daemon can still exec a helper binary of its own if it's worried about that, though it seems that (as above) exec is now completely namespace-space, and we should probably get vishvananda/netlink to make similar explicit guarantees.

• Should we use gRPC?

🤷‍♂

This proposal is entirely request-response just like exec-based CNI, so it doesn't need anything complicated. Openshift-sdn, ovn-kubernetes, and multus-cni all use a very similar shim implementation based on just marshaling the CNI request to JSON and sending it to the daemon via http-over-unix-domain-socket.

We could avoid standardizing the actual protocol at first, and say that initially, the only supported way to do this is for the daemon to use code from pkg/skel to set up the listening server, and for the runtime to use code from pkg/invoke to connect to it. (And then if we decide the initial implementation was terrible, we can just make a new implementation and call it "cnishim2" or whatever, until we have something we're happy with, and then standardize it.)

• If a plugin is used for more than one "network", does that mean multiple processes, or should plugins support "multitenancy"?

In this scenario, the CNI conf file specifies how to connect to the daemon, so you could do it either way: you could have a single daemon with a single server and write out multiple CNI conf files that all connected to that one server (which would use the "name" or other attributes of the CNI conf to decide how to behave); or you could have multiple daemons each with their own server, and so each one would write out a CNI conf file that pointed to its own server.

Would we want plugins to be able to send events to the runtime? Or should we still model this as a request-response protocol?

This approach intentionally doesn't add any new API for now, for simplicity.