Is restarting cri-containerd after configuring container networking required?

[tkellen]

First, thank you for cri-containerd. I've been using it a lot lately and it works wonderfully!

I'm experiencing an issue related to bootstrapping a cluster that I am hoping you all can help with.

If I don't restart cri-containerd after deploying my container networking pods, any pod not connected to the host network fails to start.

I have a full example process, including the cri-containerd restart documented here:
https://github.com/tkellen/kwm#first-time-user-guide

If you skip the cri-containerd restart step you'll see that kube-dns (or any other pod not connected to the host network) fails to start with the error "Failed create pod sandbox". As soon as you restart cri-containerd, the issue resolves itself. Is this expected? If not, can you help me understand what the correct process for this would be?

Thanks so much!

[tkellen]

I've posted this over on the kube-router repo because I'm unclear if this is an issue with cri-containerd or kube-router (or neither, maybe it's just me!). Here's the link:
cloudnativelabs/kube-router#286

[mikebrow]

CNI config is static, and maybe you are configuring it late in the process thus requiring a reboot to load the newly changed config? @Random-Liu wdyt

[tkellen]

Yes, when kube-router starts it alters the configuration in /etc/cni/net.d/10-kuberouter on each node to take on the correct CIDR.

[Random-Liu]

Based on the code, CNI should be updated with latest configure. If it doesn't work, it might be a bug.
However, @tkellen will you change CIDR when there are already non-hostnetwork pods running? Won't that confuse CNI?

[tkellen]

@Random-Liu This is a weird state I am encountering during cluster bootstrapping only. I do not intend to change the CIDR during the course of normal operation. My intent is to make sure things are configured correctly before I start any (non-network-host) pods.

At the moment, it seems I am in a chicken/egg scenario. Starting kube-router (as a daemonset) requires cri-containerd to be running. During the startup process of kube-router, the CNI configuration on each Node is modified:

Kube-router does not perform subnet management and relies upon Kubernetes controll manager to do the subnet allocation for each node. Each node in the cluster is allocated unique subnet from the cluster CIDR range for allocation of IP’s to the pods running on the node.
https://cloudnativelabs.github.io/post/2017-05-22-kube-pod-networking/

Can you help me understand what you mean by this?

Based on the code, CNI should be updated with latest configure. If it doesn't work, it might be a bug.

[abhi]

@Random-Liu This is a similar issue I have encountered with the cni stuff that I discussed with you last week. @tkellen I think restarting containerd might trigger a restart of cri-containerd (because of watch on cri-containerd @Random-Liu correct me if I am wrong here) So that might be the reason why things might start to work after reloading containerd. Feel free to open a bug and I can take a look at this.

[tkellen]

Thanks for following up @abhi! I've just confirmed restarting cri-containerd resolves the issue also. Where should I open a bug?

[tkellen]

Heads up I've just edited my prior posts (and the title of this issue) to reference restarting cri-containerd so future readers aren't sent on a wild goose chase. Also, here are my logs for cri-containerd from initial failure to resolution (showing initial startup + service restart after kube-router modifies its CNI configuration by communicating with the controller manager to find the correct CIDR):

Jan 16 04:43:05 tiny cri-containerd[2019]: E0116 04:43:05.279222    2019 instrumented_service.go:40] RunPodSandbox for &PodSandboxMetadata{Name:kube-dns-574cf969f4-c64mq,Uid:b6d08be0-fa77-11e7-8e33-12612308a448,Namespace:kube-system,Attem
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=info msg="Got pod network {Name:kube-dns-574cf969f4-c64mq Namespace:kube-system ID:ba21cf3657bb6e300d0d37a31cf68e6db73a718d21a5812e34787924bc737f7a NetNS:/var/ru
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=info msg="Got pod network {Name:kube-dns-574cf969f4-c64mq Namespace:kube-system ID:ba21cf3657bb6e300d0d37a31cf68e6db73a718d21a5812e34787924bc737f7a NetNS:/var/ru
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=info msg="About to add CNI network kubernetes (type=bridge)"
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=error msg="Error adding network: no IP ranges specified"
Jan 16 04:43:19 tiny cri-containerd[2019]: time="2018-01-16T04:43:19Z" level=error msg="Error while adding to cni network: no IP ranges specified"
Jan 16 04:43:19 tiny cri-containerd[2019]: E0116 04:43:19.287222    2019 instrumented_service.go:40] RunPodSandbox for &PodSandboxMetadata{Name:kube-dns-574cf969f4-c64mq,Uid:b6d08be0-fa77-11e7-8e33-12612308a448,Namespace:kube-system,Attem
Jan 16 04:43:27 tiny systemd[1]: Stopping Kubernetes containerd CRI shim...
Jan 16 04:43:27 tiny cri-containerd[2019]: E0116 04:43:27.480775    2019 service.go:217] Failed to serve grpc grpc request: accept unix /var/run/cri-containerd.sock: use of closed network connection
Jan 16 04:43:27 tiny cri-containerd[2019]: E0116 04:43:27.480816    2019 events.go:65] Failed to handle event stream: rpc error: code = Canceled desc = context canceled
Jan 16 04:43:27 tiny cri-containerd[2019]: E0116 04:43:27.480841    2019 service.go:198] Failed to start streaming server: http: Server closed
Jan 16 04:43:27 tiny systemd[1]: Stopped Kubernetes containerd CRI shim.
Jan 16 04:43:27 tiny systemd[1]: Started Kubernetes containerd CRI shim.
Jan 16 04:43:27 tiny cri-containerd[4257]: I0116 04:43:27.593002    4257 cri_containerd.go:100] Run cri-containerd &{Config:{ContainerdConfig:{RootDir:/var/lib/containerd Snapshotter:overlayfs Endpoint:/run/containerd/containerd.sock Runt
Jan 16 04:43:27 tiny cri-containerd[4257]: time="2018-01-16T04:43:27Z" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-kuberouter.conf"
Jan 16 04:43:27 tiny cri-containerd[4257]: time="2018-01-16T04:43:27Z" level=info msg="CNI network kubernetes (type=bridge) is used from /etc/cni/net.d/10-kuberouter.conf"
Jan 16 04:43:33 tiny cri-containerd[4257]: time="2018-01-16T04:43:33Z" level=info msg="Got pod network {Name:kube-dns-574cf969f4-c64mq Namespace:kube-system ID:86a2d5ac7915d99c7388e825635140e8e8936e143dd4fc82ca24556e1470bf16 NetNS:/var/ru
Jan 16 04:43:33 tiny cri-containerd[4257]: time="2018-01-16T04:43:33Z" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Jan 16 04:43:33 tiny cri-containerd[4257]: time="2018-01-16T04:43:33Z" level=info msg="Got pod network {Name:kube-dns-574cf969f4-c64mq Namespace:kube-system ID:86a2d5ac7915d99c7388e825635140e8e8936e143dd4fc82ca24556e1470bf16 NetNS:/var/ru
Jan 16 04:43:33 tiny cri-containerd[4257]: time="2018-01-16T04:43:33Z" level=info msg="About to add CNI network kubernetes (type=bridge)"

[tkellen]

Let me also share the contents of /etc/cni/net.d/10-kuberouter.conf before and after kube-router is started. I'm candidly a bit out of my depth here so apologies if this is noise.

Placed on the host during cluster bootstrapping:

{
  "name": "kubernetes",
  "type": "bridge",
  "bridge": "kube-bridge",
  "ipam": {
    "type": "host-local"
  },
  "isDefaultGateway": true
}

...how kube-router modifies it during startup (just adding the subnet):

{
  "bridge": "kube-bridge",
  "ipam": {
    "subnet": "10.1.0.0/24",
    "type": "host-local"
  },
  "isDefaultGateway": true,
  "name": "kubernetes",
  "type": "bridge"
}

[tkellen]

@abhi Is there anything else I can provide that would be useful here? You mentioned having me open a bug but I'm not sure where to do that.

[Random-Liu]

Based on the code, CNI should be updated with latest configure. If it doesn't work, it might be a bug.

@tkellen I mean our cni package uses fsnotify to watch cni config directory. If kube-router changes it, cni should pick up the latest configuration. If that doesn't work, we may want to take a look what is wrong. @abhi Do you want to take a look at this? :)

[ijc]

There are debug prints for every event from the watcher (logged as logrus.Debugf) so with debugging turned up @tkellen ought to be abkle to see those happening whenever something is changed in the pluginDir (default is /etc/cni/net.d).

A long time ago I think I observed that if /etc/cni/net.d didn't already exist when the watcher was started then the watcher did not initialise correctly. Could that be happening here I wonder?

[ijc]

A long time ago I think I observed that if /etc/cni/net.d didn't already exist when the watcher was started > then the watcher did not initialise correctly. Could that be happening here I wonder?

No, looks like there is an explicit check and error for that case these days.