WIP: add golang-based containerd setup helper

[ffromani]

WIP

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ffromani

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ffromani]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fmuyassarov]

Thanks @ffromani, I know it is still WIP but probably good time to ask few questions.

1. I'm wondering if you have considered using dbus (via go-systemd) directly instead of host systemctl?
2. Would it be rather convenient from the user's perspective to have an optional init container for runtime config (along side the driver Pod) changes rather than the Job?
3. Should be also try to cover the CRI-O?

[fmuyassarov]

or in case when NRI/CDI is disabled prior to driver installation.

[ffromani]

Thanks @ffromani, I know it is still WIP but probably good time to ask few questions.

It surely is!

1. I'm wondering if you have considered using dbus (via [go-systemd](https://github.com/coreos/go-systemd)) directly instead of host systemctl?

Nope, I used systemctl because this is what the initial initContainer implementation did and I didn't explored options yet. I think the key factors here are reliability, simplicity and privileges required. If go-systemd gives us a better alternative along these axes, we can surely switch to it.

2. Would it be rather convenient from the user's perspective to have an optional init container for runtime config (along side the driver Pod) changes rather than the Job?

I think the job is better because separation of concerns and separation of required privileges. Unless I'm mistaken, re-introducing a initContainer will also require a privilege bump, which I'd rather avoid.
These are the main reasons I'd be not thrilled about reintroducing a initContainer, but we can surely talk about them, if there are upsides I'm missing I'd be happy to consider them and revisit the decision.

3. Should be also try to cover the CRI-O?

Yes! If we decide to go forward with this approach I will deep dive in crio and add the crio counterpart code.

[fmuyassarov]

I think the job is better because separation of concerns and separation of required privileges. Unless I'm mistaken, re-introducing a initContainer will also require a privilege bump, which I'd rather avoid. These are the main reasons I'd be not thrilled about reintroducing a initContainer, but we can surely talk about them, if there are upsides I'm missing I'd be happy to consider them and revisit the decision.

If we use an init container and dbus, we would need to mount the following paths:

• /etc/containerd/ for the config.toml
• /etc/crio/crio.conf.d/ for CRI-O drop-in configuration
• /var/run/dbus/system_bus_socket if we decide to use dbus, for detecting systemd units of the container runtime and related interactions.

Compared to the current Job configuration, this reduces the exposure of the host filesystem. Instead of mounting the whole /etc, we limit access to only the specific subdirectories.

With Job, we also put operational burden on the user because they need to know the node count, run the Job manually and check if it succeeded on each node. With init container, they can just pass a single flag (e.g. --set initContainer.enabled=true or something) and the chart handles the rest. If the init container fails on a node, the driver pod doesn't start there, which gives an immediate signal rather than a silent misconfiguration.

[fmuyassarov]

With Job, we also put operational burden on the user because they need to know the node count, run the Job manually and check if it succeeded on each node. With init container, they can just pass a single flag (e.g. --set initContainer.enabled=true or something) and the chart handles the rest. If the init container fails on a node, the driver pod doesn't start there, which gives an immediate signal rather than a silent misconfiguration.

I'm not insisting to use dbus specifically or init container over Job. I just want to make sure that
we pick the best option from what's available to us, so happy to hear other thoughts.

[ffromani]

I think the job is better because separation of concerns and separation of required privileges. Unless I'm mistaken, re-introducing a initContainer will also require a privilege bump, which I'd rather avoid. These are the main reasons I'd be not thrilled about reintroducing a initContainer, but we can surely talk about them, if there are upsides I'm missing I'd be happy to consider them and revisit the decision.

If we use an init container and dbus, we would need to mount the following paths:

* `/etc/containerd/` for the config.toml

* `/etc/crio/crio.conf.d/` for CRI-O drop-in configuration

* `/var/run/dbus/system_bus_socket` if we decide to use dbus, for detecting systemd units of the container runtime and related interactions.

Compared to the current Job configuration, this reduces the exposure of the host filesystem. Instead of mounting the whole /etc, we limit access to only the specific subdirectories.

With Job, we also put operational burden on the user because they need to know the node count, run the Job manually and check if it succeeded on each node. With init container, they can just pass a single flag (e.g. --set initContainer.enabled=true or something) and the chart handles the rest. If the init container fails on a node, the driver pod doesn't start there, which gives an immediate signal rather than a silent misconfiguration.

These are all good points. The operational mode using dbus has surely upsides worth a serious look, thanks for raising. The delivery mechanism - jobs vs initContainer I'm still not thrilled but I do see your point. Let's iterate with this approach a bit more, I won't be firmly against a opt-in, disabled-by-default init container task even though I'd really prefer a stronger separation of concerns if possible. But "if possible" is the key here.

[k8s-ci-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.