40_microshift_running_check.sh does not allow additional components in `kube-system` namespace

[vraiti]

Hello,

I'm trying to enable Microshift on NVIDIA Jetson. To give Microshift pods access to the iGPU, the "nvidia-device-plugin-daemonset" (found at https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml) must be installed in the kube-admin namespace.

However, if run on an RPM-OSTree system (which I am trying to do), this will cause greenboot-healthcheck.service to fail due to the following lines in 40_microshift_running_check.sh:

declare -a csi_components=('csi-snapshot-controller')
csi_pods_ct=0
for csi_c in "${csi_components[@]}"; do
    if csiComponentShouldBeDeployed "${csi_c}"; then
        (( csi_pods_ct += 1 ))
    fi
done
if [ ${csi_pods_ct} -gt 0 ]; then
    PODS_NS_LIST+=(kube-system)
    PODS_CT_LIST+=("${csi_pods_ct}")
fi

This, if I understand correctly, statically declares csi-snapshot-controller as the only allowed component in kube-system

# Wait for MicroShift core pods to enter ready state
for i in "${!PODS_NS_LIST[@]}"; do
    CHECK_PODS_NS=${PODS_NS_LIST[${i}]}
    CHECK_PODS_CT=${PODS_CT_LIST[${i}]}

    echo "Waiting ${WAIT_TIMEOUT_SECS}s for ${CHECK_PODS_CT} pod(s) from the '${CHECK_PODS_NS}' namespace to be in 'Ready' state"
    if ! wait_for "${WAIT_TIMEOUT_SECS}" namespace_pods_ready; then
        echo "Error: Timed out waiting for ${CHECK_PODS_CT} pod(s) in the '${CHECK_PODS_NS}' namespace to be in 'Ready' state"
        create_fail_marker_and_exit
    fi
done

And then this will cause greenboot-healthcheck.service to fail if there are any other components in the kube-system namespace.

Could a configuration be added for the expected components in kube-system? I'd happily make a pull request myself to add this feature, but I couldn't find the source code for 40_microshift_running_check.sh anywhere in this repository

[ggiguash]

The greenboot check scripts can be found in this sub-directory. Note that we're renaming those scripts to ensure execution ordering when we package them in RPMs.

It looks like you're using an older version of MicroShift (4.18 or earlier) because we changed the greenboot check code in 4.19 onwards by moving it inside the microshift executable.
See this document for more information.

[ggiguash]

Incidentally, we're already testing nvidia-device-plugin in our CI (see this script), but it's using nvidia-device-plugin-privileged-with-service-account.yml.

This file deploys the plugin into nvidia-device-plugin namespace, which might be a cleaner approach in general.

You can also consider trying nvidia-device-plugin with MicroShift 4.19 to see if the new greenboot check procedure addresses your issue. This is likely to work because our new greenboot check procedure is checking deployment availability rather than specific containers that are running inside each namespace.