Intermittent Hangs at crane.Push() on Registry Push

[ranimbal]

Environment

Device and OS: Rocky 8 EC2
App version: 0.29.2
Kubernetes distro being used: RKE2 v1.26.9+rke2r1
Other: Bigbang v2.11.1

Steps to reproduce

1. zarf package deploy zarf-package-mvp-cluster-amd64-v5.0.0-alpha.7.tar.zst --confirm -l=debug
2. About 80% of the time or so, the above command gets stuck at crane.Push(). A retry usually works.

Expected result

That the zarf package deploy... command wouldn't get hung up, and continue along.

Actual Result

The zarf package deploy... command gets hung up

Visual Proof (screenshots, videos, text, etc)

��[30;100m�[30;100m  DEBUG  �[0m�[0m �[90m�[90m2023-10-23T18:37:19Z  -  Pushing ...1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3�[0m�[0m
�[30;100m�[30;100m  DEBUG  �[0m�[0m �[90m�[90m2023-10-23T18:37:19Z  -  crane.Push() /tmp/zarf-3272389118/images:registry1.dso.mil/ironbank/neuvector/neuvector/manager:5.1.3 -> 127.0.0.1:39357/ironbank/neuvector/neuvector/manager:5.1.3-zarf-487612511)�[0m�[0m
section_end:1698087620:step_script
�[0K�[31;1mERROR: Job failed: execution took longer than 35m0s seconds

Severity/Priority

There is a workaround, by keeping retrying until the process succeeds.

Additional Context

This looks exactly like #1568, which was closed.

We have a multi-node cluster on AWS EC2, our package size is about 2.9G. Here are a few things that we noticed after some extensive testing:

• this issue is not seen on a single EC2 node RKE2 cluster, it seems to only occur on multi-node clusters.
• our zarf docker registry is backed by S3. The issue is always seen in this case, but only if a multi-node cluster.
• if we back the registry with the default PVC (instead of S3), the issue is not seen at all. Since data transfer to S3 is slower than to the EBS backed PVC, maybe this extra time causes the problem to appear?
• disabling or enabling the zarf docker registry HPA doesn't seem to matter either ways.

[AbrohamLincoln]

I did some testing on this and here's what i found:

• I cannot reproduce this with a single node RKE2 cluster
• I cannot reproduce this with an EKS cluster
• I can reproduce this fairly consistently with RKE2 on a 2+ node cluster (rough math says ~80% of the time)
• I changed the CNI from Canal to Calico. While I did still encounter this issue, my rough math says the failure rate dropped down to less than 20%.

While I have not found a smoking gun for this, the testing I've done seems to indicate it might be related to the default RKE2 CNI.

[Racer159]

Yeah that is what we are leaning to after some internal testing as well - a potentially interesting data point - do you ever see this issue with zarf package mirror-resources?

https://docs.zarf.dev/docs/the-zarf-cli/cli-commands/zarf_package_mirror-resources#examples

(for the internal registry you can take the first example and swap the passwords and the package - if you don't have git configured just omit that set of flags)

[Racer159]

(a potential addition to the theory is that other things in the cluster may be stressing it as well)

[Racer159]

Also what is the node role layout for your clusters - I have heard reports that if all nodes are control plane nodes that the issue is also not seen.

[ranimbal]

Also what is the node role layout for your clusters - I have heard reports that if all nodes are control plane nodes that the issue is also not seen.

We've always had agent nodes when we saw this issue, whether with 1 or 3 control plane nodes. We've never seen this issue on single node clusters. Haven't tried a cluster with only 3 control plane nodes and no agent nodes.

[docandrew]

Just to add another data point from what we've seen - we can deploy OK with multi-node clusters but only if the nodes are all RKE2 servers. As soon as we make one an agent, the Zarf registry runs there and we see this behavior as well.

[docandrew]

Additional agent nodes are OK but we've tainted those so the Zarf registry doesn't run there.

[AbrohamLincoln]

I can confirm that adding a nodeSelector and taint/toleration to schedule the zarf registry pod(s) on the RKE2 control plane node(s) does resolve work around this issue:

kubectl patch deployment -n zarf zarf-docker-registry --patch-file=/dev/stdin <<-EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zarf-docker-registry
  namespace: zarf
spec:
  template:
    spec:
      nodeSelector:
        node-role.kubernetes.io/master: "true"
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
EOF