Created node never joins the cluster

[fdtmsft]

Following the example for self-hosted I have created my nodepool with Ubuntu 22.04 and I do see Karpenter (0.7,2) creates a node when I scale my pause deployment, but the new node that I see running on Azure never joins the cluster.

Doing some research here I have tried to update using az aks update -n cluster_name -g cluster_resource_group which successfully completed but yet still no node appears.

I have also checked the karpenter-values .yaml file contains the correct bootstrap tokens of the cluster. Any suggestions of what could be the issue?

[MattMencel]

@fdtmsft The az aks update hack only works for NAP.

I switched back to NAP from self-hosted pretty quickly. The bootstrap tokens reset every few days and there is no easy(automated) way to keep them aligned in the self-hosted configuration. So the scaling just stops until a human goes in and updates the bootstrap token in the config.

So IMO.... just use NAP. At least with NAP, if the token expires you can fix it easily and put some automation around the az aks update command. Also for this reason alone... let alone that this is a preview feature, even with NAP, I would not recommend karpenter for production workloads.

Maybe in the future this will change, but for now I don't think self-hosted is an option unless you just want to see the components deployed in their native state.

[Bryce-Soghigian]

You shouldn't need to do that at all with nap now. That was just initially. I agree that without some automated token rotation in place managing the helm values for the bootstrapping token are a major pain for self hosted making it non-viable in production environments.

CC: @charliedmcb

[MattMencel]

@Bryce-Soghigian I just experienced this again today and had to az aks update to get it scaling again. It may be related to the cluster doing automatic patch upgrades. I schedule these to happen on the weekends, and I notice the problem always needs to be addressed on Mondays for me.

I believe my cluster upgraded to v1.30.9 this weekend. I'll note if it occurs again and whether it happens the next time the cluster does an upgrade.

[maulik13]

Is there any way to debug this for NAP based clusters? Which longs should we look for from the control plane?

[niyoko]

Actualy I was able to run self hosted karpenter for production workloads smoothly for the last 2 months.
The trick is using our own bootstrap token, managed by external secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-boostrap-token-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-kv-store
  target:
    creationPolicy: Owner
    name: bootstrap-token-karp01
    template:
      engineVersion: v2
      type: bootstrap.kubernetes.io/token
      metadata:
        labels:
          kubeadm.kubernetes.io/component: bootstrap-token
        annotations:
          kubernetes.io/description: "Karpenter bootstrap token"
      data:
        token-id: karp01
        token-secret: "{{ `{{ .token }}` }}"
        usage-bootstrap-authentication: 'true'
        usage-bootstrap-signing: 'true'
  data:
    - secretKey: token
      remoteRef:
        key: karpenter-bootstrap-token

Then in the helm values just use referred secret

      - name: KUBELET_BOOTSTRAP_TOKEN_SECRET
        valueFrom:
          secretKeyRef:
            name: bootstrap-token-karp01
            key: token-secret
      - name: KUBELET_BOOTSTRAP_TOKEN
        value: "karp01.$(KUBELET_BOOTSTRAP_TOKEN_SECRET)"

But today, the nodes created by karpenter suddenly fail to join the cluster.

[maulik13]

@niyoko do you have any logs or indicators that could provide some insight? We are considering using self-hosted karpenter since the AKS managed one is creating so much issues for us.

Also which region are you using for your cluster?

[tallaxes]

The most recent failures may be related to #723

[pepordev]

Actualy I was able to run self hosted karpenter for production workloads smoothly for the last 2 months. The trick is using our own bootstrap token, managed by external secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-boostrap-token-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-kv-store
  target:
    creationPolicy: Owner
    name: bootstrap-token-karp01
    template:
      engineVersion: v2
      type: bootstrap.kubernetes.io/token
      metadata:
        labels:
          kubeadm.kubernetes.io/component: bootstrap-token
        annotations:
          kubernetes.io/description: "Karpenter bootstrap token"
      data:
        token-id: karp01
        token-secret: "{{ `{{ .token }}` }}"
        usage-bootstrap-authentication: 'true'
        usage-bootstrap-signing: 'true'
  data:
    - secretKey: token
      remoteRef:
        key: karpenter-bootstrap-token

Then in the helm values just use referred secret

      - name: KUBELET_BOOTSTRAP_TOKEN_SECRET
        valueFrom:
          secretKeyRef:
            name: bootstrap-token-karp01
            key: token-secret
      - name: KUBELET_BOOTSTRAP_TOKEN
        value: "karp01.$(KUBELET_BOOTSTRAP_TOKEN_SECRET)"

But today, the nodes created by karpenter suddenly fail to join the cluster.

we do the same but with the kubernetes provider. Something like:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-join-token
  namespace: kube-system
spec:
  refreshInterval: 5m
  secretStoreRef:
    kind: ClusterSecretStore
    name: adp-k8s
  target:
    name: karpenter-join-token
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        KUBELET_BOOTSTRAP_TOKEN: '{{ index (fromJson .token) "token-id" }}.{{ index (fromJson .token) "token-secret" }}'
  dataFrom:
  - find:
      name:
        regexp: "bootstrap-.*"
    rewrite:
    - regexp:
        source: "(.*)"
        target: "token"

Then configure karpenter to use the secret for the env var

  envFrom:
    - secretRef:
        name: karpenter-join-token

We also use reloader to automatically restart karpenter in case the secret is updated.

[superff]

Actualy I was able to run self hosted karpenter for production workloads smoothly for the last 2 months. The trick is using our own bootstrap token, managed by external secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-boostrap-token-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-kv-store
  target:
    creationPolicy: Owner
    name: bootstrap-token-karp01
    template:
      engineVersion: v2
      type: bootstrap.kubernetes.io/token
      metadata:
        labels:
          kubeadm.kubernetes.io/component: bootstrap-token
        annotations:
          kubernetes.io/description: "Karpenter bootstrap token"
      data:
        token-id: karp01
        token-secret: "{{ `{{ .token }}` }}"
        usage-bootstrap-authentication: 'true'
        usage-bootstrap-signing: 'true'
  data:
    - secretKey: token
      remoteRef:
        key: karpenter-bootstrap-token

Then in the helm values just use referred secret

      - name: KUBELET_BOOTSTRAP_TOKEN_SECRET
        valueFrom:
          secretKeyRef:
            name: bootstrap-token-karp01
            key: token-secret
      - name: KUBELET_BOOTSTRAP_TOKEN
        value: "karp01.$(KUBELET_BOOTSTRAP_TOKEN_SECRET)"

But today, the nodes created by karpenter suddenly fail to join the cluster.

we do the same but with the kubernetes provider. Something like:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: karpenter-join-token
namespace: kube-system
spec:
refreshInterval: 5m
secretStoreRef:
kind: ClusterSecretStore
name: adp-k8s
target:
name: karpenter-join-token
creationPolicy: Owner
template:
engineVersion: v2
data:
KUBELET_BOOTSTRAP_TOKEN: '{{ index (fromJson .token) "token-id" }}.{{ index (fromJson .token) "token-secret" }}'
dataFrom:

• find:
  name:
  regexp: "bootstrap-.*"
  rewrite:
  • regexp:
    source: "(.*)"
    target: "token"
    Then configure karpenter to use the secret for the env var

envFrom:
- secretRef:
name: karpenter-join-token
We also use reloader to automatically restart karpenter in case the secret is updated.

Hello, @ppodevlabs how does datafrom part work? does it read from a secret in the key vault?

  dataFrom:
  - find:
      name:
        regexp: "bootstrap-.*"
    rewrite:
    - regexp:
        source: "(.*)"
        target: "token"

[pepordev]

Actualy I was able to run self hosted karpenter for production workloads smoothly for the last 2 months. The trick is using our own bootstrap token, managed by external secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-boostrap-token-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-kv-store
  target:
    creationPolicy: Owner
    name: bootstrap-token-karp01
    template:
      engineVersion: v2
      type: bootstrap.kubernetes.io/token
      metadata:
        labels:
          kubeadm.kubernetes.io/component: bootstrap-token
        annotations:
          kubernetes.io/description: "Karpenter bootstrap token"
      data:
        token-id: karp01
        token-secret: "{{ `{{ .token }}` }}"
        usage-bootstrap-authentication: 'true'
        usage-bootstrap-signing: 'true'
  data:
    - secretKey: token
      remoteRef:
        key: karpenter-bootstrap-token

Then in the helm values just use referred secret

      - name: KUBELET_BOOTSTRAP_TOKEN_SECRET
        valueFrom:
          secretKeyRef:
            name: bootstrap-token-karp01
            key: token-secret
      - name: KUBELET_BOOTSTRAP_TOKEN
        value: "karp01.$(KUBELET_BOOTSTRAP_TOKEN_SECRET)"

But today, the nodes created by karpenter suddenly fail to join the cluster.

we do the same but with the kubernetes provider. Something like:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: karpenter-join-token
namespace: kube-system
spec:
refreshInterval: 5m
secretStoreRef:
kind: ClusterSecretStore
name: adp-k8s
target:
name: karpenter-join-token
creationPolicy: Owner
template:
engineVersion: v2
data:
KUBELET_BOOTSTRAP_TOKEN: '{{ index (fromJson .token) "token-id" }}.{{ index (fromJson .token) "token-secret" }}'
dataFrom:

• find:
  name:
  regexp: "bootstrap-.*"
  rewrite:

  • regexp:
    source: "(.*)"
    target: "token"
    Then configure karpenter to use the secret for the env var

envFrom:

• secretRef:
  name: karpenter-join-token
  We also use reloader to automatically restart karpenter in case the secret is updated.

Hello, @ppodevlabs how does datafrom part work? does it read from a secret in the key vault?

dataFrom:

• find:
  name:
  regexp: "bootstrap-.*"
  rewrite:
  • regexp:
    source: "(.*)"
    target: "token"

HI! no, we are using the kubernetes provider so it reads the bootstrap token from the cluster, kube-system namespace

[Bryce-Soghigian]

Closing this issue, but pinning it since community came up with some good solutions to token rotation