[Bug] Pod stuck at FailedMount state

[oh0873]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

A pod may stuck at FailedMount indefinitely.

If a driver pod is created but the config map is not created, the driver pod gets stuck at FailedMount state indefinitely.

This seems to happen when kyuubi-server crashes after creating a driver pod but before config map is mounted.

This was observed in 1.10.3.

Affects Version(s)

1.10.3

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[wangzhigang1999]

Thanks for filing this @oh0873. Before any fix lands I'd like to pin down a few things.

1. What Spark version are you running with Kyuubi 1.10.3?

The race window you describe lives in Spark's KubernetesClientApplication.Client.run(). The creation order is:

1. preKubernetesResources (does not include the driver conf ConfigMap)
2. driver Pod is created — its spec already references the not-yet-existing *-driver-conf-map as a volume
3. ownerRefs added back to pre-resources
4. driver ConfigMap (and other "after-pod" resources) are created, with ownerRef → Pod

Sources: Spark v3.5.3, Spark master.

If the spark-submit JVM (the Kyuubi server) is killed between step 2 and step 4 (SIGKILL / OOM / host crash / restart), the result is:

• Pod exists in apiserver, referencing a ConfigMap that will never be created
• Pod has no ownerReferences → K8s GC won't reclaim it
• The CM was never created → its ownerRef-cascade can't help either
• Kyuubi's appInfoStore is in-memory, so after restart Kyuubi has no record of this pod either

Upstream is aware but unfixed, and the same code path is present in master:

• SPARK-38079 — Not waiting for configmap before starting driver
• SPARK-44050 — Unable to Mount ConfigMap in Driver Pod (~5% submission impact reported on 3.3.1)

Anecdotally we've seen this hit older Spark releases more often than 3.5+, but I don't have hard numbers — so treat the version question as "useful context", not "newer is safer".

2. Is this scoped to FailedMount, or to orphan driver pods in general?

The same crash window can produce orphans with other reasons (ImagePullBackOff, FailedScheduling, terminal pods no longer tracked by any session). The current title and PR target FailedMount specifically. It would help to know whether this issue is intentionally narrow, or whether the broader orphan-pod class is in scope.

3. How does this interact with kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind?

Kyuubi already has this config (since 1.8.1, default NONE) which intentionally keeps terminated driver pods around for post-mortem investigation — pod events, kubelet logs, the missing CM name. FailedMount is exactly the kind of state operators usually want to preserve. Any cleanup mechanism added on top needs a clear answer for what happens to users who rely on NONE.

4. Verifying that the proposed approach actually targets this scenario.

The orphans described here arise from Kyuubi server crash/restart, after which appInfoStore (in-memory ConcurrentHashMap) is empty. A periodic checker iterating over appInfoStore would not see them. Could you walk through how the proposed fix is expected to clean up the specific pods this issue is about?

[pan3793]

older Spark releases had longer step-2→step-4 latency, while 3.5+ with server-side apply usually closes it in tens of milliseconds.

I made this change in SPARK-43356, but I wasn't aware of that. Are there any docs that explain the latency differences?

[wangzhigang1999]

@pan3793 you're right, withdrawn — I don't have a real basis for that latency claim.

For what it's worth, we've been running Kyuubi + Spark on K8s at scale and haven't seen this in production. Doesn't disprove the race, but suggests it's not common — which is why I'd like to understand @oh0873's environment first.

[pan3793]

@wangzhigang1999, we observed SPARK-44050 behavior with K8s 1.19 - a quite old version.

When creating Kubernetes resources, we occasionally encounter situations where resources such as ConfigMap cannot be successfully created, resulting in the driver pod remaining in the 'ContainerCreating' state.

I suspect it's a K8s side issue, as a workaround, we added a retry logic for creating the ConfigMap on the Spark side.

BTW, AI is good at coding and diagnoses, but please filter the slop out and try to keep words short and accurate before posting publicly.