[Bug] spark.authenticate.secret is not picked up in spark standalone with cluster mode

[karl-ubm]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

Hi,

Thanks for a great program!

When running a spark standalone cluster with security enable, it fails with:

java.lang.IllegalArgumentException: requirement failed: A secret key must be specified via the spark.authenticate.secret config.

though it works with client mode.

I believe the issue is this: https://issues.apache.org/jira/browse/SPARK-25078 (marked as resolved, but it's not according to the last comment.)

Do you think it would be possible to implement that workaround described there?

Affects Version(s)

1.11.0

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @karl-ubm,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[pan3793]

@karl-ubm, I don't have experience with an auth-enabled Spark app, so I'm not sure if such a workaround will introduce any side effects. it's clearly a Spark bug, I suggest you bring up this issue in the Spark dev mailing list to get more attention from developers who focus on this area.

[pan3793]

BTW, I re-opened SPARK-25078