MSK updates not working

[kkopuru]

We are trying to update tags, cluster configuration via kafka-controller which was not working as expected. We were able to update certain things like volume size, instance type, etc with no issues. The expectation is to manage or update every config of MSK cluster via kafka-controller.

Environment

• Kubernetes version - 1.30
• Using EKS - yes
• AWS service targeted - MSK
• kafka-controller - 1.0.11

[github-actions]

Hello @kkopuru 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[kkopuru]

@michaelhtm / @eqe-aws thanks for the quick followup on this. Can I assume that fix for this will address updating any of the MSK config not just the tags

[michaelhtm]

Hello @kkopuru
Here are the MSK Cluster configuration updates we currently support
https://github.com/aws-controllers-k8s/kafka-controller/blob/8c540670ce38ab7596cb69334de898d9bc1f8a24/pkg/resource/cluster/hooks.go#L163-L192

Tags we the last ones to be supported here: https://github.com/aws-controllers-k8s/kafka-controller/releases/tag/v1.1.2

Is there a specific one you are trying to update but couldn't?

[kkopuru]

Hi @michaelhtm
We were currently looking to update Spec.configurationInfo.revision in particular.
Is there any reason why the update is limited to certain things, tomorrow for example we may require enabling logging, monitoring.

[michaelhtm]

We mainly try to use code-generator for ACK controllers. Most of them have a straight forward Update operation, which can easily get code-generated, while MSK Cluster has different APIs to update different Spec fields.
We mainly rely on customer feedback/request to catch all the missing APIs we don't support and write custom hooks for them.

We can track the progress for the following update APIs here:
MSK Cluster currently is missing the following update operations:

• UpdateClusterConfiguration
• UpdateClusterKafkaVersion
• UpdateMonitoring

Does this look right?

[kkopuru]

Seems fine for now from what we are doing currently, but what about logging Spec.loggingInfo.brokerLogs.

[michaelhtm]

UpdateMonitoring should handle the updates for LoggingInfo
https://docs.aws.amazon.com/msk/1.0/apireference/clusters-clusterarn-monitoring.html#clusters-clusterarn-monitoring-schemas

[kkopuru]

Thanks @michaelhtm, seems good