Allow update of retention.ms during topic modification via NewTopic

[anycz]

Affects Version(s): 2.3.7.RELEASE

I'm trying to modify topic configuration after the creation of the topic (mainly retention.ms in my case, but why not other properties).

When creating a NewTopic, configuration is set correctly at creation by KafkaAdmin.addTopics. When increasing the partitions in a NewTopic, after creation, they are updated automatically by KafkaAdmin.modifyTopics.

The idea would be to update KafkaAdmin.modifyTopics to also modify retention.ms (or possibly others in NewTopics.configs).

[garyrussell]

If your broker is 2.3 or later, you can do this today with

try (AdminClient client = AdminClient.create(kafkaAdmin.getConfig()) {
    ...
    client.incrementalAlterConfigs(configs, options).all().get(10, TimeUnit.SECONDS);
}

I am not convinced that we should do this unconditionally each time an application starts but I suppose we could consider adding a describeConfigs call and comparing them to the current config.

Contributions are welcome.

[anycz]

Thanks for the answer.

Nothing pressing on this point on my side, I just thought it would be a nice addition to spring-kafka in the long run. I had already gone with a new class based that basically does what you're proposing (new class based on KafkaAdmin that read the current configuration and call incrementalAlterConfigs).

It is in kotlin and I simplified edge cases a lot though (it assumes that fatalIfBrokerNotAvailable is always true / doesn't allow calls to initialize after startup / only handles retention.ms and not all possibilities in NewTopic.configs).

Sadly, I probably won't have time to contribute on this point, especially with the edge cases to think about on whether we should take special care when updating some of the configs (like what is done on re-partitioning)

[wilkinsona]

@garyrussell Should this be moved into the 2.8.7 milestone?

[garyrussell]

@wilkinsona Done; thanks.