Improve assignement of kafka source pipelines on several indexers

[fmassot]

A user reported the following issue (I checked the cluster state):

• He has around 50 Kafka sources. The biggest Kafka source has 4 desired pipelines, most of which have one.
• He has 10 indexers, each with 4-6 pipelines. 1 indexer has all the pipelines of the biggest Kafka source.

This is bad because, as you can imagine, only 1 indexer is responsible for indexing the biggest Kafka source. And all other indexers are doing less work.

[fulmicoton]

so the merge are messing things up?
Should we consider this a duplicate of #4470

[fulmicoton]

Replied offline by @fmassot : no the problem is an unbalanced load on the different nodes.

[axzsd]

Is this fixed now?

[rdettai]

Not yet, but it is under active discussion in #5747