librdkafka 0.8 can not produce messages when the leader of the topic partition failed

[zbweng]

Hi, Magnus.
I find that librdkafka 0.8 can not produce messages to the kafka brokers when the leader of the topic partition failed. After I check the new leader is elected, the problem is not yet solved.
I use the rdkafka_example.c in the example folder and create a topic with 3 replicas and 1 partition.
After I restart the producer (rdkafka_example.c, I use it as a producer), the problem is solved.
I am not sure whether it is a bug for librdkafka 0.8.
Thanks.

[edenhill]

I will push a fix for this soon.

[zbweng]

Hi, Magnus.
I test for this issue on your latest code again. However, I find that this issue is not yet solved.
I use pkill to stop the leader of a topic partition, then rdkafka cannot produce messages.

[edenhill]

You are right.
librdkafka polls for topic leader as soon as a produce request for a topic fails, but the new leader has probably not been elected yet.
I'll add a timer to retry the poll until a leader is found.

[edenhill]

Give it another shot now on master branch.
It now regularily refreshes the broker list (metadata), and after a leader failure it aggressively refreshes it 10 times with a 250ms interval (default config).

[zbweng]

Great! I test for this issue again and find that it is solved.