【producer】deadlock on metadata requery

[hulaxiaodai]

It rarely appears in my production env.Bloew is bt ,looks like had a deadlook, but I do not know how to reproduce it, or what sence it can be looked.

#0 0x00007ff3db5bcd20 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x000000000061ea50 in rwlock_wrlock ()
#2 0x00000000006289db in rd_kafka_metadata_refresh_topics ()
#3 0x0000000000628d0e in rd_kafka_metadata_refresh_known_topics ()
#4 0x00000000005bb1b6 in rd_kafka_broker_fail ()
#5 0x00000000005bb7b1 in rd_kafka_broker_timeout_scan ()
#6 0x00000000005c15df in rd_kafka_broker_serve ()
#7 0x00000000005c1dff in rd_kafka_broker_producer_serve ()
#8 0x00000000005c7260 in rd_kafka_broker_thread_main ()
#9 0x000000000061e6a7 in _thrd_wrapper_function ()
#10 0x00007ff3db5b9aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff3db306bcd in clone () from /lib64/libc.so.6

Thread 99 (Thread 0x7ff303fff700 (LWP 21783)):
#0 0x00007ff3db5bcd20 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x000000000061ea50 in rwlock_wrlock ()
#2 0x00000000006289db in rd_kafka_metadata_refresh_topics ()
#3 0x0000000000628d0e in rd_kafka_metadata_refresh_known_topics ()
#4 0x00000000005bb1b6 in rd_kafka_broker_fail ()
#5 0x00000000005bb2b8 in rd_kafka_broker_conn_closed ()
#6 0x00000000005bd51a in rd_kafka_recv ()
#7 0x00000000005e030b in rd_kafka_transport_io_event ()
#8 0x00000000005e0459 in rd_kafka_transport_io_serve ()
#9 0x00000000005c159d in rd_kafka_broker_serve ()
#10 0x00000000005c1dff in rd_kafka_broker_producer_serve ()
#11 0x00000000005c7260 in rd_kafka_broker_thread_main ()
#12 0x000000000061e6a7 in _thrd_wrapper_function ()
#13 0x00007ff3db5b9aa1 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ff3db306bcd in clone () from /lib64/libc.so.6

Thread 98 (Thread 0x7ff3035fe700 (LWP 21784)):
#0 0x00007ff3db5bcd20 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x000000000061ea50 in rwlock_wrlock ()
#2 0x00000000006289db in rd_kafka_metadata_refresh_topics ()
#3 0x0000000000628d0e in rd_kafka_metadata_refresh_known_topics ()
#4 0x00000000005bb1b6 in rd_kafka_broker_fail ()
#5 0x00000000005bb2b8 in rd_kafka_broker_conn_closed ()
#6 0x00000000005bd51a in rd_kafka_recv ()
#7 0x00000000005e030b in rd_kafka_transport_io_event ()
#8 0x00000000005e0459 in rd_kafka_transport_io_serve ()
#9 0x00000000005c159d in rd_kafka_broker_serve ()
#10 0x00000000005c1dff in rd_kafka_broker_producer_serve ()
#11 0x00000000005c7260 in rd_kafka_broker_thread_main ()
#12 0x000000000061e6a7 in _thrd_wrapper_function ()
#13 0x00007ff3db5b9aa1 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ff3db306bcd in clone () from /lib64/libc.so.6

Thread 97 (Thread 0x7ff302bfd700 (LWP 21785)):
#0 0x00007ff3db5bcd20 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x000000000061ea50 in rwlock_wrlock ()
#2 0x00000000006289db in rd_kafka_metadata_refresh_topics ()
#3 0x0000000000628d0e in rd_kafka_metadata_refresh_known_topics ()
#4 0x00000000005bb1b6 in rd_kafka_broker_fail ()
#5 0x00000000005bb2b8 in rd_kafka_broker_conn_closed ()
#6 0x00000000005bd51a in rd_kafka_recv ()
#7 0x00000000005e030b in rd_kafka_transport_io_event ()
#8 0x00000000005e0459 in rd_kafka_transport_io_serve ()
#9 0x00000000005c159d in rd_kafka_broker_serve ()
#10 0x00000000005c1dff in rd_kafka_broker_producer_serve ()
#11 0x00000000005c7260 in rd_kafka_broker_thread_main ()
#12 0x000000000061e6a7 in _thrd_wrapper_function ()

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): v0.11.5
• Apache Kafka version: 0.8.2.1 & 0.10.2
• librdkafka client configuration: metadata.broker.list=xxxx api.version.request=false broker.version.fallback=0.8.2.1 queue.buffering.max.messages=300000 message.max.bytes=4000000 topic.metadata.refresh.interval.ms=300000 metadata.max.age.ms=1500000 queue.buffering.max.ms=50 batch.num.messages=1000 message.send.max.retries=20 message.timeout.ms=900000 request.required.acks=1 request.timeout.ms=900000 socket.timeout.ms=300000 socket.max.fails=1 max.in.flight.requests.per.connection=5 log.connection.close=false socket.keepalive.enable=true socket.send.buffer.bytes=512000 socket.receive.buffer.bytes=512000
• Operating system: Centos 6 (6x64)>

[edenhill]

Thank you for the report.
I believe there should be at least two more threads in this list, namely the rdkafka main thread and an application thread. Looking at the list here does not provide any clues as to what is holding the rd_kafka_t rw-lock.