Skip to content

Kafka: Application maximum poll interval exceeded #893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
KonstantinCodes opened this issue Jun 8, 2019 · 7 comments
Closed

Kafka: Application maximum poll interval exceeded #893

KonstantinCodes opened this issue Jun 8, 2019 · 7 comments
Labels

Comments

@KonstantinCodes
Copy link

KonstantinCodes commented Jun 8, 2019

My consumer

        consumer:
            dsn: "..."
            global:
                ### Make sure this is unique for each application / consumer group and does not change
                ### Otherwise, Kafka won't be able to track your last offset and will always start according to
                ### `auto.offset.reset` setting.
                ### See Kafka documentation regarding `group.id` property if you want to know more
                group.id: 'backend'
                metadata.broker.list: '...'
                security.protocol: sasl_ssl
                ssl.ca.location: '%kernel.project_dir%/config/kafka/ca.pem'
                sasl.username: '...'
                sasl.password: '...'
                sasl.mechanisms: "SCRAM-SHA-256"
                max.poll.interval.ms: '25000'
                enable.auto.commit: 'false'
                enable.auto.offset.store: 'false'
            topic:
                auto.offset.reset: beginning
            ### Commit async is true by default since version 0.9.9.
            ### It is suggested to set it to true in earlier versions since otherwise consumers become extremely slow,
            ### waiting for offset to be stored on Kafka before continuing.
            commit_async: true

Is always producing these messages:

%4|1560006029.619|MAXPOLL|rdkafka#consumer-21| [thrd:main]: Application maximum poll interval (25000ms) exceeded by 333ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1560006054.153|MAXPOLL|rdkafka#consumer-23| [thrd:main]: Application maximum poll interval (25000ms) exceeded by 333ms (adjust max.poll.interval.ms for long-running message processing): leaving group

Do you have an idea, how to fix this?

@Steveb-p
Copy link
Contributor

Steveb-p commented Jun 9, 2019

@KonstantinCodes you're running a long running producer of some sort?

What phprdkafka / librdkafka combination you're running?

Overall, it is kind of related to #749 (not running poll specifically). They are warnings, so your application should work despite them (confirm?), but obviously we should do something about it.

EDIT: I've looked up the warning message and - as I was kinda expecting - that this message is coming from a change introduced for librdkafka 1.0.

@KonstantinCodes
Copy link
Author

@Steveb-p yeah it's kind of interesting.

I'm running a long running consumer, but the messages start popping up after 25 seconds.

So from what I read, you just have to run KafkaConsumer::consume($timeout). Where the $timeout is smaller than your kafka config value max.poll.interval.ms.

I have tried configuring the enqueue transport this way, but for some reason it's not working. I debugged it but didn't come to an conclusion. So I built my own KafkaTransport for Symfony Messenger, and there it's working fine. So I'm really puzzled, how this is happening. Mind you, I'm specifying the exact same config values for both.

I might push the code, just to provide a reference.

@Steveb-p
Copy link
Contributor

I might push the code, just to provide a reference.

Please do, but overall it should work the same. After all, it should be the same KafkaConsumer class. It should be something not really related to enqueue, but PhpRdKafka itself. Gotta do some digging 😄

@stale
Copy link

stale bot commented Jul 18, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@dirk39
Copy link
Contributor

dirk39 commented Sep 12, 2019

This issue should be solved by #947

@Steveb-p
Copy link
Contributor

This issue should be solved by #947

Probably yes actually. Add "Fixes #893" to that PR so this will close automatically once it's merged :)

@Steveb-p Steveb-p reopened this Sep 12, 2019
@stale stale bot removed the wontfix label Sep 12, 2019
@stale
Copy link

stale bot commented Oct 12, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Oct 12, 2019
@stale stale bot closed this as completed Oct 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants