Implement worker queue limit for outgoing pools by chrzaszcz · Pull Request #4599 · esl/MongooseIM

chrzaszcz · 2025-12-12T16:41:58Z

The goal of this PR is to replace the max_worker_queue_len config option with a new mechanism, because:

It wasn't effective. In a benchmark, I managed to get 380k requests queued per worker even though the option was set to 1000. That's because the limit was applied after the requests were already sent.
It was only for rabbit. We could benefit from a more generic solution.

The new implementation drops the request just like the next_available_worker strategy.

Notes:

The new option supplement best_worker with a hard queue limit.
The rabbit-specific max_worker_queue_len option is removed to avoid confusion and to simplify implementation.
Apart from the unit tests, the new option will have an additional test for RabbitMQ in a follow-up PR. The test coverage will increase after that PR.

codecov · 2025-12-12T16:58:40Z

Codecov Report

❌ Patch coverage is 85.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.03%. Comparing base (fce556b) to head (551ed6b).
⚠️ Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
src/mongoose_rabbit_worker.erl	66.66%	2 Missing ⚠️
src/wpool/mongoose_wpool.erl	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4599      +/-   ##
==========================================
- Coverage   86.03%   86.03%   -0.01%     
==========================================
  Files         565      565              
  Lines       33920    33918       -2     
==========================================
- Hits        29184    29182       -2     
  Misses       4736     4736

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Make sure it is set only for best_worker

If the shortest queue is longer than the limit, exit:no_available_workers is raised (like for next_available_worker).

mongoose-im · 2025-12-15T09:10:17Z

elasticsearch_and_cassandra_28 / elasticsearch_and_cassandra_mnesia / 551ed6b
Reports root/ big
OK: 683 / Failed: 0 / User-skipped: 72 / Auto-skipped: 0

small_tests_27 / small_tests / 551ed6b
Reports root / small

small_tests_28 / small_tests / 551ed6b
Reports root / small

small_tests_28_arm64 / small_tests / 551ed6b
Reports root / small

ldap_mnesia_27 / ldap_mnesia / 551ed6b
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1362 / Auto-skipped: 0

ldap_mnesia_28 / ldap_mnesia / 551ed6b
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 1362 / Auto-skipped: 0

dynamic_domains_mysql_redis_28 / mysql_redis / 551ed6b
Reports root/ big
OK: 5186 / Failed: 0 / User-skipped: 157 / Auto-skipped: 0

internal_mnesia_28 / internal_mnesia / 551ed6b
Reports root/ big
OK: 2502 / Failed: 0 / User-skipped: 1218 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 551ed6b
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_28 / pgsql_mnesia / 551ed6b
Reports root/ big
OK: 5221 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

pgsql_cets_28 / pgsql_cets / 551ed6b
Reports root/ big
OK: 5311 / Failed: 0 / User-skipped: 198 / Auto-skipped: 0

mysql_redis_28 / mysql_redis / 551ed6b
Reports root/ big
OK: 5603 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_28 / odbc_mssql_mnesia / 551ed6b
Reports root/ big
OK: 5216 / Failed: 0 / User-skipped: 127 / Auto-skipped: 0

cockroachdb_cets_28 / cockroachdb_cets / 551ed6b
Reports root/ big
OK: 5316 / Failed: 4 / User-skipped: 198 / Auto-skipped: 0

mod_ping_SUITE:server_ping:server_ping_pong

{error,{test_case_failed,"Incorrect number of instrumentation events - matched: 0, expected: 1"}}

Report log

pubsub_SUITE:dag+last_item_cache:send_last_published_item_no_items_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_send_last_published_item_no_items_test_3717@localhost/res1">>,
          escalus_tcp,<0.115329.0>,
          [{event_manager,<0.115319.0>},
           {server,<<"localhost">>},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3717">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.115319.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_send_last_published_item_no_items_test_3717">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_send_last_published_item_no_items_test_3717">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3717">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"74346cbd7eb4265f">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_c...

Report log

pubsub_SUITE:dag+last_item_cache:send_last_published_item_no_items_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_send_last_published_item_no_items_test_3719@localhost/res1">>,
          escalus_tcp,<0.115407.0>,
          [{event_manager,<0.115386.0>},
           {server,<<"localhost">>},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3719">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.115386.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_send_last_published_item_no_items_test_3719">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_send_last_published_item_no_items_test_3719">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3719">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"62234100b05590e6">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_c...

Report log

pubsub_SUITE:dag+last_item_cache:send_last_published_item_no_items_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_send_last_published_item_no_items_test_3722@localhost/res1">>,
          escalus_tcp,<0.115483.0>,
          [{event_manager,<0.115464.0>},
           {server,<<"localhost">>},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3722">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.115464.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_send_last_published_item_no_items_test_3722">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_send_last_published_item_no_items_test_3722">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,
             <<"alicE_send_last_published_item_no_items_test_3722">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"016cc32f9c0bfb29">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_c...

Report log

pgsql_mnesia_27 / pgsql_mnesia / 551ed6b
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 138 / Auto-skipped: 0

pgsql_mnesia_28 / pgsql_mnesia / 551ed6b
Reports root/ big
OK: 5614 / Failed: 0 / User-skipped: 138 / Auto-skipped: 0

mssql_mnesia_28 / odbc_mssql_mnesia / 551ed6b
Reports root/ big
OK: 5609 / Failed: 0 / User-skipped: 143 / Auto-skipped: 0

cockroachdb_cets_28 / cockroachdb_cets / 551ed6b
Reports root/ big
OK: 137 / Failed: 1 / User-skipped: 2 / Auto-skipped: 0

pubsub_SUITE:tree+last_item_cache:send_last_published_item_no_items_test

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_send_last_published_item_no_items_test_142@localhost/res1">>,
          escalus_tcp,<0.4027.0>,
          [{event_manager,<0.4015.0>},
           {server,<<"localhost">>},
           {username,
             <<"alicE_send_last_published_item_no_items_test_142">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.4015.0>},
            {server,<<"localhost">>},
            {username,
              <<"alicE_send_last_published_item_no_items_test_142">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_send_last_published_item_no_items_test_142">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,fun escalus_auth:auth_plain/2},
           {wspath,undefined},
           {username,
             <<"alicE_send_last_published_item_no_items_test_142">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"0a6ede7f682821b3">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {pubsub_tools,receive_response,3,
       [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
        {line,444}]},
     {pubsub_tools,receive_and_check_respon...

Report log

telezynski

Looks good!

This comment was marked as outdated.

Sign in to view

chrzaszcz force-pushed the max-worker-queue-len branch 2 times, most recently from bd16c7e to 89f1d24 Compare December 15, 2025 08:36

This comment was marked as outdated.

Sign in to view

chrzaszcz added 3 commits December 15, 2025 09:50

Move max_worker_queue_len from rabbit to common worker pool options

054f0eb

Make sure it is set only for best_worker

Introduce best_worker strategy with queue limit

c7c8a98

If the shortest queue is longer than the limit, exit:no_available_workers is raised (like for next_available_worker).

Remove the queue limit from mongoose_rabbit_worker

6a3fc2e

chrzaszcz force-pushed the max-worker-queue-len branch from 89f1d24 to abf9204 Compare December 15, 2025 08:57

chrzaszcz added 2 commits December 15, 2025 10:00

Update small tests with the new max_wotker_queue_len implementation

0ab243b

Update docs for max_worker_queue_len

551ed6b

chrzaszcz force-pushed the max-worker-queue-len branch from abf9204 to 551ed6b Compare December 15, 2025 09:00

This comment was marked as outdated.

Sign in to view

chrzaszcz marked this pull request as ready for review December 15, 2025 10:59

telezynski approved these changes Dec 15, 2025

View reviewed changes

telezynski merged commit 8a90671 into master Dec 15, 2025
3 of 4 checks passed

telezynski deleted the max-worker-queue-len branch December 15, 2025 12:25

chrzaszcz added this to the 6.6.0 milestone Mar 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement worker queue limit for outgoing pools#4599

Implement worker queue limit for outgoing pools#4599
telezynski merged 5 commits intomasterfrom
max-worker-queue-len

chrzaszcz commented Dec 12, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

codecov bot commented Dec 12, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Dec 15, 2025 •

edited

Loading

Uh oh!

telezynski left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

chrzaszcz commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

codecov bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

telezynski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chrzaszcz commented Dec 12, 2025 •

edited

Loading

codecov bot commented Dec 12, 2025 •

edited

Loading

mongoose-im commented Dec 15, 2025 •

edited

Loading