Describe the bug

On a 3-node cluster with many at_least_once dead-lettering quorum queues, quorum-queue membership churn (a rolling node replacement: a node leaving, a replacement rejoining, and abrupt node terminations) can permanently strand a subset of the at_least_once DLX queues with a member stuck noproc and no elected leader. Recovery requires a manual ra:restart_server. Plain quorum queues (no DLX, or at_most_once) recover normally via election.

Observed on 4.2.4 (Erlang 27.3.4.2). The relevant code paths are unchanged on main.

Mechanism (from logs and source)

1. During membership churn, many at_least_once DLX queues transition leadership. On entering leader state, rabbit_fifo_dlx:state_enter(leader, ..., at_least_once, ...) calls start_worker, which does {ok, Pid} = supervisor:start_child(rabbit_fifo_dlx_sup, [QRef]).
2. The rabbit_fifo_dlx_worker instances fail to register their {dlx,{checkout}} command against the flapping leader and exit {error, ra_command_failed}, en masse (observed about 80 worker exits in a single second).
3. rabbit_fifo_dlx_sup (simple_one_for_one, intensity => 100, period => 1) becomes unavailable on every node. On one node it logged an explicit reached_max_restart_intensity give-up; on another it stopped servicing calls and subsequent start_child returned noproc.
4. With the DLX worker supervisor unavailable cluster-wide, each at_least_once DLX queue entering leader state crashes its ra_server at the {ok, Pid} = supervisor:start_child(...) line: exit:{noproc,{gen_server,call,[rabbit_fifo_dlx_sup,{start_child,...}]}}. This is the intended behavior per the comment in start_worker/1 ("if starting the rabbit_fifo_dlx_worker fails, let the Ra server process crash in which case another Ra node will become leader"), but the failure is correlated across all nodes, so there is no healthy failover target.
5. The crashing ra_server trips its per-queue ra_server_sup (intensity => 2, period => 5), which after 2 crashes in 5 seconds permanently abandons the member: noproc, no auto-restart. The queue stays stranded until ra:restart_server is run manually.

In our incident this stranded 52 at_least_once DLX queues; ra_server_sup logged 500+ max-restart-intensity give-ups per surviving node, while rabbit_fifo_dlx_sup itself gave up at most once per node.

Why existing mitigations do not cover this

• PR Improve rabbit_fifo_dlx_worker resiliency #7677 raised rabbit_fifo_dlx_sup to intensity 100, period 1 ("do not terminate the sup if a low number of children terminate") and added "last checkout wins" and "do not restart the DLX worker if the leader is non-local". Under DLX churn the 100/1 budget is still exceeded, or the supervisor is otherwise left unavailable.
• PR QQ: when invoking drain only shut down small batches at a time #14401 (in 4.2.0, present in our 4.2.4 build) batches leadership transfers during graceful drain. Our churn came from node rejoin and abrupt termination, which do not go through that path, so it did not help.
• PR Fix dlx checkout command redirection #16203 (merged 2026-04-23, backported to v4.2.x/v4.3.x, not present in our 4.2.4 build dated 2026-05-12) switches the DLX checkout command from ra:process_command/3 to ra:pipeline_command/4 with a selective receive, preventing Ra from redirecting the checkout to a new leader during failover. This reduces the rate of {error, ra_command_failed} worker crashes (workers exit with non_local_leader instead of retrying 5 times), making the mass-crash less likely. However, the checkout0 retry loop can still timeout and return ra_command_failed under sustained churn, so the structural problem remains: if enough workers crash simultaneously from any cause, rabbit_fifo_dlx_sup still dies and the same cascade occurs.
• Discussion 4.1.4: a rolling restart leaves 2 quorum queues out of 1600 without an elected leader #14780 reports a closely related symptom (a rolling restart leaving quorum queues without an elected leader); the underlying cause was not pinned there.

Reproduction steps

Tested on a 3-node cluster running RabbitMQ 4.2 (Erlang 27.3.4.2).

Setup: a quorum queue with dead-letter-strategy: at-least-once and overflow: reject-publish (required for at-least-once to activate), plus a dead-letter exchange and target queue.

Step 1 below uses unregister(rabbit_fifo_dlx_sup) to simulate the state that results from the supervisor reaching max restart intensity. This was observed on a cluster with ~564 at_least_once DLX queues with membership churn (80+ workers crashing with {error, ra_command_failed} in a 1-second window, exhausting the supervisor's intensity 100/period 1 budget). The unregister is a shortcut to reach that end state deterministically without requiring the conditions that kill the supervisor in the wild.

rabbitmqctl eval '
  %% 1. Unregister rabbit_fifo_dlx_sup so start_child returns noproc.
  %%    Simulates the supervisor being unavailable after reaching max
  %%    restart intensity. The process stays alive so rabbit_sup
  %%    (one_for_all, intensity 0) does not notice and the node stays up.
  unregister(rabbit_fifo_dlx_sup).
'

rabbitmqctl eval '
  %% 2. Transfer leadership away then back to trigger state_enter(leader).
  QName = rabbit_misc:r(<<"vhost">>, queue, <<"my_queue">>),
  {ok, Q} = rabbit_amqqueue:lookup(QName),
  Members = amqqueue:get_nodes(Q),
  OtherNodes = [N || N <- Members, N =/= node()],
  rabbit_quorum_queue:transfer_leadership(Q, hd(OtherNodes)),
  timer:sleep(2000),
  {ok, Q2} = rabbit_amqqueue:lookup(QName),
  rabbit_quorum_queue:transfer_leadership(Q2, node()).
'

# Wait for ra_server_sup (intensity 2/period 5) to give up
sleep 10

rabbitmqctl eval '
  %% 3. Verify the queue is stranded.
  {ok, RaName} = rabbit_queue_type_util:qname_to_internal_name(
    rabbit_misc:r(<<"vhost">>, queue, <<"my_queue">>)),
  whereis(RaName).  %% returns undefined = stranded
'

The ra_server crashes 3 times in ~1.1 seconds (each crash-loop iteration elects itself candidate, enters leader state, calls start_worker, hits {noproc, start_child}, crashes), exhausting ra_server_sup intensity 2/period 5. The member is permanently abandoned.

Log output:

State machine 'vhost_my_queue' terminating
Reason for termination = exit:{noproc,
                                   [rabbit_fifo_dlx_sup,
                                    {start_child,[{resource,<<"vhost">>,queue,<<"my_queue">>}]},
                                    infinity]}}
...
reason: reached_max_restart_intensity

Expected behavior

A transient inability to start DLX workers during membership churn should not permanently strand quorum-queue members.

Suggested fix: double-layer supervisor (_sup_sup pattern)

The root structural problem is that rabbit_fifo_dlx_sup is a single, globally-registered simple_one_for_one supervisor that hosts ALL DLX workers for ALL at_least_once queues on the node. A burst of worker exits that exceeds its intensity kills the supervisor, taking all remaining healthy workers with it and leaving no viable start_child target for any queue on the node.

The _sup_sup pattern already used by Ra (ra_server_sup_sup / ra_server_sup) and elsewhere in RabbitMQ (rabbit_channel_sup_sup, rabbit_amqqueue_sup_sup) eliminates this:

rabbit_fifo_dlx_sup_sup          (simple_one_for_one, children are supervisors, restart => temporary)
  +-- rabbit_fifo_dlx_worker_sup [queue A]  (one_for_one, intensity N/period M)
  |     +-- rabbit_fifo_dlx_worker [queue A]
  +-- rabbit_fifo_dlx_worker_sup [queue B]
        +-- rabbit_fifo_dlx_worker [queue B]

Why this works:

• Each DLX worker is isolated behind its own per-worker supervisor.
• A burst of worker crashes kills individual per-worker supervisors, not the top-level sup_sup.
• restart => temporary on the child spec means OTP's supervisor never restarts a dead per-worker supervisor and never increments the sup_sup intensity counter (OTP supervisor.erl do_restart/3 for temporary children deletes the child and returns without calling add_restart/1). The sup_sup is structurally immune to reached_max_restart_intensity regardless of the number or rate of child crashes.
• supervisor:start_child(rabbit_fifo_dlx_sup_sup, ...) always works because the sup_sup is always alive. When leadership settles and state_enter(leader) fires again, start_worker can create a fresh per-worker supervisor + worker. The queue recovers on the next leadership attempt rather than being permanently stranded.

Changes required:

1. rabbit_fifo_dlx_sup.erl becomes rabbit_fifo_dlx_sup_sup.erl: type => supervisor, restart => temporary, child start is {rabbit_fifo_dlx_worker_sup, start_link, []}.
2. New rabbit_fifo_dlx_worker_sup.erl: trivial one_for_one supervisor hosting one rabbit_fifo_dlx_worker.
3. rabbit_fifo_dlx:start_worker/1: target changes from rabbit_fifo_dlx_sup to rabbit_fifo_dlx_sup_sup.
4. rabbit_fifo_dlx:ensure_worker_terminated/1: terminate the per-worker supervisor (e.g. via the worker Pid's parent, similar to how ra_server_sup_sup:stop_server uses ra_directory:where_is_parent).

Additional context

See the attached rabbitmq-dlx-artifacts.md for full log evidence: the rabbit_fifo_dlx_worker exits with {error, ra_command_failed}, the rabbit_fifo_dlx_sup child_terminated and reached_max_restart_intensity reports, the ra_server "State machine ... terminating" dumps with exit:{noproc,{gen_server,call,[rabbit_fifo_dlx_sup,{start_child,...}]}}, the ra_server_sup give-up reports, and the clean reproduction log output.

rabbitmq-dlx-artifacts.md