## Log artifacts (obfuscated)

Obfuscation applied to all blocks below:
- Hostnames: `rabbit@node-a`, `rabbit@node-b`, `rabbit@node-c` (the replaced node). Region/domain suffix stripped.
- Queue names replaced with `qq-1` .. `qq-4` (all were `at_least_once` dead-lettering quorum queues; several were federation queues).
- Virtual host shown as `vh`.
- Erlang pids are left unchanged (not sensitive).

RabbitMQ 4.2.4, Erlang 27.3.4.2, 3-node cluster. All blocks are from the two surviving nodes during a rolling node replacement.

### 1. rabbit_fifo_dlx_worker exits with `{error, ra_command_failed}`, then rabbit_fifo_dlx_sup logs `child_terminated`

The worker fails to register its checkout command against the (flapping) leader and exits; this repeats across many DLX queues at once.

```
[error] <0.138291892.4>     initial call: rabbit_fifo_dlx_worker:init/1
[error] <0.138291892.4>     pid: <0.138291892.4>
[error] <0.138291892.4>     registered_name: []
[error] <0.138291892.4>     exception exit: {error,ra_command_failed}
[error] <0.138291892.4>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 2476)
[error] <0.138291892.4>     ancestors: [rabbit_fifo_dlx_sup,rabbit_sup,<0.208.0>]
[error] <0.138291892.4>     trap_exit: false
[error] <0.138291892.4>     status: running

[error] <0.5329.0>     supervisor: {local,rabbit_fifo_dlx_sup}
[error] <0.5329.0>     errorContext: child_terminated
[error] <0.5329.0>     reason: {error,ra_command_failed}
[error] <0.5329.0>     offender: [{pid,<0.138291892.4>},
[error] <0.5329.0>                {id,rabbit_fifo_dlx_worker},
[error] <0.5329.0>                {mfargs,
[error] <0.5329.0>                    {rabbit_fifo_dlx_worker,start_link,
[error] <0.5329.0>                        [{resource,<<"vh">>,queue,
[error] <0.5329.0>                             <<"qq-1">>}]}},
[error] <0.5329.0>                {restart_type,transient},
[error] <0.5329.0>                {significant,false},
[error] <0.5329.0>                {shutdown,5000},
[error] <0.5329.0>                {child_type,worker}]
```

Immediately after, the worker that is still retrying finds the supervisor gone (`{error, noproc}`):

```
[warning] <0.138291192.4> Failed to process command {dlx,{checkout,<0.138291192.4>,32}} on quorum queue leader {'vh_qq-1','rabbit@node-a'}: {error,noproc}
[warning] <0.138291192.4> Trying 5 more time(s)...
```

### 2. rabbit_fifo_dlx_sup reaches max restart intensity and gives up (one node)

On one surviving node the DLX worker supervisor itself terminated (`errorContext: shutdown`, `reason: reached_max_restart_intensity`). On the other surviving node it did not log a give-up but stopped servicing calls (subsequent `start_child` returned `noproc`, as in block 1).

```
[error] <0.7519.0>     supervisor: {local,rabbit_fifo_dlx_sup}
[error] <0.7519.0>     errorContext: shutdown
[error] <0.7519.0>     reason: reached_max_restart_intensity
[error] <0.7519.0>     offender: [{pid,<0.706203.0>},
[error] <0.7519.0>                {id,rabbit_fifo_dlx_worker},
[error] <0.7519.0>                {mfargs,
[error] <0.7519.0>                    {rabbit_fifo_dlx_worker,start_link,
[error] <0.7519.0>                        [{resource,<<"vh">>,queue,
[error] <0.7519.0>                             <<"qq-3">>}]}},
[error] <0.7519.0>                {restart_type,transient},
[error] <0.7519.0>                {significant,false},
[error] <0.7519.0>                {shutdown,5000},
[error] <0.7519.0>                {child_type,worker}]
```

### 3. The ra_server crashes on entering leader state because start_child returns `noproc`

The queue wins an election (`candidate`, `request_vote_result ... true`), enters leader state, calls `supervisor:start_child(rabbit_fifo_dlx_sup, ...)`, and crashes because the supervisor is unavailable. The large `When server state` dump is elided.

```
[error] <0.138643711.4> ** State machine 'qq-2' terminating
[error] <0.138643711.4> ** Last event = {cast,{request_vote_result,2,true}}
[error] <0.138643711.4> ** When server state  = {candidate,
   ... [full ra_server_state dump elided] ...
[error] <0.138643711.4> ** Reason for termination = exit:{noproc,
[error] <0.138643711.4>                                   {gen_server,call,
[error] <0.138643711.4>                                    [rabbit_fifo_dlx_sup,
[error] <0.138643711.4>                                     {start_child,
[error] <0.138643711.4>                                      [{resource,<<"vh">>,queue,
[error] <0.138643711.4>                                        <<"qq-2">>}]},
[error] <0.138643711.4>                                     infinity]}}
[error] <0.138643711.4> ** Callback modules = [ra_server_proc]
[error] <0.138643711.4> ** Callback mode = [state_functions,state_enter]
[error] <0.138643711.4> ** Stacktrace =
[error] <0.138643711.4> **  [{gen_server,call,3,[{file,"gen_server.erl"},{line,1222}]},
```

### 4. ra_server_sup gives up on the queue (the permanent stranding)

After the `ra_server` crash-loops, its per-queue `ra_server_sup` (`intensity => 2, period => 5`) reaches max restart intensity and abandons the member, which then stays `noproc` with no auto-restart. This report (not the `rabbit_fifo_dlx_sup` one) is what occurred 500+ times per surviving node.

```
[error] <0.138279253.4>     supervisor: {<0.138279253.4>,ra_server_sup}
[error] <0.138279253.4>     errorContext: shutdown
[error] <0.138279253.4>     reason: reached_max_restart_intensity
[error] <0.138279253.4>     offender: [{pid,<0.138291291.4>},
[error] <0.138279253.4>                {id,'qq-1'},
[error] <0.138279253.4>                {mfargs,
[error] <0.138279253.4>                 {ra_server_proc,start_link,
[error] <0.138279253.4>                  [#{id =>
[error] <0.138279253.4>                      {'qq-1','rabbit@node-a'},
   ... [config map elided] ...
```


### 5. Reproduction log (clean, no obfuscation needed)

Reproduced on a 3-node RabbitMQ 4.2 cluster (Erlang 27.3.4.2) by unregistering `rabbit_fifo_dlx_sup` and transferring leadership to the affected node. The queue had `dead-letter-strategy: at-least-once` and `overflow: reject-publish` set via policy.

```
2026-06-10 16:45:16.025570+00:00 [debug] <0.3309.0> queue 'dev_inbound.queue-1' in vhost 'dev': terminating with {noproc,{gen_server,call,[rabbit_fifo_dlx_sup,{start_child,[{resource,<<100,101,118>>,queue,<<100,101,118,95,105,110,98,111,117,110,100,46,113,117,101,117,101,45,49>>}]},infinity]}} in state candidate
2026-06-10 16:45:16.025856+00:00 [error] <0.3309.0> ** State machine 'dev_dev_inbound.queue-1' terminating
2026-06-10 16:45:16.025856+00:00 [error] <0.3309.0> ** Reason for termination = exit:{noproc,
2026-06-10 16:45:16.025856+00:00 [error] <0.3309.0>                                    [rabbit_fifo_dlx_sup,
2026-06-10 16:45:16.029335+00:00 [error] <0.3309.0>     exception exit: {noproc,
2026-06-10 16:45:16.029335+00:00 [error] <0.3309.0>                             [rabbit_fifo_dlx_sup,
2026-06-10 16:45:16.034232+00:00 [error] <0.3308.0>     reason: {noproc,
2026-06-10 16:45:16.034232+00:00 [error] <0.3308.0>                     [rabbit_fifo_dlx_sup,

2026-06-10 16:45:16.599145+00:00 [debug] <0.10739.0> queue 'dev_inbound.queue-1' in vhost 'dev': terminating with {noproc,{gen_server,call,[rabbit_fifo_dlx_sup,{start_child,[{resource,<<100,101,118>>,queue,<<100,101,118,95,105,110,98,111,117,110,100,46,113,117,101,117,101,45,49>>}]},infinity]}} in state candidate
2026-06-10 16:45:16.599452+00:00 [error] <0.10739.0> ** State machine 'dev_dev_inbound.queue-1' terminating
2026-06-10 16:45:16.599452+00:00 [error] <0.10739.0> ** Reason for termination = exit:{noproc,
2026-06-10 16:45:16.599452+00:00 [error] <0.10739.0>                                    [rabbit_fifo_dlx_sup,
2026-06-10 16:45:16.602722+00:00 [error] <0.10739.0>     exception exit: {noproc,
2026-06-10 16:45:16.602722+00:00 [error] <0.10739.0>                             [rabbit_fifo_dlx_sup,
2026-06-10 16:45:16.603609+00:00 [error] <0.3308.0>     reason: {noproc,
2026-06-10 16:45:16.603609+00:00 [error] <0.3308.0>                     [rabbit_fifo_dlx_sup,

2026-06-10 16:45:17.174696+00:00 [debug] <0.10827.0> queue 'dev_inbound.queue-1' in vhost 'dev': terminating with {noproc,{gen_server,call,[rabbit_fifo_dlx_sup,{start_child,[{resource,<<100,101,118>>,queue,<<100,101,118,95,105,110,98,111,117,110,100,46,113,117,101,117,101,45,49>>}]},infinity]}} in state candidate
2026-06-10 16:45:17.175000+00:00 [error] <0.10827.0> ** State machine 'dev_dev_inbound.queue-1' terminating
2026-06-10 16:45:17.175000+00:00 [error] <0.10827.0> ** Reason for termination = exit:{noproc,
2026-06-10 16:45:17.175000+00:00 [error] <0.10827.0>                                    [rabbit_fifo_dlx_sup,
2026-06-10 16:45:17.177650+00:00 [error] <0.10827.0>     exception exit: {noproc,
2026-06-10 16:45:17.177650+00:00 [error] <0.10827.0>                             [rabbit_fifo_dlx_sup,
2026-06-10 16:45:17.178704+00:00 [error] <0.3308.0>     reason: {noproc,
2026-06-10 16:45:17.178704+00:00 [error] <0.3308.0>                     [rabbit_fifo_dlx_sup,
2026-06-10 16:45:17.180056+00:00 [error] <0.3308.0>     reason: reached_max_restart_intensity
```

Three Ra server crashes (`<0.3309.0>` at 16:45:16.025, `<0.10739.0>` at 16:45:16.599, `<0.10827.0>` at 16:45:17.174) in 1.1 seconds, each in state `candidate` (won election, entering leader state, crashed on `start_worker`). The `ra_server_sup` (`<0.3308.0>`) gives up at 16:45:17.180 with `reached_max_restart_intensity`. Queue member permanently `noproc` thereafter.

Post-stranding state:
```
Members: ['rabbit@ip-10-0-1-74','rabbit@ip-10-0-1-213','rabbit@ip-10-0-1-61']
Online:  ['rabbit@ip-10-0-1-213','rabbit@ip-10-0-1-61']
```