Improve rabbit_fifo_dlx_worker resiliency

[ansd]

1. Terminate replaced rabbit_fifo_dlx_worker

Jepsen dead lettering tests of job qq-jepsen-test-3-12 of Concourse pipeline jepsen-tests fail sometimes with following error:

{{:try_clause, [{:undefined, #PID<12128.3596.0>, :worker, [:rabbit_fifo_dlx_worker]}, {:undefined, #PID<12128.10212.0>, :worker, [:rabbit_fifo_dlx_worker]}]}, [{:erl_eval, :try_clauses, 10, [file: 'erl_eval.erl', line: 995]}, {:erl_eval, :exprs, 2, []}]}

At the end of the Jepsen test, there are 2 DLX workers on the same node.

Analysing the logs reveals the following:

Source quorum queue node becomes leader and starts its DLX worker:

2023-03-18 12:14:04.365295+00:00 [debug] <0.1645.0> started rabbit_fifo_dlx_worker <0.3596.0> for queue 'jepsen.queue' in vhost '/'

Less than 1 second later, Mnesia reports a network partition (introduced by Jepsen).
The DLX worker does not succeed to register as consumer to its source quorum queue because the Ra command times out:

2023-03-18 12:15:04.365840+00:00 [warning] <0.3596.0> Failed to process command {dlx,{checkout,<0.3596.0>,32}} on quorum queue leader {'%2F_jepsen.queue',
2023-03-18 12:15:04.365840+00:00 [warning] <0.3596.0>                                                                                  'rabbit@concourse-qq-jepsen-312-3'}: {timeout,
2023-03-18 12:15:04.365840+00:00 [warning] <0.3596.0>                                                                                                                        {'%2F_jepsen.queue',
2023-03-18 12:15:04.365840+00:00 [warning] <0.3596.0>                                                                                                                         'rabbit@concourse-qq-jepsen-312-3'}}
2023-03-18 12:15:04.365840+00:00 [warning] <0.3596.0> Trying 5 more time(s)...

3 seconds after the DLX worker got created, the local source quorum queue node is not leader anymore:

2023-03-18 12:14:07.289213+00:00 [notice] <0.1645.0> queue 'jepsen.queue' in vhost '/': leader -> follower in term: 17 machine version: 3

But because the DLX worker at this point failed to register as consumer, it will not be terminated in

rabbitmq-server/deps/rabbit/src/rabbit_fifo_dlx.erl

Lines 264 to 275 in 865d533

	ensure_worker_terminated(#?MODULE{consumer = undefined}) ->
	ok;
	ensure_worker_terminated(#?MODULE{consumer = #dlx_consumer{pid = Pid}}) ->
	case is_local_and_alive(Pid) of
	true ->
	%% Note that we can't return a mod_call effect here
	%% because mod_call is executed on the leader only.
	ok = supervisor:terminate_child(rabbit_fifo_dlx_sup, Pid),
	rabbit_log:debug("terminated rabbit_fifo_dlx_worker ~tp", [Pid]);
	false ->
	ok
	end.

Eventually, when the local node becomes a leader again, that DLX worker succeeds to register as consumer (due to retries in

rabbitmq-server/deps/rabbit/src/rabbit_fifo_dlx_client.erl

Lines 41 to 58 in 865d533

	process_command(Cmd, State, 5).
	process_command(_Cmd, _State, 0) ->
	{error, ra_command_failed};
	process_command(Cmd, #state{leader = Leader} = State, Tries) ->
	case ra:process_command(Leader, Cmd, 60_000) of
	{ok, ok, Leader} ->
	{ok, State#state{leader = Leader}};
	{ok, ok, L} ->
	rabbit_log:warning("Failed to process command ~tp on quorum queue leader ~tp because actual leader is ~tp.",
	[Cmd, Leader, L]),
	{error, ra_command_failed};
	Err ->
	rabbit_log:warning("Failed to process command ~tp on quorum queue leader ~tp: ~tp~n"
	"Trying ~b more time(s)...",
	[Cmd, Leader, Err, Tries]),
	process_command(Cmd, State, Tries - 1)
	end.

), and stays alive. When that happens, there is a 2nd DLX worker active because the 2nd got started when the local quorum queue node transitioned to become a leader.

This commit prevents this issue.
So, last consumer who does a #checkout{} wins and the “old one” has to terminate.

2. Make rabbit_fifo_dlx_sup more resilient

by not terminating itself if a low number of its children terminate

3. Do not restart DLX worker if leader is non-local

Instead, terminate, and let the new rabbit_fifo_dlx_worker on the other node take over.

[ansd]

The failing Test Mixed Version Clusters / Test (Mixed Version Cluster) (25_3) (pull_request) seems to be unrelated to this PR and caused by #7672

[kjnilsson]

LGTM