Enabling tracing on multiple vhosts concurrently silently drops vhosts from trace_vhosts

[stefan-rust]

Describe the bug

Enabling firehose tracing on a vhost is a non-atomic read-modify-write of the single
shared rabbit app-env list trace_vhosts (rabbit_trace:update_config/1):

update_config(Fun) ->
    VHosts0 = vhosts_with_tracing_enabled(),           %% READ
    VHosts  = Fun(VHosts0),                            %% MODIFY
    application:set_env(rabbit, ?TRACE_VHOSTS, VHosts), %% WRITE (replaces whole list)
    ...

When tracing is enabled on multiple vhosts concurrently, two callers READ the same stale
list and the second WRITE overwrites the first. Vhosts are silently dropped from
trace_vhosts even though every enable returns ok; their tracer never activates and
never recovers. All enable paths funnel through here (rabbitmqctl trace_on,
PUT /api/vhosts {"tracing": true}, the management UI), so all are affected.

Source: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_trace.erl

Versions

• RabbitMQ: 4.3.2 (code path unchanged on main)
• Erlang/OTP: 27
• OS / image: official rabbitmq:4.3.2-management Docker image
• Client: none — reproduced with rabbitmqctl only

Suggested fix
Serialize all trace_vhosts toggles so the read-modify-write is atomic (e.g. route
start/1/stop/1 through a single gen_server, or guard update_config/1 with a node
lock). Happy to submit a PR if you confirm the preferred approach.

Reproduction steps

On a default single-node RabbitMQ broker (e.g. the official rabbitmq:4.3.2 image,
no extra config), create N vhosts, then enable tracing on all of them simultaneously
and count how many actually land in trace_vhosts. This rabbitmqctl eval spawns one
Erlang process per vhost, releases them from a barrier at once, and returns the count of
survivors:

# create 50 vhosts
for i in $(seq 1 50); do rabbitmqctl add_vhost "v$i"; done

# enable tracing on all 50 concurrently from inside the broker, then count survivors
rabbitmqctl eval '
  VHs=[list_to_binary("v"++integer_to_list(I))||I<-lists:seq(1,50)],
  P=self(),
  Ps=[spawn(fun()->receive go->ok end,catch rabbit_trace:start(V),P!{done,self()} end)||V<-VHs],
  [X!go||X<-Ps],
  [receive{done,_}->ok end||_<-Ps],
  {ok,En}=application:get_env(rabbit,trace_vhosts),
  length([V||V<-En,lists:member(V,VHs)]).'

Expected: 50 — every successfully-enabled vhost is present in trace_vhosts.

Actual: far fewer. On RabbitMQ 4.3.2 / OTP 27 this printed 13 on the first attempt —
37 of 50 enables were lost.

Expected behavior

Enabling tracing on a vhost is durable and independent of concurrency: if N enable
operations each return ok, all N vhosts should be present in trace_vhosts and have an
active tracer. Concurrent enables on different vhosts must not lose each other's updates.

Additional context

Root cause is a classic lost update: rabbit_trace:update_config/1 does
get_env -> modify -> set_env on the shared rabbit/trace_vhosts list with no
serialization, so concurrent callers overwrite each other.

• Affects all enable entry points, since they all call rabbit_trace:start/1 ->
  update_config/1: rabbitmqctl trace_on, PUT /api/vhosts {"tracing": true} (HTTP API),
  and the management UI checkbox.
• The dropped vhost stays broken until tracing is explicitly toggled again for it — the
  per-channel cache (rabbit_trace:init/1) is only refreshed when tracing toggles.
• Originally hit in an automated test harness that provisions a fresh vhost with tracing per
  test; reduced to the minimal rabbitmqctl eval reproduction above.
• Suggested fix: serialize toggles through a single gen_server (or a node-global lock) so
  the read-modify-write is atomic.

[michaelklishin]

@stefan-rust rabbitmq_tracing was never intended to handle tracing in 50 virtual hosts concurrently. It's a debugging feature that's supposed to be enabled during short periods of time, not a permanent setting.

Instead of introducing a gen_server for serialized writes, a global lock would do for this node-local plugin.

[stefan-rust]

Hi @michaelklishin
Thank you for the quick reply. I had already suspected that the tracing function wasn't designed for this purpose. However, I wanted to report the issue because we use the tracing function for verification in our integration tests (using a dedicated RabbitMQ verifier client). Since we use direct exchanges for the RPC calls and the RabbitMQ reply-to queue for the responses, we couldn't find any other way to verify requests/responses. As suggested, we are currently using a global lock in our client when creating the vHost with tracing enabled.