Quorum queue stats show ??? in management UI after stepwise upgrade from 3.13 to 4.3 and a 4.3 restart afterwards

[SABEELKAZI]

Describe the bug

After upgrading from RabbitMQ 3.13.x to 4.3 one version at a time (3.13 → 4.0 → 4.1 → 4.2 → 4.3), and then on a subsequent restart of the 4.3 node, quorum queues in the management UI show ??? for message count fields (messages, messages_ready, messages_unacknowledged), and the HTTP API omits those fields from the JSON response entirely. Classic queues are unaffected.

The management UI looks like this:

Queue name     Type    State    messages   messages_ready
test-quorum    quorum  running  ???        ???            <- broken
test-classic   classic running  0          0             <- fine

With debug logging enabled (rabbitmqctl set_log_level debug), the following appears every 5 seconds for every quorum queue whose data originated in 3.13.x, starting from the very first tick after startup:

2026-07-19 13:28:16.473728+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command {handle_tick,
2026-07-19 13:28:16.474363+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command tick
2026-07-19 13:28:21.475291+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command {handle_tick,
2026-07-19 13:28:21.476336+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command tick
2026-07-19 13:28:26.477354+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command {handle_tick,
2026-07-19 13:28:26.478043+00:00 [debug] <0.813.0> queue 'test-quorum' in vhost '/': rabbit_fifo: unhandled aux command tick

These messages come from the catch-all clause in rabbit_fifo:handle_aux/5. Because the intended handler clause never matches, stats are never written.

Reproduction steps

Step 1: Start RabbitMQ 3.13 and create a quorum queue

docker run -d --name rmq -h rmq-host -v /tmp/rmq-data:/var/lib/rabbitmq -e RABBITMQ_DEFAULT_USER=admin -e [`RABBITMQ_DEFAULT_PASS=admin`](url) -p 15672:15672 rabbitmq:3.13-management

Wait about 30 seconds for startup, then create a quorum queue:

curl -u admin:admin -X PUT http://localhost:15672/api/queues/%2F/test-quorum \
    -H "Content-Type: application/json" \
    -d '{"durable":true,"arguments":{"x-queue-type":"quorum"}}'

Note: Or create the queue through the management UI at http://localhost:15672.

docker stop rmq && docker rm rmq

Step 2: Upgrade stepwise to 4.3

For each version below, start the container with the same volume and hostname (-v /tmp/rmq-data:/var/lib/rabbitmq and -h rmq-host), wait for it to fully start, then stop and remove it before moving to the next version.

rabbitmq:4.0-management   start -> wait -> stop
rabbitmq:4.1-management   start -> wait -> stop
rabbitmq:4.2-management   start -> wait -> stop
rabbitmq:4.3-management   start -> wait -> check stats (shows 0, works fine) -> stop

At the end of 4.3 first boot, stats report correctly (messages = 0). The bug is not yet visible.

Step 3: Restart 4.3 (this is when the bug appears)

docker run -d --name rmq -h rmq-host -v /tmp/rmq-data:/var/lib/rabbitmq -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -p 15672:15672 rabbitmq:4.3-management

Wait about 20 seconds, then check the queue:

curl -s -u admin:admin http://localhost:15672/api/queues/%2F/test-quorum | grep -o '"messages":[0-9]*'

Expected: "messages": 0
Actual: the messages key is absent from the response entirely

You can also open http://localhost:15672 (admin/admin) and see ??? in the message count columns for test-quorum. Publishing and consuming messages still works normally.

Automation

The steps above are scripted in reproduce.ps1 (Windows/PowerShell), attached here. The script leaves the final 4.3 container running after restart so you can inspect the management UI at http://localhost:15672 (admin/admin) and see the ??? message counts directly.

# reproduce.ps1
# Reproduces the RabbitMQ bug: quorum queue management stats missing after
# upgrading from 3.13.x to 4.x.
#
# Requires: Docker Desktop
#
# What this script does:
#   1. Starts each RabbitMQ version in sequence (3.13 -> 4.0 -> 4.1 -> 4.2 -> 4.3)
#      using a shared persistent volume and fixed hostname.
#   2. On 3.13, creates one quorum queue.
#   3. Starts 4.3 a SECOND time (restart). This is when the bug appears:
#      the first 4.3 boot writes a machine-version-8 noop to the Raft log;
#      the second boot replays it during recovery, which sets the aux handler
#      to the new API before any tick can migrate the stale aux state.
#   4. Leaves the container running so you can inspect the management UI.
#
# Management UI: http://localhost:15672  (admin / admin)
# Clean up:      docker rm -f rmq-bug-repro

# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------
$Port      = "15672"
$User      = "admin"
$Pass      = "admin"
$Container = "rmq-bug-repro"
$Hostname  = "rmq-bug-host"
$DataDir   = "$PSScriptRoot\data"
$BaseUrl   = "http://localhost:$Port"
$Headers   = @{
    Authorization  = "Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("${User}:${Pass}"))
    "Content-Type" = "application/json"
}

$UpgradePath = @(
    "rabbitmq:3.13-management"
    "rabbitmq:4.0-management"
    "rabbitmq:4.1-management"
    "rabbitmq:4.2-management"
    "rabbitmq:4.3-management"
)

# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
function Start-Container([string]$Image) {
    docker run -d --name $Container `
        -h $Hostname `
        -v "${DataDir}:/var/lib/rabbitmq" `
        -e RABBITMQ_DEFAULT_USER=$User `
        -e RABBITMQ_DEFAULT_PASS=$Pass `
        -p "${Port}:15672" `
        $Image | Out-Null
}

function Wait-Until-Ready {
    Write-Host -NoNewline "  Waiting"
    for ($i = 0; $i -lt 60; $i++) {
        try {
            Invoke-WebRequest "$BaseUrl/api/overview" -Headers $Headers `
                -TimeoutSec 3 -UseBasicParsing -ErrorAction Stop | Out-Null
            Write-Host " ready."; return
        } catch {}
        Write-Host -NoNewline "."
        Start-Sleep -Seconds 3
    }
    Write-Host " TIMEOUT"; exit 1
}

function Stop-Container {
    docker stop $Container | Out-Null
    docker rm   $Container | Out-Null
}

function Get-MessageCount {
    try {
        $q = Invoke-RestMethod "$BaseUrl/api/queues/%2F/test-quorum" -Headers $Headers
        if ($q.PSObject.Properties.Name -contains "messages") { return [string]$q.messages }
    } catch {}
    return $null   # null means stats are missing (the bug)
}

# ---------------------------------------------------------------------------
# Setup: wipe any previous data and container
# ---------------------------------------------------------------------------
docker rm -f $Container 2>$null | Out-Null
if (Test-Path $DataDir) { Remove-Item -Recurse -Force $DataDir }
New-Item -ItemType Directory $DataDir | Out-Null

# ---------------------------------------------------------------------------
# Step 1: Stepwise upgrade (3.13 -> 4.0 -> 4.1 -> 4.2 -> 4.3)
# The queue is created on 3.13; each subsequent version just starts and stops.
# ---------------------------------------------------------------------------
Write-Host ""
Write-Host "Upgrade path: $($UpgradePath[0]) -> $($UpgradePath[-1])" -ForegroundColor Cyan

foreach ($image in $UpgradePath) {
    $version = ($image -replace "rabbitmq:", "") -replace "-management", ""
    Write-Host ""
    Write-Host "[$version]  $image"

    Start-Container $image
    Wait-Until-Ready
    Start-Sleep -Seconds 8

    if ($image -eq $UpgradePath[0]) {
        Write-Host "  Creating quorum queue 'test-quorum'..."
        $body = ConvertTo-Json @{ durable = $true; arguments = @{ "x-queue-type" = "quorum" } }
        Invoke-RestMethod "$BaseUrl/api/queues/%2F/test-quorum" `
            -Method Put -Headers $Headers -Body $body | Out-Null
        Start-Sleep -Seconds 5
    }

    $msgs = Get-MessageCount
    Write-Host "  messages = $(if ($null -eq $msgs) { '(null)' } else { $msgs })"

    $finalMessages = $msgs   # track the last step's result for the summary below

    Stop-Container
}

# ---------------------------------------------------------------------------
# Step 2: Restart 4.3 -- this is where the bug manifests
# The first 4.3 boot (above) runs cleanly. The second boot replays the
# machine-version-8 noop written by the first boot, which sets the aux
# handler to the new API before any tick can migrate the stale aux state.
# Comment this block out to confirm the bug does NOT appear on first boot.
# ---------------------------------------------------------------------------
Write-Host ""
Write-Host "[4.3 RESTART]  rabbitmq:4.3-management" -ForegroundColor Yellow

Start-Container "rabbitmq:4.3-management"
Wait-Until-Ready
Start-Sleep -Seconds 8

$finalMessages = Get-MessageCount
Write-Host "  messages = $(if ($null -eq $finalMessages) { '(null)  <-- BUG' } else { $finalMessages })"

# ---------------------------------------------------------------------------
# Result
# ---------------------------------------------------------------------------
Write-Host ""
if ($null -eq $finalMessages) {
    Write-Host "BUG CONFIRMED" -ForegroundColor Red
    Write-Host "The HTTP API returns null for quorum queue message counts."
    Write-Host "The management UI shows ??? instead of 0."
    Write-Host "The queue itself is functional (rabbitmqctl shows correct counts)."
} else {
    Write-Host "No bug observed (messages = $finalMessages)" -ForegroundColor Green
}

Write-Host ""
Write-Host "Container left running for UI inspection:" -ForegroundColor Cyan
Write-Host "  http://localhost:$Port   (user: $User / pass: $Pass)"
Write-Host ""
Write-Host "Clean up when done:  docker rm -f $Container"

Expected behavior

After upgrade and node restart, the management UI and HTTP API should report correct message counts for quorum queues - the same values that rabbitmqctl list_queues already shows. The messages field should be present in the API response and display a numeric value (e.g. 0) rather than ???.

Classic queues are unaffected; only quorum queues whose data originated in 3.13.x exhibit this behaviour.

Additional context

Root Cause (from AI tool)

The root cause analysis below was identified with the assistance of an AI tool.

rabbit_fifo.erl contains migration clauses to upgrade stale aux_state formats on first use:

handle_aux(RaftState, Tag, Cmd, AuxV2, RaAux)
  when element(1, AuxV2) == aux_v2 -> ...   %% present

handle_aux(RaftState, Tag, Cmd, AuxV3, RaAux)
  when element(1, AuxV3) == aux_v3 -> ...   %% present

%% MISSING: no clause for element(1, Aux) == aux  (original pre-versioned format)

Queues created in 3.13.x have no initial_machine_version in their stored Ra config — the field did not exist then. On startup this defaults to 0, which maps to rabbit_fifo_v0, so aux_state is initialised as #aux{}.

On the first 4.3 boot the old {handle_aux, 6} API is briefly active during log replay, long enough for a tick to migrate #aux{} before the version-8 noop switches to {handle_aux, 5} — so the first boot works.

On any subsequent boot the version-8 noop is already in the Raft log. It is replayed before the first tick fires, setting the handler to {handle_aux, 5} while aux_state is still #aux{}. No migration clause matches, the catch-all fires on every tick, and rabbit_core_metrics:queue_stats is never called.

---

Suggested Fix (from AI tool)

The fix was identified by AI and then verified by patching the compiled beam files into an existing RabbitMQ 4.3.2 Docker image. After applying the patch and running the same upgrade sequence, the management UI shows correct count instead of ??? on both the first and subsequent boots.

Fix 1: deps/rabbit/src/rabbit_fifo.erl (required)

Add the missing migration clause before the existing aux_v2 clause (around line 1302):

+handle_aux(RaftState, Tag, Cmd, AuxV0, RaAux)
+  when is_tuple(AuxV0) andalso element(1, AuxV0) =:= aux ->
+    Name = element(2, AuxV0),
+    AuxV4 = init_aux(Name),
+    handle_aux(RaftState, Tag, Cmd, AuxV4, RaAux);
 handle_aux(RaftState, Tag, Cmd, AuxV2, RaAux)   %% existing clause unchanged
   when element(1, AuxV2) == aux_v2 ->
     ...

This closes the gap in the migration chain. With the fix the chain becomes: aux -> aux_v2 -> aux_v3 -> aux_v4

Fix 2: deps/rabbit/src/rabbit_quorum_queue.erl (secondary improvement)

Include initial_machine_version in make_mutable_config/1 so the stored Ra config is updated on every server restart. This ensures init_aux is called with the current module from the start and removes the root condition entirely:

 make_mutable_config(Q) ->
     ...
     #{tick_timeout                     => ...,
       min_recovery_checkpoint_interval  => ...,
       ra_event_formatter                => ...,
+      initial_machine_version           => rabbit_fifo:version()}.

Fix 1 alone is sufficient to resolve the issue. Fix 2 prevents the underlying condition from recurring in any future version migrations.

[kjnilsson]

thank you for this and for the analysis. I think actually this is an issue in Ra when recovering from a recovery snapshot. I will take a look at this and report back.

[kjnilsson]

rabbitmq/ra#645 contains a potential fix

[SABEELKAZI]

I tried to patch the 4.3.2 image with the latest changes and now the stats show correctly: messages = 0, incrementing to 1 on publish. Thanks for the quick turnaround!

[kjnilsson]

I tried to patch the 4.3.2 image with the latest changes and now the stats show correctly: messages = 0, incrementing to 1 on publish. Thanks for the quick turnaround!

you tried with the Ra PR changes?

[SABEELKAZI]

Yes. I pulled the fix from the aux-init-bug branch, built it into our 4.3.2 image, and ran the full 3.13 → 4.3 upgrade + restart. Stats reported correctly afterwards :)

[kjnilsson]

Yes. I pulled the fix from the aux-init-bug branch, built it into our 4.3.2 image, and ran the full 3.13 → 4.3 upgrade + restart. Stats reported correctly afterwards :)

Excellent. I will only test 3.13.x -> 4.2 -> 4.3 then as that is another supported upgrade path which may hide some issues.

[kjnilsson]

@SABEELKAZI on reflection the fix in Ra isn't enough - I will also take (a version of) your fix no 1.

[the-mikedavis]

The #aux{} handling here didn't make sense to me at first but now I see it. When recovering in ra_server:init/1, if a snapshot does not exist then the aux state is initialized using machine version 0, which creates an #aux{}. Otherwise an #aux{} could only be initialized if your effective machine version was 0 or 1 (< RabbitMQ v3.10.x) at time of snapshot. And setting initial_machine_version (done since RabbitMQ v4.1.x) prevents this from affecting queues created on v4.1+. So this should only affect QQs created on v4.0.x or lower which don't see enough traffic to snapshot, and then are upgraded to v4.3.x..

But I believe the proposed secondary improvement (initial_machine_version in mutable config) would not be correct:

• When recovering without a snapshot, the initial_machine_version would guide the server to replay through potentially a different machine version than it played through before recovery.
• In a mixed version cluster during a rolling upgrade, for a queue with no snapshot, if a member is on a different version than the others, it can use that version without coordinating it with the other cluster members. If the machine versions have any behavior differences then the replicas diverge.

[kjnilsson]

in #16990 we just handle the state upgrade and leave initial_machine_version (which is only meant to be used at declaration time)

[SABEELKAZI]

Perfect, Looking forward to #16990!
Thank you once again.😀