cluster/observability: fix empty partition in decom / reconfiguration status ouputs

[bharathv]

Often we seen outputs like this

DECOMMISSION PROGRESS
=====================
PARTITION                        MOVING-TO  COMPLETION-%  PARTITION-SIZE
kafka/prod_observations/1        26         0             0
kafka/prod_observations/8        26         0             0
kafka/prod_observations/19       26         0             0
kafka/prod_observations/28       26         0             0

Sometimes the partition size info and completion percentage don’t show up even though the partition is actually moving just fine. This happens because the recovery state isn’t being polled from the leader; instead, it’s gathered from the shards local to whatever node you’re querying. If that node doesn’t have the replica or isn’t the leader, the info is missing. The fix is to route reconciliation state queries to the leader node instead.

Fixes: https://redpandadata.atlassian.net/browse/CORE-14975

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v25.3.x
• v25.2.x
• v25.1.x

Release Notes

Bug Fixes

• Fixes incorrect reporting of partition movement progress in decommission/reconfiguration status output.

[Copilot]

Pull request overview

This PR fixes an issue where decommissioning and reconfiguration status outputs show empty partition metadata (partition size and completion percentage) even when partitions are actively moving. The root cause was that reconciliation state was being queried from local shards instead of partition leaders, resulting in incomplete information when the queried node doesn't host the replica or isn't the leader.

Key changes:

• Routes reconciliation state queries to partition leaders instead of local shards
• Adds local_size to serialization fields in recovery_state
• Adds comprehensive test coverage for decommission status reporting

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/v/cluster/controller_api.h	Adds partition_leaders_table dependency and new method get_partition_leader_reconciliation_state
src/v/cluster/controller_api.cc	Implements leader-based reconciliation state queries and refactors partition reconfiguration state gathering to use concurrent queries
src/v/cluster/types.h	Updates recovery_state serde version and includes local_size in serialization
src/v/cluster/types.cc	Updates recovery_state output operator to include local_size
src/v/redpanda/admin/partition.cc	Changes admin handler to use new leader-based reconfiguration state API
src/v/cluster/controller.cc	Passes partition_leaders reference to controller_api constructor
tests/rptest/tests/nodes_decommissioning_test.py	Adds test validating decommission status is properly reported across all nodes

[Copilot]

The magic number 16 for max concurrency should be extracted as a named constant or configuration parameter to make its purpose clear and allow easier tuning.

[Copilot]

The magic number 2 << 30 (2GB) should be extracted as a named constant to clarify that this is setting a high recovery rate to allow decommissioning to complete.

[vbotbuildovich]

CI test results

test results on build#78085

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
DatalakeDLQTest	test_dlq_table_for_mixed_records	{"catalog_type": "rest_jdbc", "cloud_storage_type": 1, "query_engine": "spark"}	integration	https://buildkite.com/redpanda/redpanda/builds/78085#019b2fc6-cb9d-4790-ad6e-282d192fb02e	FLAKY	50/51	Test PASSES after retries.Inconclusive result after max retries(baseline=0.0021, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.0064, p1=0.7241, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DatalakeDLQTest&test_method=test_dlq_table_for_mixed_records
MountUnmountIcebergTest	test_simple_remount	{"cloud_storage_type": 1}	integration	https://buildkite.com/redpanda/redpanda/builds/78085#019b2fcf-95dd-42b4-aa1e-3fd2d65f7fcb	FLAKY	7/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.1707, p0=0.2362, reject_threshold=0.0100. adj_baseline=0.4297, p1=0.3108, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=MountUnmountIcebergTest&test_method=test_simple_remount

[vbotbuildovich]

/backport v25.3.x

[vbotbuildovich]

/backport v25.2.x

[vbotbuildovich]

Failed to create a backport PR to v25.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-29047-v25.2.x-549 remotes/upstream/v25.2.x
git cherry-pick -x 29c6050df1 124201459b f8c76ea882 cab8675039 bfd976166c da299be686 aa11692ada

Workflow run logs.