[v26.1.x] [CORE-15812] ct/scale: raise fetch concurrency for cloud_topics MPT variants#30867
Merged
oleiman merged 2 commits intoJun 23, 2026
Conversation
The cloud_topics ManyPartitionsTest variants (regular and tiered_storage) time out at the default fetch_max_read_concurrency=1. Set it to 4 for the cloud_topics path. A cloud-topic partition read is latency-bound: per L1 object it does a metastore lookup, a footer read, and -- on a cache miss -- an object-store GET to populate the local cache. That GET dominates (~70-90% of read wall-clock) and is a round-trip cost, not bandwidth: aggregate throughput with concurrency is ~170 MB/s, far below object-store bandwidth. fetch_max_read_concurrency caps how many partition reads in a fetch run concurrently. At the default of 1 a broker reads its hundreds of partitions serially, so the per-read round-trips serialize; across tens of thousands of small objects that is tens of minutes of pure latency and the consumer never keeps up. The regular (local-log) topic in the same test drains fine because a local read is not a latency-bound network op. Running the reads concurrently pipelines the GETs and hides the latency. Not the cause: - Cache size: cranking the cloud cache (2.4->50 GiB) and the L1 reader cache (128->10k) did not help; the reads are round-trip-bound, not cache-size-bound, and a high-concurrency run drains with default caches. - The L1 reader cache cannot warm here: each small partition is read to completion in one pass, so a finished reader has nothing to reuse. - The max_bytes early-return was suspected, but reverting it still failed -- it is not the cause. This is a scale limitation of the default serial read, not a regression. Follow-up: the many-small-partition read pattern is inherently inefficient (each GET amortizes over little data, readers are uncacheable). A real optimization -- coalescing reads across partitions co-resident in one L1 object, or prefetching object downloads off the fetch critical path -- is tracked separately. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> (cherry picked from commit d4a3eaa)
SlowDown/InternalError/RequestTimeout are retried by the caller (matching send_request); logging them at error tripped test log scanners in scale tests. Also treat reconnect std::system_error and ss::timed_out_error as retryable (warn), matching handle_client_transport_error. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> (cherry picked from commit 5392acc)
oleiman
approved these changes
Jun 23, 2026
oleiman
left a comment
oleiman
left a comment
Member
There was a problem hiding this comment.
clean backport. going to wait for a bit of feedback from the perf team before merging
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #30861
Conflict details
fetch_max_read_concurrency: 4config under theif cloud_topics_enabled:guard, but the target branch already has aCLOUD_TOPICS_CONFIG_STR: Truesetting in that same guarded block (not present on dev). Merged both into a singleif cloud_topics_enabled:block, keeping the target's existing setting and adding the new concurrency config.