ct: cloud topics compaction configs #29287
Merged
dotnwat merged 5 commits intoJan 29, 2026
Merged
Conversation
For controlling the size of L1 objects produced during compaction.
For controlling the frequency of cloud topics compaction (separate from local storage compaction interval).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces three new configuration parameters for cloud topics compaction: cloud_topics_compaction_max_object_size, cloud_topics_compaction_interval_ms, and cloud_topics_compaction_key_map_memory. These configurations allow independent tuning of cloud topics compaction behavior, separate from the existing general compaction settings.
Changes:
- Added three new cloud topics compaction configuration properties with appropriate defaults and bounds
- Updated memory reservation system to allocate separate memory for cloud topics compaction
- Modified compaction worker, scheduler, and sink components to use the new cloud-topics-specific configurations instead of general compaction settings
- Updated tests to reference the new configuration parameter names
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/v/config/configuration.h |
Declared three new configuration properties for cloud topics compaction |
src/v/config/configuration.cc |
Defined the new configuration properties with defaults and metadata |
src/v/resource_mgmt/memory_groups.h |
Added struct and methods for cloud topics compaction memory reservation |
src/v/resource_mgmt/memory_groups.cc |
Implemented memory reservation logic for cloud topics compaction |
src/v/cloud_topics/level_one/compaction/worker.h |
Added config binding for compaction interval |
src/v/cloud_topics/level_one/compaction/worker.cc |
Updated worker to use new cloud topics compaction configs and memory reservation |
src/v/cloud_topics/level_one/compaction/sink.h |
Changed max object size from constant to config binding |
src/v/cloud_topics/level_one/compaction/sink.cc |
Updated sink to use configurable max object size |
src/v/cloud_topics/level_one/compaction/scheduler.h |
Updated comment to reference new config name |
src/v/cloud_topics/level_one/compaction/scheduler.cc |
Changed scheduler to bind to new cloud topics compaction interval config |
src/v/cloud_topics/level_one/compaction/log_info_collector.cc |
Updated to use new cloud topics compaction interval config |
tests/rptest/tests/random_node_operations_smoke_test.py |
Added cloud topics compaction interval config to test setup |
tests/rptest/tests/cloud_topics/iceberg_test.py |
Replaced log compaction interval with cloud topics compaction interval |
tests/rptest/tests/cloud_topics/e2e_test.py |
Updated test to use new cloud topics compaction config parameters |
src/v/cloud_topics/level_one/compaction/tests/reducer_test.cc |
Added mock binding for max object size in test |
src/v/cloud_topics/level_one/compaction/tests/scheduler_test.cc |
Updated test to use new config name |
src/v/cloud_topics/level_one/compaction/tests/scheduler_multithread_test.cc |
Updated test to use new config name |
src/v/cloud_topics/level_one/compaction/tests/BUILD |
Added config dependency for tests |
src/v/cloud_topics/level_one/compaction/BUILD |
Added config and memory_groups dependencies |
WillemKauf
force-pushed
the
cloud_topics_compaction_configs
branch
2 times, most recently
from
8003aab to
1951e44
Compare
Collaborator
Retry command for Build#79169please wait until all jobs are finished before running the slash command |
WillemKauf
force-pushed
the
cloud_topics_compaction_configs
branch
2 times, most recently
from
f6f9fa6 to
228c3d5
Compare
And the related memory group reservation, separate from that used for local storage.
We were trying to catch the wrong kind of exception here...
WillemKauf
force-pushed
the
cloud_topics_compaction_configs
branch
from
228c3d5 to
8b2dd83
Compare
Collaborator
Retry command for Build#79199please wait until all jobs are finished before running the slash command |
Collaborator
dotnwat
reviewed
Jan 21, 2026
| : _compaction_reserved_memory( | ||
| compaction.reserved_bytes(total_available_memory)) | ||
| , _cloud_topics_compaction_reserved_memory( | ||
| cloud_topics_compaction.reserved_bytes()) |
Member
There was a problem hiding this comment.
curious why reserved_bytes here isn't a function of the total available memory like the other reservations?
Contributor
Author
There was a problem hiding this comment.
I didn't add a option to configure the reserved memory as a percentage a la storage_compaction_key_map_memory_limit_percent, so we will always just reserve a flat cloud_topics_compaction_key_map_memory.
Member
There was a problem hiding this comment.
ahh, that's what I had figured. thanks
dotnwat
approved these changes
Jan 21, 2026
Contributor
Author
|
One side effect of this is that we now allocate 256 MiB (per shard) by default for local storage compaction + cloud topics compaction. |
Contributor
Author
|
@dotnwat based on our conversations here, is this mergeable in its current form? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cloud_topics_compaction_max_object_sizecloud_topics_compaction_interval_mscloud_topics_compaction_key_map_memoryBackports Required
Release Notes