Migrate several indexing and compaction integration tests to embedded-tests #18207
+2,485
−3,235
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kfaraz
changed the title
ITPerfectRollupParallelIndexTest and ITBestEffortRollupParallelIndexTest to embedded-tests
…druid into add_embedded_perfect_rollup_test
kfaraz
changed the title
ITPerfectRollupParallelIndexTest and ITBestEffortRollupParallelIndexTest to embedded-tests![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...ests/src/test/java/org/apache/druid/testing/embedded/compact/EmbeddedAutoCompactionTest.java
Fixed
Show fixed
Hide fixed
...rc/test/java/org/apache/druid/testing/embedded/indexing/EmbeddedKafkaClusterMetricsTest.java
Fixed
Show fixed
Hide fixed
10 tasks
kfaraz
added a commit
that referenced
this pull request
Jul 8, 2025
Bug: Concurrent append uses lock of type APPEND which always uses a lock version of epoch 1970-01-01. This can cause data loss in a flow as follows: - Ingest data using an APPEND task to an empty interval - Mark all the segments as unused - Re-run the APPEND task - Data is not visible since old segment IDs (now unused) are allocated again Fix: In segment allocation, do not reuse an old segment ID, used or unused. This fix was already done for some cases back in #16380 . An embedded test for this has been included in #18207
| @@ -0,0 +1,3 @@ | |||
| {"timestamp": "2013-08-31T01:02:33Z", "page": "Gypsy Danger", "language" : "en", "tags": ["t1", "t2"], "user" : "nuclear", "unpatrolled" : "true", "newPage" : "true", "robot": "false", "anonymous": "false", "namespace":"article", "continent":"North America", "country":"United States", "region":"Bay Area", "city":"San Francisco", "added": 57, "deleted": 200, "delta": -143} | |||
There was a problem hiding this comment.
drive by comment/nit: i know this isn't new, but imo we should fix the problem of referring to this dataset as 'wikipedia' because it is confusing with the quickstart wikipedia data which is also going to be used in some tests, and this stuff only has a vaguely similar schema, maybe tiny-wikipedia or something to indicate that its a very small dataset would help clear things up?
There was a problem hiding this comment.
Thanks for the suggestion! I will rename these datasets accordingly.
10 tasks
10 tasks
Comment on lines
+354
to
+358
| InlineSchemaDataSourceCompactionConfig | ||
| .builder() | ||
| .forDataSource(dataSource) | ||
| .withSkipOffsetFromLatest(Period.seconds(0)) | ||
| .withMaxRowsPerSegment(compactedMaxRowsPerSegment) |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Changes
TaskBuilderutility to createTaskobjects using fluent syntaxembedded-testsTest migrations
ITPerfectRollupParallelIndexTestEmbeddedIndexParallelTaskTestITBestEffortRollupParallelIndexTestEmbeddedIndexParallelTaskTest, added as a new test parameter which uses dynamic partitioningITAutoCompactionTestEmbeddedAutoCompactionTestITAutoCompactionLockContentionTestEmbeddedAutoCompactionUpgradeTestITAutoCompactionLockContentionTestEmbeddedKafkaClusterMetricsTest, methodtest_ingestClusterMetrics_compactionSkipsLockedIntervals()ITCompactionTaskTestEmbeddedCompactionTaskTestITCompactionSparseColumnTestEmbeddedCompactionSparseColumnTestITOverlordResourceTestEmbeddedOverlordClientTestITOverlordResourceNotFoundTestEmbeddedOverlordClientTestNew nested tests
EmbeddedCentralizedSchemaPublishFailureTestfor the groupcds-task-schema-publish-disabledEmbeddedCentralizedSchemaMetadataQueryDisabledTestfor the groupcds-coordinator-metadata-query-disabledTest run times
EmbeddedIndexParallelTaskTest(indexer)EmbeddedAutoCompactionTest(indexer)EmbeddedCompactionSparseColumnTest(indexer)ITPerfectRollupParallelIndexTestITPerfectRollupParallelIndexTest(Indexer, shuffle deep store test, only 1 config changed)
ITPerfectRollupParallelIndexTest(MM, shuffle deep store test, only 1 config changed)
ITBestEffortRollupParallelIndexTestITBestEffortRollupParallelIndexTestITAutoCompactionTest(middle manager)ITAutoCompactionTest(indexer)ITCompactionSparseColumnTest(indexer)This PR has: