feat(parquet): introduce inverted index applier to reader#3130
Merged
waynexia merged 16 commits intoGreptimeTeam:mainfrom Jan 11, 2024
Merged
feat(parquet): introduce inverted index applier to reader#3130waynexia merged 16 commits intoGreptimeTeam:mainfrom
waynexia merged 16 commits intoGreptimeTeam:mainfrom
Conversation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3130 +/- ##
==========================================
- Coverage 85.48% 85.04% -0.45%
==========================================
Files 822 822
Lines 134403 134560 +157
==========================================
- Hits 114899 114431 -468
- Misses 19504 20129 +625 |
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
killme2008
approved these changes
Jan 10, 2024
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Collaborator
Author
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
…x-sst-reader-intro
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
evenyag
reviewed
Jan 11, 2024
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Collaborator
Author
evenyag
approved these changes
Jan 11, 2024
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
Add the index applier in the Parquet reader to filter row groups:
inverted_index_availableproperty toSstInfoandFileMetarow_groups_to_readmethod forParquetReaderBuilder, which returns row groups that still need to be read after being filtered through the inverted index and min-max indexMoreover, once
inverted_index_availablebecomes a property ofFileMeta, it not only represents a single SST File but also includes the associated index files. Therefore, when handling deletions, they should be deleted together.Checklist
Refer to a related PR or issue link (optional)
#2705