feat(mito): enable inverted index#3158
Merged
zhongzc merged 11 commits intoGreptimeTeam:mainfrom Jan 15, 2024
Merged
Conversation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
7 tasks
… Engine Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3158 +/- ##
==========================================
- Coverage 85.43% 85.09% -0.34%
==========================================
Files 823 829 +6
Lines 134922 135714 +792
==========================================
+ Hits 115268 115492 +224
- Misses 19654 20222 +568 |
zhongzc
commented
Jan 15, 2024
evenyag
reviewed
Jan 15, 2024
Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
82322c2 to
6403d01
Compare
…to field of WriteCache Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
fengjiachun
reviewed
Jan 15, 2024
evenyag
approved these changes
Jan 15, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
This PR has enabled the inverted index for the mito engine.
Main changes:
IntermediateManagerwith the objectives:IntermediateManagerbe a singleton to prevent repetitive deletions that could cause errors.IntermediateLocation. Previously, the path was placed near the data files, but with the introduction of theIntermediateManager, data files and intermediate files were completely isolated, hence a more customized path was adopted.Indexer, embedded withinParquetWriter.Indexeris used to create the index and hides error handling internally, exposing three methods toParquetWriterthat do not return errors:update,finish, andabort.InvertedIndexConfigtoMitoConfig, which includes the following parameters:create_on_flush,create_on_compaction,apply_on_queryintermediate_path: The file system path for intermediatesmem_threshold_on_create: Memory control when creating the indexMitoConfig::sanitize, takingdata_homeas an input because bothintermediate_pathandexperimental_write_cache_pathdepend ondata_homefor setting default paths.ScanRegiondisables index apply during queries based on theapply_on_queryparameter.SstWriteRequestintroducescreate_inverted_indexandmem_threshold_index_create. The decision to create an index during flush and compaction is controlled by passing the configs fromMitoConfigto these two parameters.Checklist
Refer to a related PR or issue link (optional)
#2705