Skip to content

feat(inverted_index.create): add external sorter#2950

Merged
zhongzc merged 27 commits intoGreptimeTeam:developfrom
zhongzc:zhongzc/inverted-index-create-sorter
Dec 19, 2023
Merged

feat(inverted_index.create): add external sorter#2950
zhongzc merged 27 commits intoGreptimeTeam:developfrom
zhongzc:zhongzc/inverted-index-create-sorter

Conversation

@zhongzc
Copy link
Copy Markdown
Collaborator

@zhongzc zhongzc commented Dec 18, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

  • Add the Sorter trait for handling sorting of incremental input.
  • Add the ExternalSorter implementation, together with the dependency ExternalTempFileProvider to enable access to external files. The ExternalSorter will employ external sorting, has a parameter memory_usage_threshold, and tracks memory usage to ensure that the sorting is completed with a minimal memory footprint.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#2705

next PR to be proposed:
feat(inverted_index.create): add index creator

… files

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
…ls to poll

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Comment thread src/index/src/inverted_index/create/sort.rs Outdated
Comment thread src/index/src/inverted_index/create/sort/external_sort.rs Outdated
Comment thread src/index/src/inverted_index/create/sort/external_sort.rs Outdated
Comment thread src/index/src/inverted_index/create/sort/external_sort.rs
Comment thread src/index/src/inverted_index/create/sort/external_sort.rs Outdated
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 18, 2023

Codecov Report

Merging #2950 (bfed3d5) into develop (6b8dbcf) will decrease coverage by 0.50%.
The diff coverage is 98.21%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2950      +/-   ##
===========================================
- Coverage    85.77%   85.27%   -0.50%     
===========================================
  Files          758      762       +4     
  Lines       122730   123123     +393     
===========================================
- Hits        105267   104995     -272     
- Misses       17463    18128     +665     

Comment thread src/index/src/inverted_index/create/sort/external_sort.rs Outdated
Co-authored-by: Yingwen <realevenyag@gmail.com>
Copy link
Copy Markdown
Member

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/index/src/inverted_index/create/sort/external_sort.rs
Comment thread src/index/src/inverted_index/create/sort/external_sort.rs
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc force-pushed the zhongzc/inverted-index-create-sorter branch from 6c87118 to 6cd5cea Compare December 19, 2023 07:29
@killme2008 killme2008 added this pull request to the merge queue Dec 19, 2023
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Dec 19, 2023
@killme2008
Copy link
Copy Markdown
Member

@zhongzc There is a clippy warning.

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc
Copy link
Copy Markdown
Collaborator Author

zhongzc commented Dec 19, 2023

There is a clippy warning.

Due to the toolchain update of the develop branch, fixed now.

@zhongzc zhongzc enabled auto-merge December 19, 2023 08:04
@zhongzc zhongzc added this pull request to the merge queue Dec 19, 2023
Merged via the queue into GreptimeTeam:develop with commit 83de399 Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants