Skip to content

feat(inverted_index.create): add read/write for external intermediate files#2942

Merged
zhongzc merged 7 commits intoGreptimeTeam:developfrom
zhongzc:zhongzc/inverted-index-create-intermediate
Dec 18, 2023
Merged

feat(inverted_index.create): add read/write for external intermediate files#2942
zhongzc merged 7 commits intoGreptimeTeam:developfrom
zhongzc:zhongzc/inverted-index-create-intermediate

Conversation

@zhongzc
Copy link
Copy Markdown
Collaborator

@zhongzc zhongzc commented Dec 15, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

  • Add related format and codec for intermediate files
  • Add the facility for reading and writing intermediate files, using FramedRead and FramedWrite, to achieve streaming read-write capabilities

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#2705

next PRs to be proposed:
feat(inverted_index.create): add external sorter
feat(inverted_index.create): add index creator

… files

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 15, 2023

Codecov Report

Merging #2942 (f592c18) into develop (e35a494) will decrease coverage by 0.50%.
Report is 5 commits behind head on develop.
The diff coverage is 93.97%.

❗ Current head f592c18 differs from pull request most recent head 7d131f1. Consider uploading reports for the commit 7d131f1 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2942      +/-   ##
===========================================
- Coverage    85.05%   84.55%   -0.50%     
===========================================
  Files          754      756       +2     
  Lines       119742   119908     +166     
===========================================
- Hits        101847   101392     -455     
- Misses       17895    18516     +621     

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc force-pushed the zhongzc/inverted-index-create-intermediate branch from c4dfecc to 2c7a737 Compare December 15, 2023 15:30
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Copy link
Copy Markdown
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 😋

Comment thread src/index/Cargo.toml Outdated
Comment thread src/index/src/inverted_index/create/sort/intermediate_rw.rs
Copy link
Copy Markdown
Contributor

@evenyag evenyag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/index/Cargo.toml Outdated
Copy link
Copy Markdown
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc added this pull request to the merge queue Dec 18, 2023
Merged via the queue into GreptimeTeam:develop with commit 029ff2f Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants