Skip to content

feat: use raft-engine crate to reimplement logstore#799

Merged
v0y4g3r merged 19 commits intoGreptimeTeam:developfrom
v0y4g3r:feat/raft-engine-logstore
Jan 5, 2023
Merged

feat: use raft-engine crate to reimplement logstore#799
v0y4g3r merged 19 commits intoGreptimeTeam:developfrom
v0y4g3r:feat/raft-engine-logstore

Conversation

@v0y4g3r
Copy link
Copy Markdown
Contributor

@v0y4g3r v0y4g3r commented Dec 28, 2022

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

  • refactor api of logstore
  • implement a raft-engine based logstore
  • optimize compaction functionality
  • remove LocalFileLogstore

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#793

Comment thread src/log-store/src/raft_engine.rs Outdated
@v0y4g3r v0y4g3r force-pushed the feat/raft-engine-logstore branch 5 times, most recently from 0fa96ed to cc50ccc Compare January 4, 2023 03:47
@v0y4g3r v0y4g3r marked this pull request as ready for review January 4, 2023 03:47
@v0y4g3r v0y4g3r force-pushed the feat/raft-engine-logstore branch from 0aa143d to 03a7093 Compare January 4, 2023 03:51
@codecov
Copy link
Copy Markdown

codecov bot commented Jan 4, 2023

Codecov Report

Merging #799 (522d145) into develop (89d5306) will decrease coverage by 0.05%.
The diff coverage is 87.58%.

@@             Coverage Diff             @@
##           develop     #799      +/-   ##
===========================================
- Coverage    85.95%   85.89%   -0.06%     
===========================================
  Files          421      416       -5     
  Lines        55976    55160     -816     
===========================================
- Hits         48114    47381     -733     
+ Misses        7862     7779      -83     
Flag Coverage Δ
rust 85.89% <87.58%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/catalog/src/system.rs 90.18% <ø> (ø)
src/datanode/src/sql.rs 74.13% <ø> (ø)
src/log-store/src/error.rs 0.00% <ø> (ø)
src/log-store/src/lib.rs 100.00% <ø> (ø)
src/mito/src/engine.rs 98.91% <ø> (ø)
src/mito/src/table/test_util.rs 100.00% <ø> (ø)
src/storage/src/flush.rs 82.96% <ø> (ø)
src/store-api/src/logstore.rs 0.00% <0.00%> (ø)
src/log-store/src/raft_engine.rs 71.87% <71.87%> (ø)
src/log-store/src/noop.rs 77.27% <77.27%> (ø)
... and 43 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Comment thread src/log-store/proto/logstore.proto Outdated
Comment thread src/log-store/Cargo.toml Outdated
Comment thread src/log-store/Cargo.toml Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs
Comment thread src/script/Cargo.toml Outdated
Comment thread src/store-api/src/logstore.rs
@v0y4g3r v0y4g3r force-pushed the feat/raft-engine-logstore branch 2 times, most recently from e1afede to fdba708 Compare January 4, 2023 09:46
Comment thread src/log-store/Cargo.toml Outdated
Comment thread src/log-store/src/error.rs Outdated
Comment thread src/log-store/src/error.rs
Comment thread src/log-store/src/raft_engine.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/storage/src/region/tests.rs
Comment thread src/log-store/src/raft_engine.rs
Comment thread src/log-store/build.rs
@v0y4g3r v0y4g3r requested review from evenyag and fengjiachun January 5, 2023 06:32
@v0y4g3r v0y4g3r force-pushed the feat/raft-engine-logstore branch from 58f30df to 2de59ff Compare January 5, 2023 07:18
Comment thread src/log-store/src/raft_engine/log_store.rs Outdated
Comment thread src/log-store/src/raft_engine/log_store.rs
Copy link
Copy Markdown
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/store-api/src/logstore/entry.rs Outdated
Comment thread src/storage/src/wal.rs Outdated
Comment thread src/datanode/src/instance.rs
Comment thread src/datanode/src/instance.rs Outdated
Comment thread src/log-store/src/raft_engine.rs Outdated
@v0y4g3r
Copy link
Copy Markdown
Contributor Author

v0y4g3r commented Jan 5, 2023

@waynexia PTAL

Copy link
Copy Markdown
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@v0y4g3r v0y4g3r merged commit 8f5ecef into GreptimeTeam:develop Jan 5, 2023
v0y4g3r added a commit that referenced this pull request Jan 6, 2023
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* chore: remove useless method in Entry trait, add proto definition for entry and namespace

* feat: add proto definition for raft-engine based logstore

* feat: introduce RaftEngineLogstore

* feat: impl read for raft engine log store

* feat: impl raft engine logstore

* feat: raft engine logstore start and stop

* feat: add purge bg task

* fix: license header

* fix: clippy

* fix: toml files

* feat: add some test cases

* fix: CR comments

* fix: CR comments

* fix: check namespace validity and state of logstore

* fix: CR comments; add config item to control sync/async flush per write

* fix: remove unused error variants

* fix: unit tests

* fix: use compare and exchange to stop logstore

* fix: CR comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants