Skip to content

feat: compaction scheduler and rate limiter#947

Merged
v0y4g3r merged 9 commits intoGreptimeTeam:developfrom
v0y4g3r:feat/compaction-scheduler
Feb 9, 2023
Merged

feat: compaction scheduler and rate limiter#947
v0y4g3r merged 9 commits intoGreptimeTeam:developfrom
v0y4g3r:feat/compaction-scheduler

Conversation

@v0y4g3r
Copy link
Copy Markdown
Contributor

@v0y4g3r v0y4g3r commented Feb 6, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This is the first part of work towards table compaction which contains:

  • table compaction scheduler that polls and executes table compaction requests in a FIFO manner
  • rate limiter framework to limit the resources used by compaction tasks;
    • currently it only limits the max concurrently running tasks number
  • compaction strategy (Picker) part is still working, but the tests involves the trait so a primitive trait is defined, but it's not the final trait definition.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch from bf55aa9 to d3ab4ec Compare February 6, 2023 14:17
@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch from d3ab4ec to e31ed97 Compare February 6, 2023 14:20
@v0y4g3r v0y4g3r marked this pull request as ready for review February 6, 2023 15:16
@v0y4g3r v0y4g3r marked this pull request as draft February 7, 2023 04:23
@v0y4g3r v0y4g3r marked this pull request as ready for review February 7, 2023 07:00
@v0y4g3r v0y4g3r requested review from evenyag, killme2008 and waynexia and removed request for evenyag and waynexia February 7, 2023 07:00
@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch 2 times, most recently from 8fb4210 to aa9ae7a Compare February 7, 2023 07:27
@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch from aa9ae7a to c64dc9a Compare February 7, 2023 07:29
Comment thread src/storage/src/compaction/scheduler.rs
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 7, 2023

Codecov Report

Merging #947 (a4fb5ab) into develop (4ae63b7) will increase coverage by 0.17%.
The diff coverage is 91.88%.

@@             Coverage Diff             @@
##           develop     #947      +/-   ##
===========================================
+ Coverage    85.79%   85.96%   +0.17%     
===========================================
  Files          443      455      +12     
  Lines        59427    61039    +1612     
===========================================
+ Hits         50984    52474    +1490     
- Misses        8443     8565     +122     
Flag Coverage Δ
rust 85.96% <91.88%> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/storage/src/error.rs 86.48% <0.00%> (-2.41%) ⬇️
src/storage/src/lib.rs 100.00% <ø> (ø)
src/storage/src/compaction/picker.rs 60.00% <60.00%> (ø)
src/storage/src/compaction/task.rs 72.72% <72.72%> (ø)
src/storage/src/compaction/dedup_deque.rs 83.33% <83.33%> (ø)
src/storage/src/compaction/scheduler.rs 93.85% <93.85%> (ø)
src/storage/src/compaction/rate_limit.rs 100.00% <100.00%> (ø)
src/object-store/src/test_util.rs 0.00% <0.00%> (-100.00%) ⬇️
src/meta-srv/src/selector/load_based.rs 28.07% <0.00%> (-49.63%) ⬇️
src/sql/src/statements/create.rs 75.00% <0.00%> (-25.00%) ⬇️
... and 85 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Comment thread src/storage/src/compaction/picker.rs
Comment thread src/storage/src/compaction/rate_limit.rs Outdated
Comment thread src/storage/src/compaction/scheduler.rs
Comment thread src/storage/src/compaction/rate_limit.rs
Comment thread src/storage/src/compaction/rate_limit.rs
Comment thread src/storage/src/compaction/scheduler.rs Outdated
Comment thread src/storage/src/compaction/scheduler.rs
Comment thread src/storage/src/compaction/rate_limit.rs
Comment thread src/storage/src/compaction/rate_limit.rs Outdated
@v0y4g3r v0y4g3r requested review from evenyag and killme2008 February 8, 2023 03:34
@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch from 1035c07 to 2b2b893 Compare February 8, 2023 06:43
Copy link
Copy Markdown
Member

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@v0y4g3r v0y4g3r requested a review from fengjiachun February 8, 2023 08:04
Comment thread src/storage/src/compaction/dedup_deque.rs Outdated
Comment thread src/storage/src/compaction/scheduler.rs
@v0y4g3r v0y4g3r requested a review from fengjiachun February 9, 2023 03:01
@v0y4g3r v0y4g3r force-pushed the feat/compaction-scheduler branch from 25b6661 to a4fb5ab Compare February 9, 2023 03:03
Copy link
Copy Markdown
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@v0y4g3r v0y4g3r merged commit 1e9918d into GreptimeTeam:develop Feb 9, 2023
@v0y4g3r v0y4g3r mentioned this pull request Feb 10, 2023
15 tasks
@v0y4g3r v0y4g3r added this to the v0.1 milestone Feb 15, 2023
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* wip: compaction schdduler

* feat: imple simple compaction scheduler

* fix: typo

* feat: add generic parameter to make scheduler friendly to tests

* chore: add more tests

* fix: CR comments

* fix: CR comments

* fix: ensure idempotency for rate limit token

* fix: Cr ct omments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants