Skip to content

feat(mito): Implement file cache#3022

Merged
killme2008 merged 8 commits into
GreptimeTeam:mainfrom
evenyag:feat/file-cache
Jan 3, 2024
Merged

feat(mito): Implement file cache#3022
killme2008 merged 8 commits into
GreptimeTeam:mainfrom
evenyag:feat/file-cache

Conversation

@evenyag
Copy link
Copy Markdown
Contributor

@evenyag evenyag commented Dec 27, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements a cache for files in the WriteCache. The design is similar to the ReadCache.

The differences are

  • This cache always caches the whole SST file as it serves as a write-through cache
  • This cache doesn't read files from remote object stores

This cache has the following features:

  • Caches files on local disk (the local store).
  • Uses {region_id}.{file_id} as cache key and cached file name
  • Uses an in-memory LRU cache as file index
  • If the LRU cache removes the key, it also evicts the file on the local store.
  • Recovers the file index from the local store

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

@github-actions github-actions Bot added docs-not-required This change does not impact docs. Size: M labels Dec 27, 2023
@evenyag evenyag changed the base branch from develop to main December 28, 2023 03:53
@evenyag evenyag marked this pull request as ready for review December 28, 2023 08:01
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 28, 2023

Codecov Report

Attention: 20 lines in your changes are missing coverage. Please review.

Comparison is base (b58296d) 85.54% compared to head (2ae9523) 85.05%.
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3022      +/-   ##
==========================================
- Coverage   85.54%   85.05%   -0.49%     
==========================================
  Files         793      794       +1     
  Lines      128447   128743     +296     
==========================================
- Hits       109880   109508     -372     
- Misses      18567    19235     +668     

Comment thread src/mito2/src/cache/file_cache.rs Outdated
Copy link
Copy Markdown
Collaborator

@zhongzc zhongzc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused about the responsibilities. Cache is not responsible for storing files but deleting files, but it's not a big problem. LGTM

Copy link
Copy Markdown
Member

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@killme2008 killme2008 added this pull request to the merge queue Jan 3, 2024
Merged via the queue into GreptimeTeam:main with commit 2e686fe Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required This change does not impact docs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants