Skip to content

Series index: don't rely so heavily on NOSQL to dedupe writes. #957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tomwilkie opened this issue Aug 24, 2018 · 1 comment
Closed

Series index: don't rely so heavily on NOSQL to dedupe writes. #957

tomwilkie opened this issue Aug 24, 2018 · 1 comment

Comments

@tomwilkie
Copy link
Contributor

The series store will write the index entries (metric:label -> series) for every chunk, and rely on the underlying store to dedupe these. We could keep a cache of the most recently written entries and not bother writing ones we already know exist, removing some write load on the database.

For an ingester with ~1m series, an average chunk size of 6hs, and 10 labels per chunks, we're writing ~500 entries / sec. A cache of 10m entries would potentially reduce this to ~entries / sec.

@bboreham
Copy link
Contributor

bboreham commented Sep 3, 2018

Somewhat related to #607

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants