Skip to content

Larger buckets in chunk index #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jml opened this issue Sep 9, 2016 · 5 comments
Closed

Larger buckets in chunk index #10

jml opened this issue Sep 9, 2016 · 5 comments
Assignees
@juliusv

Comments

@jml
Copy link
Contributor

jml commented Sep 9, 2016

From @tomwilkie

Should reduce index size, super important for scalability.

Copied from original issue: tomwilkie/frankenstein#9
@jml
Copy link
Contributor Author

jml commented Sep 9, 2016

From @jml

@tomwilkie Do you have some idea for how much larger they should be, or thoughts on how to determine how much larger?

@jml
Copy link
Contributor Author

jml commented Sep 9, 2016

From @tomwilkie

Chunk index 'big' bucket size mainly affects the number of range queries you need to do on query. Smaller buckets = more range queries = more dynamodb cost & slower queries.

OTOH larger buckets mean more chunks / bucket, which means more items returned in the range queries. Limit on dynamodb range queries is 25 items before you start paging. So we should look and see how many chunks / query we are getting, and tune that until it hits 25. I have a feeling it will be 24hrs...

@jml jml mentioned this issue Sep 9, 2016
Closed
@jml jml modified the milestone: SF Prometheus Meetup Sep 9, 2016
@rade rade removed this from the SF Prometheus Meetup milestone Nov 7, 2016
@tomwilkie
Copy link
Contributor

We need to gather stats on pages per query before deciding on this, but its looks like we are going toward bigger chunks, so 24hr buckets might be a good idea.

@juliusv juliusv added this to the Dogfooding milestone Jan 3, 2017
@tomwilkie tomwilkie mentioned this issue Jan 3, 2017
Closed
6 tasks
@juliusv juliusv mentioned this issue Jan 4, 2017
Merged
@tomwilkie
Copy link
Contributor

Code is in, flag just needs deploying to dev and testing.

@tomwilkie tomwilkie mentioned this issue Jan 5, 2017
Merged
@tomwilkie tomwilkie removed this from the Dogfooding milestone Jan 6, 2017
@tomwilkie
Copy link
Contributor

Has been deployed to prod and will go live tomorrow.

roystchiang pushed a commit to roystchiang/cortex that referenced this issue Apr 6, 2022
@alvinlin123
Merge pull request cortexproject#10 from aws-observability/1.7.0
d7c8203 
Cortex Release
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juliusv juliusv

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rade @jml @tomwilkie @juliusv