Fix removal of persisted chunks from memory #64
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There's an OBOB in there. According to my understanding, it would currently keep one already persisted chunk in memory and then persist it again every minute, until there's another full chunk (then the old one would finally get dropped). Indeed, looking at how many series the ingester has in memory: prism_ingester_memory_series ...and comparing that to how many chunks are being written out per minute: rate(prism_ingester_chunk_utilization_count[1h]) * 60 The number of chunks being persisted every minute is coming quite close to the number of series in memory. We should watch for a sharp drop of the latter rate after merging this.
jml
approved these changes
Oct 24, 2016
|
@jml It's safe to revert, but it should be a bug fix. If there's no drop in persists, my thinking is off... |
tomwilkie
added a commit
that referenced
this pull request
Jan 21, 2017
b783528 Tweak test script so it can be run on a mca a3b18bf Merge pull request #65 from weaveworks/fix-integration-tests ecb5602 Fix integration tests f9dcbf6 ... without tab (clearly not my day) a6215c3 Add break I forgot 0e6832d Remove incorrectly added tab eb26c68 Merge pull request #64 from weaveworks/remove-test-package-linting f088e83 Review feedback 2c6e83e Remove test package linting git-subtree-dir: tools git-subtree-split: b783528b75f6ba52a51832bd087bd8347b26f131
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There's an OBOB in there. According to my understanding, it would
currently keep one already persisted chunk in memory and then persist it
again every minute, until there's another full chunk (then the old one
would finally get dropped).
Indeed, looking at how many series the ingester has in memory:
prism_ingester_memory_series
...and comparing that to how many chunks are being written out per
minute:
rate(prism_ingester_chunk_utilization_count[1h]) * 60
The number of chunks being persisted every minute is coming quite close
to the number of series in memory. We should watch for a sharp drop of
the latter rate after merging this.