Tune chunk size

[jml]

From @tomwilkie

Currently 10mins, should be 1hr.

Copied from original issue: tomwilkie/frankenstein#10

[jml]

From @jml

Why should it be 1hr?

[jml]

From @tomwilkie

We should investigate what the right number is, but the parameters are:

• data batched up in the ingesters is a risk of loss in event of machine failure. we should bound this.
• chunks are 1kb, and making bigger chunks doesn't necessarily make things more efficient. we should try and fill chunks as much as possible (something we already monitoring for).

Ticket should really say "max 1hr" to bound the loss, if that give good utilization

[jml]

This is possibly related to the dynamo errors we are seeing in #85

[juliusv]

Oh wow yeah, the default chunk max age of 10 minutes seems way too low. I'm wondering why we're still achieving such decent chunk utilization (sum(cortex_ingester_chunk_utilization_sum) / sum(cortex_ingester_chunk_utilization_count) is around 0.43) with such a low max age. Under certain circumstances, chunks can last for hours or days, so maybe it's the frequent scraping plus noisiness of the data that makes the chunks fill up that fast. Still, I would set the max age to an hour or so (as you said, it depends a bit on our risk profile, of course).

[tomwilkie]

I suspect it can't flush chunks quickly enough, and therefore they are
getting more than 10mins worth of data.

On Wednesday, 2 November 2016, Julius Volz notifications@github.com wrote:

Oh wow yeah, the default chunk max age of 10 minutes seems way too low.
I'm wondering why we're still achieving such decent chunk utilization (
sum(cortex_ingester_chunk_utilization_sum) / sum(cortex_ingester_chunk_
utilization_count) is around 0.43) with such a low max age. Under certain
circumstances, chunks can last for hours or days, so maybe it's the
frequent scraping plus noisiness of the data that makes the chunks fill up
that fast. Still, I would set the max age to an hour or so (as you said, it
depends a bit on our risk profile, of course).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbGhYFnewSerf7ltmWR169MdiiOQ0bfks5q6SEHgaJpZM4J5RTd
.

[juliusv]

I suspect it can't flush chunks quickly enough, and therefore they are getting more than 10mins worth of data.

At least the failures should not have a big effect because during normal operation, only ~4% of chunk puts fail (sum(rate(cortex_ingester_chunk_store_failures_total[1m])) / sum(rate(cortex_ingester_chunk_utilization_count[1m])) -> 0.043). Maybe general latency in non-failed puts delays things somewhat, but the effect cannot be huge, as sum(cortex_ingester_memory_chunks) / sum(cortex_ingester_memory_series) shows us that there's just 1.12 chunks per series in memory at a given time (there's always at least one open head chunk for active series).

[tomwilkie]

Actually make sense, since we're on doubledelta (not varbit). So its about 3.3 bytes per sample, at 15s scrape interval == about 20mins per chunk. With 10mins, you'd expect 50% utilisation.