DynamoDB auto-scaling should use max(queue) not sum

[bboreham]

We had a situation where one ingester was somehow hitting more throttling than the others, and the queue overall didn't get big enough to trigger a scale-up but that one ingester OOMed.

Any lines like this should use max() not sum():

cortex/pkg/chunk/aws/metrics_autoscaling.go

Line 30 in 9fe46d2

defaultQueueLenQuery = `sum(avg_over_time(cortex_ingester_flush_queue_length{job="cortex/ingester"}[2m]))`

Need to be a bit careful releasing that change, as it gives a very different meaning to --metrics.target-queue-length. Now I think about it, maybe we could have both - a target value which guides gentle scaling, and a max value for one ingester which triggers more urgent measures.

See also #921

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[bboreham]

let's keep this alive at least a little longer.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.