Ingesters log too much

[tomwilkie]

9m log entries in 8hrs, which is about 40% of the clusters output (another 40% being kube-dns, which is now fixed).

66% of the logs lines from the ingester are “Flushing chunk".

[tomwilkie]

[tomwilkie]

• “Flushing” occurs 9,126,042 times of 18,512,409 log lines.
• 2,040,808 log lines don’t include include POST or Flushing: https://weave.loggly.com/search#terms=-Flushing%20-POST&from=2016-09-28T17%3A27%3A56.901Z&until=2016-09-29T17%3A27%3A56.901Z&source_group=&filter=json.kubernetes.namespace_name%3Bprism
  • Generally its "Got update to ring” and "Heartbeating to consul…"

[juliusv]

I'd say kill the logs for these things, use metrics instead! (except for errors and their exact messages)

[jml]

I was going to fix the "Flushing" one as part of #37, but it's actually from prism.go in our prometheus fork. I don't know what the state of that is (@juliusv says it's going away), but it's a one line fix, so I've submitted PR tomwilkie/prometheus#82

[jml]

Probably fixed by #46. Please re-open if not.

[jml]

Now getting cycles of these:

time="2016-10-18T16:13:23Z" level=info msg="Done flushing chunks." source="ingester.go:409"
time="2016-10-18T16:14:22Z" level=info msg="Flushing chunks... (exiting: false)" source="ingester.go:386"
time="2016-10-18T16:14:22Z" level=info msg="Flushing user 2..." source="ingester.go:412"
time="2016-10-18T16:14:22Z" level=info msg="Flushing user 1..." source="ingester.go:412"
time="2016-10-18T16:14:23Z" level=info msg="Done flushing user 1." source="ingester.go:433"
time="2016-10-18T16:14:23Z" level=info msg="Done flushing user 2." source="ingester.go:433"

I haven't yet looked up what they signify.