authfe shows much higher latency for cortex than distributor

[tomwilkie]

From @jml on November 2, 2016 17:18

Filing under service, as it's the most general.

The Cortex dashboard starts with a graph of perceived QPS & latency from authfe—i.e. QPS & latency for cortex routes as measured by authfe.

Sometimes, the 99%ile authfe latency is very high (e.g. O(seconds)) compared to the distributor latency (e.g. O(10ms)).

We need to understand why. Current best guess is that it's due to a discrepancy in the buckets we're using.

Copied from original issue: weaveworks/service#988

[tomwilkie]

Belongs in monitoring or cortex, but not here.

[tomwilkie]

Cortex is not exposing latency metrics for queries at all.

[tomwilkie]

Not fixed yet.

[tomwilkie]

http://frontend.dev.weave.works/admin/prometheus/graph?g0.range_input=1h&g0.expr=histogram_quantile(0.5%2C+sum(rate(cortex_request_duration_seconds_bucket%7Bjob%3D%22cortex%2Fdistributor%22%7D%5B5m%5D))+by+(le%2C+route))&g0.tab=0

[tomwilkie]

Okay they are now for some reason.

[tomwilkie]

but not in the dashboard

[tomwilkie]

Okay I think this is actually working, just queries make up a small number of the QPS.