Make ruler HA

[jml]

Currently the ruler is not HA. There's only one instance that runs all the rules. If it goes down, rules are not being processed.

[jml]

Current plan:

• store state about how long condition has been firing for somewhere else (probably dynamodb on weaveworks deployment)
• have rule evaluator that gets sent a rule (group?) and then evaluates it, storing state about firing conditions in thing above, and
• have single scheduler that sends requests for rule evaluation to evaluators, relying on k8s to do load balancing for us

We'll sort out the config mess later.

Not personally going to do any of this any time soon, although I would personally like to do it. I also think this should wait until #555 is done.

[jml]

prometheus/prometheus#422 is relevant here. AFAICT, much of the discussion on the ticket assumes that Prometheus's metric store is the only acceptable place to store 'for' state.

Cortex doesn't have that constraint, but ideally the work would be done such that the storage is behind a golang interface, such that the current in-memory approach and a hypothetical TSDB-backed approach would both work, in addition to whatever dynamodb / bigtable thing we do.

[jml]

Lots of this came up from a conversation I had w/ @juliusv a while back. The good ideas are probably his, the bad ones mine.

[tomwilkie]

Fixed in #1258. Ruler is now horizontally scalable, and rules are sharded using the ring. I believe this should be enough.