Merge remote-tracking branch 'upstream/master' into vendor

Signed-off-by: Ganesh Vernekar <[email protected]>

Author: codesome

LICENSE

@@ -1,4 +1,3 @@
-
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -176,7 +175,18 @@
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2014-2017 Weaveworks Ltd.
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

Makefile

@@ -50,6 +50,7 @@ $(foreach exe, $(EXES), $(eval $(call dep_exe, $(exe))))
 pkg/ingester/client/cortex.pb.go: pkg/ingester/client/cortex.proto
 pkg/ring/ring.pb.go: pkg/ring/ring.proto
 pkg/querier/frontend/frontend.pb.go: pkg/querier/frontend/frontend.proto
+pkg/querier/queryrange/queryrange.pb.go: pkg/querier/queryrange/queryrange.proto
 pkg/chunk/storage/caching_index_client.pb.go: pkg/chunk/storage/caching_index_client.proto
 pkg/distributor/ha_tracker.pb.go: pkg/distributor/ha_tracker.proto
 all: $(UPTODATE_FILES)

docs/architecture.md

@@ -24,6 +24,8 @@ The **distributor** service is responsible for handling samples written by Prome
 
 Distributors communicate with ingesters via [gRPC](https://grpc.io). They are stateless and can be scaled up and down as needed.
 
+If the HA Tracker is enabled, the Distributor will deduplicate incoming samples that contain both a cluster and replica label. It talks to a KVStore to store state about which replica per cluster it's accepting samples from for a given user ID. Samples with one or neither of these labels will be accepted by default.
+
 #### Hashing
 
 Distributors use consistent hashing, in conjunction with the (configurable) replication factor, to determine *which* instances of the ingester service receive each sample.
@@ -147,4 +149,4 @@ The interface works somewhat differently across the supported databases:
 
 A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Cortex has evolved, mainly in an attempt to better load balance writes and improve query performance.
 
-> The current schema recommendation is the **v10 schema**.
+> The current schema recommendation is the **v10 schema**.

docs/arguments.md

@@ -92,6 +92,67 @@ The ingester query API was improved over time, but defaults to the old behaviour
 - `-distributor.extra-query-delay`
    This is used by a component with an embedded distributor (Querier and Ruler) to control how long to wait until sending more than the minimum amount of queries needed for a successful response.
 
+- `distributor.ha-tracker.enable-for-all-users`
+   Flag to enable, for all users, handling of samples with external labels identifying replicas in an HA Prometheus setup. This defaults to false, and is technically defined in the Distributor limits.
+
+- `distributor.ha-tracker.enable` 
+   Enable the distributors HA tracker so that it can accept samples from Prometheus HA replicas gracefully (requires labels). Global (for distributors), this ensures that the necessary internal data structures for the HA handling are created. The option `enable-for-all-users` is still needed to enable ingestion of HA samples for all users.
+
+### Ring/HA Tracker Store
+
+The KVStore client is used by both the Ring and HA Tracker.
+- `{ring,distributor.ha-tracker}.prefix`
+   The prefix for the keys in the store. Should end with a /. For example with a prefix of foo/, the key bar would be stored under foo/bar.
+- `{ring,distributor.ha-tracker}.store`
+   Backend storage to use for the ring (consul, etcd, inmemory).
+
+#### Consul
+
+By default these flags are used to configure Consul used for the ring. To configure Consul for the HA tracker,
+prefix these flags with `distributor.ha-tracker.`
+
+- `consul.hostname`
+   Hostname and port of Consul.
+- `consul.acltoken`
+   ACL token used to interact with Consul.
+- `consul.client-timeout`
+   HTTP timeout when talking to Consul.
+- `consul.consistent-reads`
+   Enable consistent reads to Consul.
+
+#### etcd
+
+By default these flags are used to configure etcd used for the ring. To configure etcd for the HA tracker,
+prefix these flags with `distributor.ha-tracker.`
+
+- `etcd.endpoints`
+   The etcd endpoints to connect to.
+- `etcd.dial-timeout`
+   The timeout for the etcd connection.
+- `etcd.max-retries`
+   The maximum number of retries to do for failed ops.
+
+### HA Tracker
+
+HA tracking has two of it's own flags:
+- `distributor.ha-tracker.cluster`
+   Prometheus label to look for in samples to identify a Prometheus HA cluster. (default "cluster")
+- `distributor.ha-tracker.replica`
+   Prometheus label to look for in samples to identify a Prometheus HA replica. (default "__replica__")
+
+It's reasonable to assume people probably already have a `cluster` label, or something similar. If not, they should add one along with `__replica__`
+via external labels in their Prometheus config.
+
+HA Tracking looks for the two labels (which can be overwritten per user)
+
+It also talks to a KVStore and has it's own copies of the same flags used by the Distributor to connect to for the ring.
+- `distributor.ha-tracker.failover-timeout`
+   If we don't receive any samples from the accepted replica for a cluster in this amount of time we will failover to the next replica we receive a sample from. This value must be greater than the update timeout (default 30s)
+- `distributor.ha-tracker.store`
+   Backend storage to use for the ring (consul, etcd, inmemory). (default "consul")
+- `distributor.ha-tracker.update-timeout`
+   Update the timestamp in the KV store for a given cluster/replica only after this amount of time has passed since the current stored timestamp. (default 15s)
+
 ## Ingester
 
 - `-ingester.normalise-tokens`
@@ -185,4 +246,4 @@ Valid fields are (with their corresponding flags for default values):
 
 - `s3.force-path-style`
 
-  Set this to `true` to force the request to use path-style addressing (`http://s3.amazonaws.com/BUCKET/KEY`). By default, the S3 client will use virtual hosted bucket addressing when possible (`http://BUCKET.s3.amazonaws.com/KEY`).
+  Set this to `true` to force the request to use path-style addressing (`http://s3.amazonaws.com/BUCKET/KEY`). By default, the S3 client will use virtual hosted bucket addressing when possible (`http://BUCKET.s3.amazonaws.com/KEY`).

go.mod

@@ -36,6 +36,7 @@ require (
 	github.com/hashicorp/go-cleanhttp v0.5.1
 	github.com/jonboulle/clockwork v0.1.0
 	github.com/json-iterator/go v1.1.6
+	github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
 	github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 // indirect
 	github.com/lann/builder v0.0.0-20150808151131-f22ce00fd939 // indirect
 	github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
@@ -48,7 +49,7 @@ require (
 	github.com/onsi/gomega v1.5.0 // indirect
 	github.com/opentracing-contrib/go-grpc v0.0.0-20180928155321-4b5a12d3ff02
 	github.com/opentracing-contrib/go-stdlib v0.0.0-20170113013457-1de4cc2120e7
-	github.com/opentracing/opentracing-go v1.0.2
+	github.com/opentracing/opentracing-go v1.1.0
 	github.com/philhofer/fwd v0.0.0-20160129035939-98c11a7a6ec8 // indirect
 	github.com/pkg/errors v0.8.1
 	github.com/prometheus/alertmanager v0.13.0
@@ -58,22 +59,27 @@ require (
 	github.com/prometheus/tsdb v0.9.1
 	github.com/segmentio/fasthash v0.0.0-20180216231524-a72b379d632e
 	github.com/sercand/kuberesolver v2.1.0+incompatible // indirect
+	github.com/sirupsen/logrus v1.4.2 // indirect
+	github.com/spaolacci/murmur3 v1.1.0 // indirect
 	github.com/stretchr/testify v1.3.0
 	github.com/tinylib/msgp v0.0.0-20161221055906-38a6f61a768d // indirect
 	github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 // indirect
 	github.com/uber-go/atomic v1.3.2 // indirect
-	github.com/uber/jaeger-client-go v2.14.0+incompatible
-	github.com/uber/jaeger-lib v1.5.0 // indirect
+	github.com/uber/jaeger-client-go v2.16.0+incompatible
+	github.com/uber/jaeger-lib v2.0.0+incompatible // indirect
 	github.com/weaveworks/billing-client v0.0.0-20171006123215-be0d55e547b1
 	github.com/weaveworks/common v0.0.0-20190714171817-ddeaa31513fd
 	github.com/weaveworks/mesh v0.0.0-20170131170447-5015f896ab62
 	github.com/weaveworks/promrus v1.2.0 // indirect
 	go.etcd.io/bbolt v1.3.3
 	go.etcd.io/etcd v0.0.0-20190709142735-eb7dd97135a5
 	go.uber.org/zap v1.10.0 // indirect
-	golang.org/x/net v0.0.0-20190403144856-b630fd6fe46b
+	golang.org/x/crypto v0.0.0-20190530122614-20be4c3c3ed5 // indirect
+	golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
+	golang.org/x/sys v0.0.0-20190602015325-4c4f7f33c9ed // indirect
+	golang.org/x/text v0.3.2 // indirect
 	golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
-	golang.org/x/tools v0.0.0-20190312170243-e65039ee4138
+	golang.org/x/tools v0.0.0-20190602112858-2de7f9bf822c
 	google.golang.org/api v0.4.0
 	google.golang.org/grpc v1.19.1
 	gopkg.in/fsnotify/fsnotify.v1 v1.4.7 // indirect