Merge remote-tracking branch 'upstream/master' into tokens-file

Signed-off-by: Ganesh Vernekar <[email protected]>

Author: Ganesh Vernekar

.github/pull_request_template.md

@@ -0,0 +1,15 @@
+<!--  Thanks for sending a pull request!  Before submitting:
+
+1. Read our CONTRIBUTING.md guide
+2. Rebase your PR if it gets out of sync with master
+-->
+
+**What this PR does**:
+
+**Which issue(s) this PR fixes**:
+Fixes #<issue number>
+
+**Checklist**
+- [ ] Tests updated
+- [ ] Documentation added
+- [ ] `CHANGELOG.md` updated

.lintignore

@@ -2,3 +2,4 @@
 ./tools*
 ./vendor*
 ./pkg/configs/legacy_promql*
+./.pkg*

ADOPTERS.md

@@ -3,6 +3,7 @@
 This is the list of organisations that are using Cortex in **production environments** to power their metrics and monitoring systems. Please send PRs to add or remove organisations.
 
 * [Aspen Mesh](https://aspenmesh.io/)
+* [DigitalOcean](https://www.digitalocean.com/)
 * [Electronic Arts](https://www.ea.com/)
 * [GoJek](https://www.gojek.io/)
 * [GrafanaLabs](https://grafana.com/)

CHANGELOG.md

@@ -1,6 +1,15 @@
+# Changelog
+
 ## master / unreleased
 
+* [CHANGE] The frontend component has been refactored to be easier to re-use. When upgrading the frontend, cache entries will be discarded and re-created with the new protobuf schema. #1734
+* [CHANGE] Remove direct DB/API access from the ruler
 * [CHANGE] Removed `Delta` encoding. Any old chunks with `Delta` encoding cannot be read anymore. If `ingester.chunk-encoding` is set to `Delta` the ingester will fail to start. #1706
+* [FEATURE] Global limit on the max series per user and metric #1760
+  * `-ingester.max-global-series-per-user`
+  * `-ingester.max-global-series-per-metric`
+* [FEATURE] Flush chunks with stale markers early with `ingester.max-stale-chunk-idle`. #1759
+* [FEATURE] EXPERIMENTAL: Added new KV Store backend based on memberlist library. Components can gossip about tokens and ingester states, instead of using Consul or Etcd. #1721
 * [ENHANCEMENT] Allocation improvements in adding samples to Chunk. #1706
 * [ENHANCEMENT] Consul client now follows recommended practices for blocking queries wrt returned Index value. #1708
 * [ENHANCEMENT] Consul client can optionally rate-limit itself during Watch (used e.g. by ring watchers) and WatchPrefix (used by HA feature) operations. Rate limiting is disabled by default. New flags added: `--consul.watch-rate-limit`, and `--consul.watch-burst-size`. #1708

CONTRIBUTING.md

@@ -6,6 +6,15 @@ Welcome! We're excited that you're interested in contributing. Below are some ba
 
 Cortex follows a standard GitHub pull request workflow. If you're unfamiliar with this workflow, read the very helpful [Understanding the GitHub flow](https://guides.github.com/introduction/flow/) guide from GitHub.
 
+You are welcome to create draft PRs at any stage of readiness - this
+can be helpful to ask for assistance or to develop an idea. But before
+a piece of work is finished it should:
+
+* Be organised into one or more commits, each of which has a commit message that describes all changes made in that commit ('why' more than 'what' - we can read the diffs to see the code that changed).
+* Each commit should build towards the whole - don't leave in back-tracks and mistakes that you later corrected.
+* Have tests for new functionality or tests that would have caught the bug being fixed.
+* Include a CHANGELOG message if users of Cortex need to hear about what you did.
+
 ## Developer Certificates of Origin (DCOs)
 
 Before submitting your work in a pull request, make sure that *all* commits are signed off with a **Developer Certificate of Origin** (DCO). Here's an example:

Makefile

@@ -74,14 +74,6 @@ RM := --rm
 # in any custom cloudbuild.yaml files
 TTY := --tty
 GO_FLAGS := -ldflags "-extldflags \"-static\" -s -w" -tags netgo
-NETGO_CHECK = @strings $@ | grep cgo_stub\\\.go >/dev/null || { \
-       rm $@; \
-       echo "\nYour go standard library was built without the 'netgo' build tag."; \
-       echo "To fix that, run"; \
-       echo "    sudo go clean -i net"; \
-       echo "    sudo go install -tags netgo std"; \
-       false; \
-}
 
 ifeq ($(BUILD_IN_CONTAINER),true)
 
@@ -121,12 +113,11 @@ exes: $(EXES)
 
 $(EXES):
 	CGO_ENABLED=0 go build $(GO_FLAGS) -o $@ ./$(@D)
-	$(NETGO_CHECK)
 
 protos: $(PROTO_GOS)
 
 %.pb.go:
-	protoc -I $(GOPATH)/src:./vendor:./$(@D) --gogoslick_out=plugins=grpc:./$(@D) ./$(patsubst %.pb.go,%.proto,$@)
+	protoc -I $(GOPATH)/src:./vendor:./$(@D) --gogoslick_out=plugins=grpc,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,:./$(@D) ./$(patsubst %.pb.go,%.proto,$@)
 
 lint:
 	./tools/lint -notestpackage -novet -ignorespelling queriers -ignorespelling Queriers .

docs/architecture.md

@@ -145,4 +145,4 @@ The interface works somewhat differently across the supported databases:
 
 A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Cortex has evolved, mainly in an attempt to better load balance writes and improve query performance.
 
-> The current schema recommendation is the **v10 schema**.
+> The current schema recommendation is the **v10 schema**. v11 schema is an experimental schema.

docs/arguments.md

@@ -142,6 +142,39 @@ prefix these flags with `distributor.ha-tracker.`
 - `etcd.max-retries`
    The maximum number of retries to do for failed ops.
 
+#### memberlist (EXPERIMENTAL)
+
+Flags for configuring KV store based on memberlist library. This feature is experimental, please don't use it yet.
+
+- `memberlist.nodename`
+   Name of the node in memberlist cluster. Defaults to hostname.
+- `memberlist.retransmit-factor`
+   Multiplication factor used when sending out messages (factor * log(N+1)). If not set, default value is used.
+- `memberlist.join`
+   Other cluster members to join. Can be specified multiple times.
+- `memberlist.abort-if-join-fails`
+   If this node fails to join memberlist cluster, abort.
+- `memberlist.left-ingesters-timeout`
+   How long to keep LEFT ingesters in the ring. Note: this is only used for gossiping, LEFT ingesters are otherwise invisible.
+- `memberlist.leave-timeout`
+   Timeout for leaving memberlist cluster.
+- `memberlist.gossip-interval`
+   How often to gossip with other cluster members. Uses memberlist LAN defaults if 0.
+- `memberlist.gossip-nodes`
+   How many nodes to gossip with in each gossip interval. Uses memberlist LAN defaults if 0.
+- `memberlist.pullpush-interval`
+   How often to use pull/push sync. Uses memberlist LAN defaults if 0.
+- `memberlist.bind-addr`
+   IP address to listen on for gossip messages. Multiple addresses may be specified. Defaults to 0.0.0.0.
+- `memberlist.bind-port`
+   Port to listen on for gossip messages. Defaults to 7946.
+- `memberlist.packet-dial-timeout`
+   Timeout used when connecting to other nodes to send packet.
+- `memberlist.packet-write-timeout`
+   Timeout for writing 'packet' data.
+- `memberlist.transport-debug`
+   Log debug transport messages. Note: global log.level must be at debug level as well.
+
 ### HA Tracker
 
 HA tracking has two of it's own flags:
@@ -175,6 +208,14 @@ It also talks to a KVStore and has it's own copies of the same flags used by the
 
   The maximum duration of a timeseries chunk in memory. If a timeseries runs for longer than this the current chunk will be flushed to the store and a new chunk created. (default 12h)
 
+- `-ingester.max-chunk-idle`
+  
+  If a series doesn't receive a sample for this duration, it is flushed and removed from memory.
+
+- `-ingester.max-stale-chunk-idle`
+  
+  If a series receives a [staleness marker](https://www.robustperception.io/staleness-and-promql), then we wait for this duration to get another sample before we close and flush this series, removing it from memory. You want it to be at least 2x the scrape interval as you don't want a single failed scrape to cause a chunk flush. 
+
 - `-ingester.chunk-age-jitter`
 
   To reduce load on the database exactly 12 hours after starting, the age limit is reduced by a varying amount up to this. (default 20m)

docs/ha-pair-handling.md

@@ -0,0 +1,41 @@
+# Config for sending HA Pairs data to Cortex
+
+## Context
+
+You can have more than a single Prometheus monitoring and ingesting the same metrics for redundancy. Cortex already does replication for redundancy and it doesn't make sense to ingest the same data twice. So in Cortex, we made sure we can dedupe the data we receive from HA Pairs of Prometheus. We do this via the following:
+
+Assume that there are two teams, each running their own Prometheus, monitoring different services. Let's call the Prometheis T1 and T2. Now, if the teams are running HA pairs, let's call the individual Prometheis, T1.a, T1.b and T2.a and T2.b.
+
+In Cortex we make sure we only ingest from one of T1.a and T1.b, and only from one of T2.a and T2.b. We do this by electing a leader replica for each cluster of Prometheus. For example, in the case of T1, let it be T1.a. As long as T1.a is the leader, we drop the samples sent by T1.b. And if Cortex sees no new samples from T1.a for a short period (30s by default), it'll switch the leader to be T1.b.
+
+This means if T1.a goes down for a few minutes Cortex's HA sample handling will have switched and elected T1.b as the leader. This failover timeout is what enables us to only accept samples from a single replica at a time, but ensure we don't drop too much data in case of issues. Note that with the default scrape period of 15s, and the default timeouts in Cortex, in most cases you'll only lose a single scrape of data in the case of a leader election failover. For any rate queries the rate window should be at least 4x the scrape period to account for any of these failover scenarios, for example with the default scrape period of 15s then you should calculate rates over at least 1m periods.
+
+Now we do the same leader election process T2.
+
+## Config
+
+### Client Side
+
+So for Cortex to achieve this, we need 2 identifiers for each process, one identifier for the cluster (T1 or T2, etc) and one identifier to identify the replica in the cluster (a or b). The easiest way to do with is by setting external labels, ideally `cluster` and `replica` (note the default is `__replica__`). For example:
+
+```
+cluster: prom-team1
+replica: replica1 (or pod-name)
+```
+
+and
+
+```
+cluster: prom-team1
+replica: replica2
+```
+
+Note: These are external labels and have nothing to do with remote_write config.
+
+These two label names are configurable per-tenant within Cortex, and should be set to something sensible. For example, cluster label is already used by some workloads, and you should set the label to be something else but uniquely identifies the cluster. Good examples for this label-name would be `team`, `cluster`, `prometheus`, etc.
+
+The replica label should be set so that the value for each prometheus is unique in that cluster. Note: Cortex drops this label when ingesting data, but preserves the cluster label. This way, your timeseries won't change when replicas change.
+
+### Server Side
+
+To enable handling of samples, see the [distibutor flags](./arguments.md#ha-tracker) having `ha-tracker` in them.

docs/single-process-config.yaml

@@ -51,7 +51,7 @@ ingester:
 # for the chunks.
 schema:
   configs:
-  - from: 2019-03-25
+  - from: 2019-07-29
     store: boltdb
     object_store: filesystem
     schema: v10
@@ -65,4 +65,3 @@ storage:
 
   filesystem:
     directory: /tmp/cortex/chunks
-