Merge remote-tracking branch 'upstream/master' into wal

Signed-off-by: Ganesh Vernekar <[email protected]>

Author: codesome

.circleci/config.yml

@@ -3,7 +3,7 @@ version: 2.1
 # https://circleci.com/blog/circleci-hacks-reuse-yaml-in-your-circleci-config-with-yaml/
 defaults: &defaults
   docker:
-    - image: cortexproject/build-image:master-d74af5958
+    - image: quay.io/cortexproject/build-image:update-lint-ae47f740a
   working_directory: /go/src/github.com/cortexproject/cortex
 
 workflows:
@@ -68,12 +68,19 @@ jobs:
     - run:
         name: Lint
         command: make BUILD_IN_CONTAINER=false lint
+    # fails to run everything first time - see https://github.com/golangci/golangci-lint/issues/866
+    - run:
+        name: Lint again
+        command: make BUILD_IN_CONTAINER=false lint
     - run:
         name: Check vendor directory is consistent.
         command: make BUILD_IN_CONTAINER=false mod-check
     - run:
         name: Check protos are consistent.
         command: make BUILD_IN_CONTAINER=false check-protos
+    - run:
+        name: Check generated documentation is consistent.
+        command: make BUILD_IN_CONTAINER=false check-doc
 
   test:
     <<: *defaults

.errcheck-exclude

@@ -0,0 +1,3 @@
+io/ioutil.WriteFile
+io/ioutil.ReadFile
+(github.com/go-kit/kit/log.Logger).Log

.golangci.yml

@@ -0,0 +1,8 @@
+output:
+  format: line-number
+
+linters-settings:
+  errcheck:
+    # path to a file containing a list of functions to exclude from checking
+    # see https://github.com/kisielk/errcheck#excluding-functions for details
+    exclude: ./.errcheck-exclude

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## master / unreleased
 
+Note that the ruler flags need to be changed in this upgrade. You're moving from a single node ruler to something that might need to be sharded.
+If you are running with a high `-ruler.num-workers` and if you're not able to execute all your rules in `-ruler.evaluation-interval`, then you'll need to shard.
+Further, if you're using the configs service, we've upgraded the migration library and this requires some manual intervention. See full
+instructions below to upgrade your Postgres.
+
+* [CHANGE] The frontend component now does not cache results if it finds a `Cache-Control` header and if one of its values is `no-store`. #1974
 * [CHANGE] Flags changed with transition to upstream Prometheus rules manager:
   * `ruler.client-timeout` is now `ruler.configs.client-timeout` in order to match `ruler.configs.url`
   * `ruler.group-timeout`has been removed
@@ -12,11 +18,44 @@
 * [CHANGE] Use relative links from /ring page to make it work when used behind reverse proxy. #1896
 * [CHANGE] Deprecated `-distributor.limiter-reload-period` flag. #1766
 * [CHANGE] Ingesters now write only normalised tokens to the ring, although they can still read denormalised tokens used by other ingesters. `-ingester.normalise-tokens` is now deprecated, and ignored. If you want to switch back to using denormalised tokens, you need to downgrade to Cortex 0.4.0. Previous versions don't handle claiming tokens from normalised ingesters correctly. #1809
+* [CHANGE] Overrides mechanism has been renamed to "runtime config", and is now separate from limits. Runtime config is simply a file that is reloaded by Cortex every couple of seconds. Limits and now also multi KV use this mechanism.<br />New arguments were introduced: `-runtime-config.file` (defaults to empty) and `-runtime-config.reload-period` (defaults to 10 seconds), which replace previously used `-limits.per-user-override-config` and `-limits.per-user-override-period` options. Old options are still used if `-runtime-config.file` is not specified. This change is also reflected in YAML configuration, where old `limits.per_tenant_override_config` and `limits.per_tenant_override_period` fields are replaced with `runtime_config.file` and `runtime_config.period` respectively. #1749
+* [CHANGE] Cortex now rejects data with duplicate labels. Previously, such data was accepted, with duplicate labels removed with only one value left. #1964
+* [CHANGE] Changed the default value for `-distributor.ha-tracker.prefix` from `collectors/` to `ha-tracker/` in order to not clash with other keys (ie. ring) stored in the same key-value store. #1940
 * [FEATURE] The distributor can now drop labels from samples (similar to the removal of the replica label for HA ingestion) per user via the `distributor.drop-label` flag. #1726
+* [FEATURE] Added flag `debug.mutex-profile-fraction` to enable mutex profiling #1969
 * [FEATURE] Added `global` ingestion rate limiter strategy. Deprecated `-distributor.limiter-reload-period` flag. #1766
 * [FEATURE] Added support for Microsoft Azure blob storage to be used for storing chunk data. #1913
+* [FEATURE] Added readiness probe endpoint`/ready` to queriers. #1934
+* [FEATURE] EXPERIMENTAL: Added `/series` API endpoint support with TSDB blocks storage. #1830
+* [FEATURE] Added "multi" KV store that can interact with two other KV stores, primary one for all reads and writes, and secondary one, which only receives writes. Primary/secondary store can be modified in runtime via runtime-config mechanism (previously "overrides"). #1749
+* [ENHANCEMENT] metric `cortex_ingester_flush_reasons` gets a new `reason` value: `Spread`, when `-ingester.spread-flushes` option is enabled. #1978
+* [ENHANCEMENT] Added `password` and `enable_tls` options to redis cache configuration. Enables usage of Microsoft Azure Cache for Redis service. #1923
+* [ENHANCEMENT] Experimental TSDB: Open existing TSDB on startup to prevent ingester from becoming ready before it can accept writes. #1917
+  * `--experimental.tsdb.max-tsdb-opening-concurrency-on-startup`
+* [ENHANCEMENT] Experimental TSDB: Added `cortex_ingester_shipper_dir_syncs_total`, `cortex_ingester_shipper_dir_sync_failures_total`, `cortex_ingester_shipper_uploads_total` and `cortex_ingester_shipper_upload_failures_total` metrics from TSDB shipper component. #1983
 * [BUGFIX] Fixed unnecessary CAS operations done by the HA tracker when the jitter is enabled. #1861
-  
+* [BUGFIX] Fixed #1904 ingesters getting stuck in a LEAVING state after coming up from an ungraceful exit. #1921
+* [BUGFIX] Reduce memory usage when ingester Push() errors. #1922
+* [BUGFIX] TSDB: Fixed handling of out of order/bound samples in ingesters with the experimental TSDB blocks storage. #1864
+* [BUGFIX] TSDB: Fixed querying ingesters in `LEAVING` state with the experimental TSDB blocks storage. #1854
+* [BUGFIX] TSDB: Fixed error handling in the series to chunks conversion with the experimental TSDB blocks storage. #1837
+* [BUGFIX] TSDB: Fixed TSDB creation conflict with blocks transfer in a `JOINING` ingester with the experimental TSDB blocks storage. #1818
+* [BUGFIX] TSDB: `experimental.tsdb.ship-interval` of <=0 treated as disabled instead of allowing panic. #1975
+* [BUGFIX] TSDB: Fixed `cortex_ingester_queried_samples` and `cortex_ingester_queried_series` metrics when using block storage. #1981
+* [BUGFIX] TSDB: Fixed `cortex_ingester_memory_series` and `cortex_ingester_memory_users` metrics when using with the experimental TSDB blocks storage. #1982
+
+### Upgrading Postgres (if you're using configs service)
+
+Reference: https://github.com/golang-migrate/migrate/tree/master/database/postgres#upgrading-from-v1
+
+1. Install the migrate package cli tool: https://github.com/golang-migrate/migrate/tree/master/cmd/migrate#installation
+2. Drop the `schema_migrations` table: `DROP TABLE schema_migrations;`.
+2. Run the migrate command:
+
+```bash
+migrate  -path <absolute_path_to_cortex>/cmd/cortex/migrations -database postgres://localhost:5432/database force 2 
+```
+
 ## 0.4.0 / 2019-12-02
 
 * [CHANGE] The frontend component has been refactored to be easier to re-use. When upgrading the frontend, cache entries will be discarded and re-created with the new protobuf schema. #1734
@@ -36,6 +75,7 @@
 * [FEATURE] EXPERIMENTAL: Use TSDB in the ingesters & flush blocks to S3/GCS ala Thanos. This will let us use an Object Store more efficiently and reduce costs. #1695
 * [FEATURE] Allow Query Frontend to log slow queries with `frontend.log-queries-longer-than`. #1744
 * [FEATURE] Add HTTP handler to trigger ingester flush & shutdown - used when running as a stateful set with the WAL enabled.  #1746
+* [FEATURE] EXPERIMENTAL: Added GCS support to TSDB blocks storage. #1772
 * [ENHANCEMENT] Reduce memory allocations in the write path. #1706
 * [ENHANCEMENT] Consul client now follows recommended practices for blocking queries wrt returned Index value. #1708
 * [ENHANCEMENT] Consul client can optionally rate-limit itself during Watch (used e.g. by ring watchers) and WatchPrefix (used by HA feature) operations. Rate limiting is disabled by default. New flags added: `--consul.watch-rate-limit`, and `--consul.watch-burst-size`. #1708
@@ -45,6 +85,7 @@
 * [BUGFIX] Fix bug where duplicate labels can be returned through metadata APIs. #1790
 * [BUGFIX] Fix reading of old, v3 chunk data. #1779
 * [BUGFIX] Now support IAM roles in service accounts in AWS EKS. #1803
+* [BUGFIX] Fixed duplicated series returned when querying both ingesters and store with the experimental TSDB blocks storage. #1778
 
 In this release we updated the following dependencies:
 - gRPC v1.25.0  (resulted in a drop of 30% CPU usage when compression is on)

GOVERNANCE.md

@@ -4,7 +4,7 @@ This document defines project governance for the project.
 
 ## Voting
 
-The Cortex project employs voting to ensure no single member can dominate the project. Any maintainer may cast a vote.
+The Cortex project employs voting to ensure no single member can dominate the project. Any maintainer may cast a vote. To avoid having a single company dominate the project, at most two votes from maintainers working for the same company will count.
 
 For formal votes, a specific statement of what is being voted on should be added to the relevant github issue or PR, and a link to that issue or PR added to the maintainers meeting agenda document.
 Maintainers should indicate their yes/no vote on that issue or PR, and after a suitable period of time, the votes will be tallied and the outcome noted.

MAINTAINERS

@@ -1,7 +1,8 @@
 Bryan Boreham, Weaveworks <[email protected]> (@bboreham)
-Chris Marchbanks, independent <[email protected]> (@csmarchbanks)
-Cody Boggs, independent <[email protected]> (@cboggs)
+Chris Marchbanks, Splunk <[email protected]> (@csmarchbanks)
+Cody Boggs, Splunk <[email protected]> (@cboggs)
 Jacob Lisi, Grafana Labs <[email protected]> (@jtlisi)
 Ken Haines, Microsoft <[email protected]> (@khaines)
+Marco Pracucci, Grafana Labs <[email protected]> (@pracucci)
 Tom Wilkie, Grafana Labs <[email protected]> (@tomwilkie)
 Goutham Veeramachaneni, Grafana Labs <[email protected]> (@gouthamve)

Makefile

@@ -80,27 +80,24 @@ GO_FLAGS := -ldflags "-extldflags \"-static\" -s -w" -tags netgo
 
 ifeq ($(BUILD_IN_CONTAINER),true)
 
+GOVOLUMES=	-v $(shell pwd)/.cache:/go/cache:delegated \
+			-v $(shell pwd)/.pkg:/go/pkg:delegated \
+			-v $(shell pwd):/go/src/github.com/cortexproject/cortex:delegated
+
 exes $(EXES) protos $(PROTO_GOS) lint test shell mod-check check-protos web-build web-pre web-deploy: build-image/$(UPTODATE)
 	@mkdir -p $(shell pwd)/.pkg
 	@mkdir -p $(shell pwd)/.cache
 	@echo
 	@echo ">>>> Entering build container: $@"
-	@$(SUDO) time docker run $(RM) $(TTY) -i \
-		-v $(shell pwd)/.cache:/go/cache \
-		-v $(shell pwd)/.pkg:/go/pkg \
-		-v $(shell pwd):/go/src/github.com/cortexproject/cortex \
-		$(BUILD_IMAGE) $@;
+	@$(SUDO) time docker run $(RM) $(TTY) -i $(GOVOLUMES) $(BUILD_IMAGE) $@;
 
 configs-integration-test: build-image/$(UPTODATE)
 	@mkdir -p $(shell pwd)/.pkg
 	@mkdir -p $(shell pwd)/.cache
 	@DB_CONTAINER="$$(docker run -d -e 'POSTGRES_DB=configs_test' postgres:9.6)"; \
 	echo ; \
 	echo ">>>> Entering build container: $@"; \
-	$(SUDO) docker run $(RM) $(TTY) -i \
-		-v $(shell pwd)/.cache:/go/cache \
-		-v $(shell pwd)/.pkg:/go/pkg \
-		-v $(shell pwd):/go/src/github.com/cortexproject/cortex \
+	$(SUDO) docker run $(RM) $(TTY) -i $(GOVOLUMES) \
 		-v $(shell pwd)/cmd/cortex/migrations:/migrations \
 		--workdir /go/src/github.com/cortexproject/cortex \
 		--link "$$DB_CONTAINER":configs-db.cortex.local \
@@ -123,11 +120,8 @@ protos: $(PROTO_GOS)
 	protoc -I $(GOPATH)/src:./vendor:./$(@D) --gogoslick_out=plugins=grpc,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,:./$(@D) ./$(patsubst %.pb.go,%.proto,$@)
 
 lint:
-	./tools/lint -notestpackage -novet -ignorespelling queriers -ignorespelling Queriers .
-
-	# -stdmethods=false disables checks for non-standard signatures for methods with familiar names.
-	# This is needed because the Prometheus storage interface requires a non-standard Seek() method.
-	go vet -stdmethods=false ./pkg/...
+	misspell -error docs
+	golangci-lint run --new-from-rev ed7c302fd968 --build-tags netgo --timeout=5m --enable golint --enable misspell --enable gofmt
 
 test:
 	./tools/test -netgo
@@ -194,5 +188,16 @@ prime-minikube: save-images
 		fi \
 	done
 
+# Generates the config file documentation.
+doc:
+	cp ./docs/configuration/config-file-reference.template ./docs/configuration/config-file-reference.md
+	go run ./tools/doc-generator/ >> ./docs/configuration/config-file-reference.md
+
+clean-doc:
+	rm -f ./docs/configuration/config-file-reference.md
+
+check-doc: clean-doc doc
+	@git diff --exit-code -- ./docs/configuration/config-file-reference.md
+
 web-serve:
 	cd website && hugo --config config.toml -v server

README.md

@@ -25,8 +25,9 @@ Read the [getting started guide](docs/getting_started.md) if you're new to the
 project. Before deploying Cortex with a permanent storage backend you
 should read:
 1. [An overview of Cortex's architecture](docs/architecture.md)
-1. [A general guide to running Cortex](docs/running.md)
-1. [Information regarding configuring Cortex](docs/arguments.md)
+1. [A general guide to running Cortex](docs/guides/running.md)
+1. [Information regarding configuring Cortex](docs/configuration/arguments.md)
+1. [Steps to run Cortex with Cassandra](docs/guides/cortex-with-cassandra.md)
 
 For a guide to contributing to Cortex, see the [contributor guidelines](CONTRIBUTING.md).
 

RELEASE.md

@@ -12,7 +12,8 @@ Our goal is to provide a new minor release every 4 weeks. This is a new process
 | v0.2.0         | 2019-08-28                                 | Goutham Veeramachaneni (Github: @gouthamve) |
 | v0.3.0         | 2019-10-09                                 | Bryan Boreham (@bboreham)                   |
 | v0.4.0         | 2019-11-13                                 | Tom Wilkie (@tomwilkie)                     |
-| v0.5.0         | 2019-12-11                                 | **searching for volunteer**                 |
+| v0.5.0         | 2020-01-08                                 | _Abandoned_                                   |
+| v0.6.0         | 2020-01-20                                 | **searching for a volunteer**                   |
 
 ## Release shepherd responsibilities
 

build-image/Dockerfile

@@ -4,9 +4,6 @@ RUN apt-get update && apt-get install -y curl python-requests python-yaml file j
 RUN curl -sL https://deb.nodesource.com/setup_6.x | sh -
 RUN apt-get install -y nodejs npm && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 RUN npm install -g postcss-cli autoprefixer
-RUN go clean -i net && \
-	go install -tags netgo std && \
-	go install -race -tags netgo std
 ENV HUGO_VERSION=v0.59.1
 RUN git clone https://github.com/gohugoio/hugo.git --branch ${HUGO_VERSION} --depth 1 && \
 	cd hugo  && go install --tags extended && cd ../ && \
@@ -15,17 +12,13 @@ RUN curl -fsSLo shfmt https://github.com/mvdan/sh/releases/download/v1.3.0/shfmt
 	echo "b1925c2c405458811f0c227266402cf1868b4de529f114722c2e3a5af4ac7bb2  shfmt" | sha256sum -c && \
 	chmod +x shfmt && \
 	mv shfmt /usr/bin
+RUN curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s -- -b /usr/bin v1.21.0
 RUN GO111MODULE=on go get -tags netgo \
-		github.com/fzipp/gocyclo \
-		golang.org/x/lint/golint \
-		github.com/kisielk/[email protected] \
 		github.com/client9/misspell/cmd/[email protected] \
 		github.com/golang/protobuf/[email protected] \
 		github.com/gogo/protobuf/[email protected] \
 		github.com/gogo/protobuf/[email protected] && \
 	rm -rf /go/pkg /go/src
-RUN curl -Ls https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 -o $GOPATH/bin/dep && \
-  chmod +x $GOPATH/bin/dep
 
 ENV NODE_PATH=/usr/lib/node_modules
 COPY build.sh /

cmd/cortex/main.go

@@ -27,14 +27,20 @@ func init() {
 
 func main() {
 	var (
-		cfg             cortex.Config
-		configFile      = ""
-		eventSampleRate int
-		ballastBytes    int
+		cfg                  cortex.Config
+		configFile           = ""
+		eventSampleRate      int
+		ballastBytes         int
+		mutexProfileFraction int
 	)
 	flag.StringVar(&configFile, "config.file", "", "Configuration file to load.")
 	flag.IntVar(&eventSampleRate, "event.sample-rate", 0, "How often to sample observability events (0 = never).")
 	flag.IntVar(&ballastBytes, "mem-ballast-size-bytes", 0, "Size of memory ballast to allocate.")
+	flag.IntVar(&mutexProfileFraction, "debug.mutex-profile-fraction", 0, "Fraction at which mutex profile vents will be reported, 0 to disable")
+
+	if mutexProfileFraction > 0 {
+		runtime.SetMutexProfileFraction(mutexProfileFraction)
+	}
 
 	flagext.RegisterFlags(&cfg)
 	flag.Parse()

cmd/cortex/migrations/002_immutable_configs.up.sql

@@ -2,6 +2,10 @@
 -- process (as exemplified in users/db/migrations/006 to 010), but currently
 -- there are no data in production and only one row in dev.
 
+-- https://github.com/mattes/migrate/tree/master/database/postgres#upgrading-from-v1
+-- Wrap all commands in BEGIN and COMMIT to accommodate upgrade
+BEGIN;
+
 -- The existing id, type columns are the id & type of the entity that owns the
 -- config.
 ALTER TABLE configs RENAME COLUMN id TO owner_id;
@@ -12,3 +16,5 @@ ALTER TABLE configs ADD COLUMN id SERIAL;
 
 ALTER TABLE configs DROP CONSTRAINT configs_pkey;
 ALTER TABLE configs ADD PRIMARY KEY (id, owner_id, owner_type, subsystem);
+
+COMMIT;

docs/_index.md

@@ -24,7 +24,7 @@ Prometheus sources in a single cluster, allowing untrusted parties to share the
 - **Long term storage:** Cortex supports Amazon DynamoDB, Google Bigtable, Cassandra, S3 and GCS for long term storage of metric data.  This allows you to durably store data for longer than the lifetime of any single machine, and use this data for long term capacity planning.
 
 Cortex is a [CNCF](https://cncf.io) sandbox project used in several production systems including [Weave Cloud](https://cloud.weave.works) and [Grafana Cloud](https://grafana.com/cloud).
-Cortex is primarily used used as a [remote write](https://prometheus.io/docs/operating/configuration/#remote_write) destination for Prometheus, exposing a Prometheus-compatible query API.
+Cortex is primarily used as a [remote write](https://prometheus.io/docs/operating/configuration/#remote_write) destination for Prometheus, exposing a Prometheus-compatible query API.
 
 ## Documentation