From d6bcc6666c77ec49bfb11fe3d3136e35161d4735 Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 20:59:38 +0100 Subject: [PATCH 1/7] Update README with more description and links. Signed-off-by: Tom Wilkie --- CONTRIBUTING.md | 67 +++++++++++++++++++ README.md | 166 ++++++++++++++++++------------------------------ 2 files changed, 129 insertions(+), 104 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ea93f56f30..72dcb9654c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,3 +15,70 @@ git commit -s -m "Here is my signed commit" ``` You can find further instructions [here](https://github.com/probot/dco#how-it-works). + +## Building Cortex + +To build: +``` +make +``` + +(By default, the build runs in a Docker container, using an image built +with all the tools required. The source code is mounted from where you +run `make` into the build container as a Docker volume.) + +To run the test suite: +``` +make test +``` + +## Playing in `minikube` + +First, start `minikube`. + +You may need to load the Docker images into your minikube environment. There is +a convenience rule in the Makefile to do this: + +``` +make prime-minikube +``` + +Then run Cortex in minikube: +``` +kubectl apply -f ./k8s +``` + +(these manifests use `latest` tags, i.e. this will work if you have +just built the images and they are available on the node(s) in your +Kubernetes cluster) + +Cortex will sit behind an nginx instance exposed on port 30080. A job is deployed to scrape itself. Try it: + +http://192.168.99.100:30080/api/prom/api/v1/query?query=up + +If that doesn't work, your Minikube might be using a different ip address. Check with `minikube status`. + +### Dependency management + +We uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. +This requires a working Go environment with version 1.11 or greater, git and [bzr](http://wiki.bazaar.canonical.com/Download) installed. + +To add or update a new dependency, use the `go get` command: + +```bash +# Pick the latest tagged release. +go get example.com/some/module/pkg + +# Pick a specific version. +go get example.com/some/module/pkg@vX.Y.Z +``` + +Tidy up the `go.mod` and `go.sum` files: + +```bash +go mod tidy +git add go.mod go.sum +git commit +``` + +You have to commit the changes to `go.mod` and `go.sum` before submitting the pull request. diff --git a/README.md b/README.md index bf5d96ffe4..a1538ecba3 100644 --- a/README.md +++ b/README.md @@ -1,123 +1,60 @@

Cortex Logo

-# Open source, horizontally scalable, multi-tenant Prometheus as a service - [![Circle CI](https://circleci.com/gh/cortexproject/cortex/tree/master.svg?style=shield)](https://circleci.com/gh/cortexproject/cortex/tree/master) [![GoDoc](https://godoc.org/github.com/cortexproject/cortex?status.svg)](https://godoc.org/github.com/cortexproject/cortex) +Go Report Card +Slack -Cortex is a [CNCF](https://cncf.io) sandbox project used in several production -systems including [Weave Cloud](https://cloud.weave.works), [GrafanaCloud](https://grafana.com/cloud) -and [FreshTracks.io](https://www.freshtracks.io/). - -Cortex provides horizontally scalable, multi-tenant, long term storage for -[Prometheus](https://prometheus.io) metrics when used as a [remote -write](https://prometheus.io/docs/operating/configuration/#remote_write) -destination, and a horizontally scalable, Prometheus-compatible query -API. - -Multi-tenant means it handles metrics from multiple independent -Prometheus sources, keeping them separate. - -[Instrumenting Your App: Best Practices](https://www.weave.works/docs/cloud/latest/tasks/monitor/best-instrumenting/) +# Cortex: horizontally scalable, highly available, multi-tenant, long term storage for Prometheus. -## Hosted Cortex (Prometheus as a service) +Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for +[Prometheus](https://prometheus.io). -There are several commercial services where you can use Cortex -on-demand: - -### Weave Cloud - -[Weave Cloud](https://cloud.weave.works) from -[Weaveworks](https://weave.works) lets you deploy, manage, and monitor -container-based applications. Sign up at https://cloud.weave.works -and follow the instructions there. Additional help can also be found -in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/overview/). +- **Horizontally scalable:** Cortex can run across multiple machines in a cluster, exceeding the throughput and storage of a single machine. This enables you to send the metrics from multiple Prometheus servers to a single Cortex cluster and run "globally aggregated" queries across all data in a single place. +- **Highly available:** When run in a cluster, Cortex can replicate data between machines. This allows you to survive machine failure without gaps in your graphs. +- **Multi-tenant:** Cortex can isolate data and queries from multiple different independent +Prometheus sources in a single cluster, allowing untrusted parties to share the same cluster. +- **Long term storage:** Cortex supports Amazon DynamoDB, Google Bigtable, Cassandra, S3 and GCS for long term storage of metric data. This allows you to durably store data for longer than the lifetime of any single machine, and use this data for long term capacity planning. -### GrafanaCloud +Cortex is a [CNCF](https://cncf.io) sandbox project used in several production systems including [Weave Cloud](https://cloud.weave.works) and [GrafanaCloud](https://grafana.com/cloud). +Cortex is a primarily used as a [remote write](https://prometheus.io/docs/operating/configuration/#remote_write) destination for Prometheus, with a Prometheus-compatible query API. -To use Cortex as part of Grafana Cloud, sign up for [GrafanaCloud](https://grafana.com/cloud) -by clicking "Log In" in the top right and then "Sign Up Now". Cortex is included -as part of the Starter and Basic Hosted Grafana plans. +## Documentaion -## Contributing +Read the [getting started guide](docs/getting_started.md) if you're new to the project. For a guide to contributing to Cortex, see the [contributor guidelines](CONTRIBUTING.md). -## For Developers - -To build: -``` -make -``` - -(By default, the build runs in a Docker container, using an image built -with all the tools required. The source code is mounted from where you -run `make` into the build container as a Docker volume.) - -To run the test suite: -``` -make test -``` - -## Playing in `minikube` - -First, start `minikube`. - -You may need to load the Docker images into your minikube environment. There is -a convenience rule in the Makefile to do this: - -``` -make prime-minikube -``` - -Then run Cortex in minikube: -``` -kubectl apply -f ./k8s -``` - -(these manifests use `latest` tags, i.e. this will work if you have -just built the images and they are available on the node(s) in your -Kubernetes cluster) - -Cortex will sit behind an nginx instance exposed on port 30080. A job is deployed to scrape itself. Try it: - -http://192.168.99.100:30080/api/prom/api/v1/query?query=up - -If that doesn't work, your Minikube might be using a different ip address. Check with `minikube status`. - -### Dependency management - -We uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. -This requires a working Go environment with version 1.11 or greater, git and [bzr](http://wiki.bazaar.canonical.com/Download) installed. - -To add or update a new dependency, use the `go get` command: - -```bash -# Pick the latest tagged release. -go get example.com/some/module/pkg - -# Pick a specific version. -go get example.com/some/module/pkg@vX.Y.Z -``` - -Tidy up the `go.mod` and `go.sum` files: - -```bash -go mod tidy -git add go.mod go.sum -git commit -``` - -You have to commit the changes to `go.mod` and `go.sum` before submitting the pull request. +You can also read [an overview of Cortex's architecture](docs/architecture.md) and [information about configuring Cortex](docs/arguments.md). ## Further reading -To learn more about Cortex, consult the following documents/talks: - -- [Original design document for Project Frankenstein](http://goo.gl/prdUYV) -- PromCon 2016 Talk: "Project Frankenstein: Multitenant, Scale-Out Prometheus": [video](https://youtu.be/3Tb4Wc0kfCM), [slides](http://www.slideshare.net/weaveworks/project-frankenstein-a-multitenant-horizontally-scalable-prometheus-as-a-service) -- KubeCon Prometheus Day talk "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" [slides](http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service) -- CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" [slides](https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6) +To learn more about Cortex, consult the following documents & talks: + +- Dev 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" [video][kubecon-2018-video], [slide][kubecon-2018-slides] +- Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" +- Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" [slides][cloudnative-london-2018-slides] +- Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" [slides][cncf-toc-presentation] +- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" [videos][promcon-2017-video], [slides][promcon-2017-slides] +- June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" [video][prometheus-london-2017-video] +- Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" [video][kubecon-2016-video], [slides][kubecon-2016-slides] +- Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": [video][promcon-2016-video], [slides][promcon-2016-slides] +- [The original design document for Project Frankenstein](http://goo.gl/prdUYV) + +[kubecon-2018-talk]: https://kccna18.sched.com/event/GrXL/cortex-infinitely-scalable-prometheus-bryan-boreham-weaveworks +[kubecon-2018-video]: https://www.youtube.com/watch?v=iyN40FsRQEo +[kubecon-2018-slides]: https://static.sched.com/hosted_files/kccna18/9b/Cortex%20CloudNativeCon%202018.pdf +[cloudnative-london-2018-slides]: https://www.slideshare.net/grafana/cortex-horizontally-scalable-highly-available-prometheus +[cncf-2018-blog]: https://www.cncf.io/blog/2018/12/18/cortex-a-multi-tenant-horizontally-scalable-prometheus-as-a-service/ +[cncf-toc-presentation]: https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6 +[prometheus-london-2017-video]: https://www.youtube.com/watch?v=Xi4jq2IUbLs +[promcon-2017-talk]: https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/ +[promcon-2017-video]: https://www.youtube.com/watch?v=_8DmPW4iQBQ +[promcon-2017-slides]: https://promcon.io/2017-munich/slides/cortex-prometheus-as-a-service-one-year-on.pdf +[kubecon-2016-video]: https://www.youtube.com/watch?v=9Uctgnazfwk +[kubecon-2016-slides]: http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service +[promcon-2016-video]: https://youtu.be/3Tb4Wc0kfCM +[promcon-2016-slides]: http://www.slideshare.net/weaveworks/project-frankenstein-a-multitenant-horizontally-scalable-prometheus-as-a-service ## Getting Help @@ -128,3 +65,24 @@ If you have any questions about Cortex: - Send an email to cortex-users@lists.cncf.io Your feedback is always welcome. + +## Hosted Cortex (Prometheus as a service) + +There are several commercial services where you can use Cortex +on-demand: + +### Weave Cloud + +[Weave Cloud](https://cloud.weave.works) from +[Weaveworks](https://weave.works) lets you deploy, manage, and monitor +container-based applications. Sign up at https://cloud.weave.works +and follow the instructions there. Additional help can also be found +in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/overview/). + +[Instrumenting Your App: Best Practices](https://www.weave.works/docs/cloud/latest/tasks/monitor/best-instrumenting/) + +### GrafanaCloud + +To use Cortex as part of Grafana Cloud, sign up for [GrafanaCloud](https://grafana.com/cloud) +by clicking "Log In" in the top right and then "Sign Up Now". Cortex is included +as part of the Starter and Basic Hosted Grafana plans. \ No newline at end of file From 3535fb1097452baa9208fb72f105906e2a7db6df Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 21:08:50 +0100 Subject: [PATCH 2/7] Consolidate further reading into the README. Signed-off-by: Tom Wilkie --- README.md | 18 ++++++++++-------- docs/architecture.md | 14 +------------- 2 files changed, 11 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index a1538ecba3..6fbc35946b 100644 --- a/README.md +++ b/README.md @@ -31,15 +31,16 @@ You can also read [an overview of Cortex's architecture](docs/architecture.md) a To learn more about Cortex, consult the following documents & talks: -- Dev 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" [video][kubecon-2018-video], [slide][kubecon-2018-slides] +- Dev 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" [video][kubecon-2018-video], ([slide][kubecon-2018-slides]) - Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" -- Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" [slides][cloudnative-london-2018-slides] -- Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" [slides][cncf-toc-presentation] -- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" [videos][promcon-2017-video], [slides][promcon-2017-slides] -- June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" [video][prometheus-london-2017-video] -- Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" [video][kubecon-2016-video], [slides][kubecon-2016-slides] -- Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": [video][promcon-2016-video], [slides][promcon-2016-slides] -- [The original design document for Project Frankenstein](http://goo.gl/prdUYV) +- Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides]) +- Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation]) +- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides]) +- July 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" +- June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video]) +- Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" ([video][kubecon-2016-video], [slides][kubecon-2016-slides]) +- Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": ([video][promcon-2016-video], [slides][promcon-2016-slides]) +- June 2018 design document; "[Project Frankenstein: A Multi Tenant, Scale Out Prometheus](http://goo.gl/prdUYV)" [kubecon-2018-talk]: https://kccna18.sched.com/event/GrXL/cortex-infinitely-scalable-prometheus-bryan-boreham-weaveworks [kubecon-2018-video]: https://www.youtube.com/watch?v=iyN40FsRQEo @@ -51,6 +52,7 @@ To learn more about Cortex, consult the following documents & talks: [promcon-2017-talk]: https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/ [promcon-2017-video]: https://www.youtube.com/watch?v=_8DmPW4iQBQ [promcon-2017-slides]: https://promcon.io/2017-munich/slides/cortex-prometheus-as-a-service-one-year-on.pdf +[cortex-query-optimisation-2017]: https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY [kubecon-2016-video]: https://www.youtube.com/watch?v=9Uctgnazfwk [kubecon-2016-slides]: http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service [promcon-2016-video]: https://youtu.be/3Tb4Wc0kfCM diff --git a/docs/architecture.md b/docs/architecture.md index cd213dfa0c..f2539ab406 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -128,16 +128,4 @@ The interface works somewhat differently across the supported databases: A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Cortex has evolved, mainly in an attempt to better load balance writes and improve query performance. -> The current schema recommendation is the **v6 schema**. - -## See also - -For more details on Cortex's architecture, we recommend reading/watching: - -- "[Project Frankenstein: A multi tenant, scale out Prometheus](https://docs.google.com/document/d/1C7yhMnb1x2sfeoe45f4mnnKConvroWhJ8KQZwIHJOuw/edit#heading=h.nimsq29kl184)" (the original design doc) -- "[Multitenant, Scale-Out Prometheus](https://promcon.io/2016-berlin/talks/multitenant-scale-out-prometheus/)" (PromCon 2016 talk) -- "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" (KubeCon Prometheus Day talk) [slides](http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service) [video](https://www.youtube.com/watch?v=9Uctgnazfwk) -- "[Cortex: Prometheus as a Service, One Year On](https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/)" (PromCon 2017 Talk) -- "Horizontally Scalable, Multi-tenant Prometheus" [slides](https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6) (CNCF TOC Presentation) -- [Cortex Query Woes](https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY) provides details on Cortex query optimizations. -- [Cortex: Horizontally Scalable, Highly Available Prometheus Monitoring](https://speakerdeck.com/grafana/cortex-horizontally-scalable-highly-available-prometheus-monitoring) (Presentation at Cloud Native London Meetup, Nov 2018) +> The current schema recommendation is the **v6 schema**. \ No newline at end of file From 97968f0956cfc32df48516fc646963f48c7ecf4c Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 21:13:00 +0100 Subject: [PATCH 3/7] Add PromCon 2018 panel. Signed-off-by: Tom Wilkie --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 6fbc35946b..1dad658a5c 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,7 @@ To learn more about Cortex, consult the following documents & talks: - Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" - Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides]) - Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation]) +- Aug 2018 PromCon panel; "[Prometheus Long-Term Storage Approaches][promcon-2018-panel]" ([video][promcon-2018-video]) - Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides]) - July 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" - June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video]) @@ -48,6 +49,8 @@ To learn more about Cortex, consult the following documents & talks: [cloudnative-london-2018-slides]: https://www.slideshare.net/grafana/cortex-horizontally-scalable-highly-available-prometheus [cncf-2018-blog]: https://www.cncf.io/blog/2018/12/18/cortex-a-multi-tenant-horizontally-scalable-prometheus-as-a-service/ [cncf-toc-presentation]: https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6 +[promcon-2018-panel]: https://promcon.io/2018-munich/talks/panel-discussion-prometheus-long-term-storage-approaches/ +[promcon-2018-video]: https://www.youtube.com/watch?v=3pTG_N8yGSU [prometheus-london-2017-video]: https://www.youtube.com/watch?v=Xi4jq2IUbLs [promcon-2017-talk]: https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/ [promcon-2017-video]: https://www.youtube.com/watch?v=_8DmPW4iQBQ From 84b81e652dd040d4f54f0e5bc066c4147021a6a0 Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 21:21:46 +0100 Subject: [PATCH 4/7] Add various blog posts. Signed-off-by: Tom Wilkie --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1dad658a5c..7e4dcc847a 100644 --- a/README.md +++ b/README.md @@ -31,18 +31,23 @@ You can also read [an overview of Cortex's architecture](docs/architecture.md) a To learn more about Cortex, consult the following documents & talks: +- Feb 2019 blog post & podcast; "[Prometheus Scalability with Bryan Boreham][prometheus-scalability]" ([podcast][prometheus-scalability-podcast]) +- Feb 2019 blog post; "[How Aspen Mesh Runs Cortex in Production][aspen-mesh-2019]" - Dev 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" [video][kubecon-2018-video], ([slide][kubecon-2018-slides]) - Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" - Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides]) - Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation]) - Aug 2018 PromCon panel; "[Prometheus Long-Term Storage Approaches][promcon-2018-panel]" ([video][promcon-2018-video]) -- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides]) +- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides], write up [part 1][promcon-2017-writeup-1], [part 2][promcon-2017-writeup-2], [part 3][promcon-2017-writeup-3]) - July 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" - June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video]) - Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" ([video][kubecon-2016-video], [slides][kubecon-2016-slides]) - Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": ([video][promcon-2016-video], [slides][promcon-2016-slides]) - June 2018 design document; "[Project Frankenstein: A Multi Tenant, Scale Out Prometheus](http://goo.gl/prdUYV)" +[prometheus-scalability]: https://www.weave.works/blog/prometheus-scalability-with-bryan-boreham +[prometheus-scalability-podcast]: https://softwareengineeringdaily.com/2019/01/21/prometheus-scalability-with-bryan-boreham/ +[aspen-mesh-2019]: https://www.weave.works/blog/how-aspen-mesh-runs-cortex-in-production [kubecon-2018-talk]: https://kccna18.sched.com/event/GrXL/cortex-infinitely-scalable-prometheus-bryan-boreham-weaveworks [kubecon-2018-video]: https://www.youtube.com/watch?v=iyN40FsRQEo [kubecon-2018-slides]: https://static.sched.com/hosted_files/kccna18/9b/Cortex%20CloudNativeCon%202018.pdf @@ -55,6 +60,9 @@ To learn more about Cortex, consult the following documents & talks: [promcon-2017-talk]: https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/ [promcon-2017-video]: https://www.youtube.com/watch?v=_8DmPW4iQBQ [promcon-2017-slides]: https://promcon.io/2017-munich/slides/cortex-prometheus-as-a-service-one-year-on.pdf +[promcon-2017-writeup-1]: https://kausal.co/blog/cortex-prometheus-aas-promcon-1/ +[promcon-2017-writeup-2]: https://kausal.co/blog/cortex-prometheus-aas-promcon-2/ +[promcon-2017-writeup-3]: https://kausal.co/blog/cortex-prometheus-aas-promcon-3/ [cortex-query-optimisation-2017]: https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY [kubecon-2016-video]: https://www.youtube.com/watch?v=9Uctgnazfwk [kubecon-2016-slides]: http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service From 7797773c6205c8c007b6bc7b179e4b4081f7dc2a Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 21:23:17 +0100 Subject: [PATCH 5/7] Add another blog post. Signed-off-by: Tom Wilkie --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 7e4dcc847a..fa3d572c74 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ To learn more about Cortex, consult the following documents & talks: - Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" - Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides]) - Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation]) +- Sept 2018 blog post; "[What is Cortex?][what-is-cortex]" - Aug 2018 PromCon panel; "[Prometheus Long-Term Storage Approaches][promcon-2018-panel]" ([video][promcon-2018-video]) - Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides], write up [part 1][promcon-2017-writeup-1], [part 2][promcon-2017-writeup-2], [part 3][promcon-2017-writeup-3]) - July 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" @@ -54,6 +55,7 @@ To learn more about Cortex, consult the following documents & talks: [cloudnative-london-2018-slides]: https://www.slideshare.net/grafana/cortex-horizontally-scalable-highly-available-prometheus [cncf-2018-blog]: https://www.cncf.io/blog/2018/12/18/cortex-a-multi-tenant-horizontally-scalable-prometheus-as-a-service/ [cncf-toc-presentation]: https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6 +[what-is-cortex]: https://medium.com/weaveworks/what-is-cortex-2c30bcbd247d [promcon-2018-panel]: https://promcon.io/2018-munich/talks/panel-discussion-prometheus-long-term-storage-approaches/ [promcon-2018-video]: https://www.youtube.com/watch?v=3pTG_N8yGSU [prometheus-london-2017-video]: https://www.youtube.com/watch?v=Xi4jq2IUbLs From 88d5edae5956e6ade48795297d726ca489844a6f Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Thu, 16 May 2019 21:26:23 +0100 Subject: [PATCH 6/7] Consistency Signed-off-by: Tom Wilkie --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index fa3d572c74..bcec37a180 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Prometheus sources in a single cluster, allowing untrusted parties to share the Cortex is a [CNCF](https://cncf.io) sandbox project used in several production systems including [Weave Cloud](https://cloud.weave.works) and [GrafanaCloud](https://grafana.com/cloud). Cortex is a primarily used as a [remote write](https://prometheus.io/docs/operating/configuration/#remote_write) destination for Prometheus, with a Prometheus-compatible query API. -## Documentaion +## Documentation Read the [getting started guide](docs/getting_started.md) if you're new to the project. @@ -33,18 +33,18 @@ To learn more about Cortex, consult the following documents & talks: - Feb 2019 blog post & podcast; "[Prometheus Scalability with Bryan Boreham][prometheus-scalability]" ([podcast][prometheus-scalability-podcast]) - Feb 2019 blog post; "[How Aspen Mesh Runs Cortex in Production][aspen-mesh-2019]" -- Dev 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" [video][kubecon-2018-video], ([slide][kubecon-2018-slides]) +- Dec 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" ([video][kubecon-2018-video], [slides][kubecon-2018-slides]) - Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]" - Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides]) - Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation]) - Sept 2018 blog post; "[What is Cortex?][what-is-cortex]" - Aug 2018 PromCon panel; "[Prometheus Long-Term Storage Approaches][promcon-2018-panel]" ([video][promcon-2018-video]) - Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides], write up [part 1][promcon-2017-writeup-1], [part 2][promcon-2017-writeup-2], [part 3][promcon-2017-writeup-3]) -- July 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" -- June 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video]) +- Jul 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]" +- Jun 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video]) - Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" ([video][kubecon-2016-video], [slides][kubecon-2016-slides]) - Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": ([video][promcon-2016-video], [slides][promcon-2016-slides]) -- June 2018 design document; "[Project Frankenstein: A Multi Tenant, Scale Out Prometheus](http://goo.gl/prdUYV)" +- Jun 2018 design document; "[Project Frankenstein: A Multi Tenant, Scale Out Prometheus](http://goo.gl/prdUYV)" [prometheus-scalability]: https://www.weave.works/blog/prometheus-scalability-with-bryan-boreham [prometheus-scalability-podcast]: https://softwareengineeringdaily.com/2019/01/21/prometheus-scalability-with-bryan-boreham/ From dcce1ef0b45453b983d018fd22c3ffe6ba166206 Mon Sep 17 00:00:00 2001 From: Tom Wilkie Date: Fri, 17 May 2019 11:13:04 +0100 Subject: [PATCH 7/7] Apply suggestions from code review Co-Authored-By: Jacob Lisi Signed-off-by: Tom Wilkie --- CONTRIBUTING.md | 5 +++-- README.md | 8 ++++---- docs/architecture.md | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 72dcb9654c..c35549aeb7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -37,7 +37,7 @@ make test First, start `minikube`. You may need to load the Docker images into your minikube environment. There is -a convenience rule in the Makefile to do this: +a convenient rule in the Makefile to do this: ``` make prime-minikube @@ -77,7 +77,8 @@ Tidy up the `go.mod` and `go.sum` files: ```bash go mod tidy -git add go.mod go.sum +go mod vendor +git add go.mod go.sum vendor git commit ``` diff --git a/README.md b/README.md index bcec37a180..3897049640 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,13 @@ Go Report Card Slack -# Cortex: horizontally scalable, highly available, multi-tenant, long term storage for Prometheus. +# Cortex: horizontally scalable, highly available, multi-tenant, long term storage for Prometheus. Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for [Prometheus](https://prometheus.io). -- **Horizontally scalable:** Cortex can run across multiple machines in a cluster, exceeding the throughput and storage of a single machine. This enables you to send the metrics from multiple Prometheus servers to a single Cortex cluster and run "globally aggregated" queries across all data in a single place. -- **Highly available:** When run in a cluster, Cortex can replicate data between machines. This allows you to survive machine failure without gaps in your graphs. +- **Horizontally scalable:** Cortex can run across multiple machines in a cluster, exceeding the throughput and storage of a single machine. This enables you to send the metrics from multiple Prometheus servers to a single Cortex cluster and run "globally aggregated" queries across all data in a single place. +- **Highly available:** When run in a cluster, Cortex can replicate data between machines. This allows you to survive machine failure without gaps in your graphs. - **Multi-tenant:** Cortex can isolate data and queries from multiple different independent Prometheus sources in a single cluster, allowing untrusted parties to share the same cluster. - **Long term storage:** Cortex supports Amazon DynamoDB, Google Bigtable, Cassandra, S3 and GCS for long term storage of metric data. This allows you to durably store data for longer than the lifetime of any single machine, and use this data for long term capacity planning. @@ -100,4 +100,4 @@ in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/ove To use Cortex as part of Grafana Cloud, sign up for [GrafanaCloud](https://grafana.com/cloud) by clicking "Log In" in the top right and then "Sign Up Now". Cortex is included -as part of the Starter and Basic Hosted Grafana plans. \ No newline at end of file +as part of the Starter and Basic Hosted Grafana plans. diff --git a/docs/architecture.md b/docs/architecture.md index f2539ab406..37b9e14de0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -128,4 +128,4 @@ The interface works somewhat differently across the supported databases: A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Cortex has evolved, mainly in an attempt to better load balance writes and improve query performance. -> The current schema recommendation is the **v6 schema**. \ No newline at end of file +> The current schema recommendation is the **v10 schema**. \ No newline at end of file