Merge pull request #1393 from cortexproject/docs

Update README with more description and links.

Author: tomwilkie

CONTRIBUTING.md

@@ -15,3 +15,71 @@ git commit -s -m "Here is my signed commit"
 ```
 
 You can find further instructions [here](https://github.com/probot/dco#how-it-works).
+
+## Building Cortex
+
+To build:
+```
+make
+```
+
+(By default, the build runs in a Docker container, using an image built
+with all the tools required. The source code is mounted from where you
+run `make` into the build container as a Docker volume.)
+
+To run the test suite:
+```
+make test
+```
+
+## Playing in `minikube`
+
+First, start `minikube`.
+
+You may need to load the Docker images into your minikube environment. There is
+a convenient rule in the Makefile to do this:
+
+```
+make prime-minikube
+```
+
+Then run Cortex in minikube:
+```
+kubectl apply -f ./k8s
+```
+
+(these manifests use `latest` tags, i.e. this will work if you have
+just built the images and they are available on the node(s) in your
+Kubernetes cluster)
+
+Cortex will sit behind an nginx instance exposed on port 30080.  A job is deployed to scrape itself.  Try it:
+
+http://192.168.99.100:30080/api/prom/api/v1/query?query=up
+
+If that doesn't work, your Minikube might be using a different ip address. Check with `minikube status`.
+
+### Dependency management
+
+We uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages.
+This requires a working Go environment with version 1.11 or greater, git and [bzr](http://wiki.bazaar.canonical.com/Download) installed.
+
+To add or update a new dependency, use the `go get` command:
+
+```bash
+# Pick the latest tagged release.
+go get example.com/some/module/pkg
+
+# Pick a specific version.
+go get example.com/some/module/[email protected]
+```
+
+Tidy up the `go.mod` and `go.sum` files:
+
+```bash
+go mod tidy
+go mod vendor
+git add go.mod go.sum vendor
+git commit
+```
+
+You have to commit the changes to `go.mod` and `go.sum` before submitting the pull request.

README.md

@@ -1,24 +1,85 @@
 <p align="center"><img src="imgs/logo.png" alt="Cortex Logo"></p>
 
-# Open source, horizontally scalable, multi-tenant Prometheus as a service
-
 [![Circle CI](https://circleci.com/gh/cortexproject/cortex/tree/master.svg?style=shield)](https://circleci.com/gh/cortexproject/cortex/tree/master)
 [![GoDoc](https://godoc.org/github.com/cortexproject/cortex?status.svg)](https://godoc.org/github.com/cortexproject/cortex)
+<a href="https://goreportcard.com/report/github.com/cortexproject/cortex"><img src="https://goreportcard.com/badge/github.com/cortexproject/cortex" alt="Go Report Card" /></a>
+<a href="https://cloud-native.slack.com/messages/cortex/"><img src="https://img.shields.io/badge/join%20slack-%23cortex-brightgreen.svg" alt="Slack" /></a>
 
-Cortex is a [CNCF](https://cncf.io) sandbox project used in several production
-systems including [Weave Cloud](https://cloud.weave.works), [GrafanaCloud](https://grafana.com/cloud)
-and [FreshTracks.io](https://www.freshtracks.io/).
+# Cortex: horizontally scalable, highly available, multi-tenant, long term storage for Prometheus.
 
-Cortex provides horizontally scalable, multi-tenant, long term storage for
-[Prometheus](https://prometheus.io) metrics when used as a [remote
-write](https://prometheus.io/docs/operating/configuration/#remote_write)
-destination, and a horizontally scalable, Prometheus-compatible query
-API.
+Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for
+[Prometheus](https://prometheus.io).
 
-Multi-tenant means it handles metrics from multiple independent
-Prometheus sources, keeping them separate.
+- **Horizontally scalable:** Cortex can run across multiple machines in a cluster, exceeding the throughput and storage of a single machine. This enables you to send the metrics from multiple Prometheus servers to a single Cortex cluster and run "globally aggregated" queries across all data in a single place.
+- **Highly available:** When run in a cluster, Cortex can replicate data between machines. This allows you to survive machine failure without gaps in your graphs.
+- **Multi-tenant:** Cortex can isolate data and queries from multiple different independent
+Prometheus sources in a single cluster, allowing untrusted parties to share the same cluster.
+- **Long term storage:** Cortex supports Amazon DynamoDB, Google Bigtable, Cassandra, S3 and GCS for long term storage of metric data.  This allows you to durably store data for longer than the lifetime of any single machine, and use this data for long term capacity planning.
 
-[Instrumenting Your App: Best Practices](https://www.weave.works/docs/cloud/latest/tasks/monitor/best-instrumenting/)
+Cortex is a [CNCF](https://cncf.io) sandbox project used in several production systems including [Weave Cloud](https://cloud.weave.works) and [GrafanaCloud](https://grafana.com/cloud).
+Cortex is a primarily used as a [remote write](https://prometheus.io/docs/operating/configuration/#remote_write) destination for Prometheus, with a Prometheus-compatible query API.
+
+## Documentation
+
+Read the [getting started guide](docs/getting_started.md) if you're new to the project.
+
+For a guide to contributing to Cortex, see the [contributor guidelines](CONTRIBUTING.md).
+
+You can also read [an overview of Cortex's architecture](docs/architecture.md) and [information about configuring Cortex](docs/arguments.md).
+
+## Further reading
+
+To learn more about Cortex, consult the following documents & talks:
+
+- Feb 2019 blog post & podcast; "[Prometheus Scalability with Bryan Boreham][prometheus-scalability]" ([podcast][prometheus-scalability-podcast])
+- Feb 2019 blog post; "[How Aspen Mesh Runs Cortex in Production][aspen-mesh-2019]"
+- Dec 2018 KubeCon talk; "[Cortex: Infinitely Scalable Prometheus][kubecon-2018-talk]" ([video][kubecon-2018-video], [slides][kubecon-2018-slides])
+- Dec 2018 CNCF blog post; "[Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service][cncf-2018-blog]"
+- Nov 2018 CloudNative London meetup talk; "Cortex: Horizontally Scalable, Highly Available Prometheus" ([slides][cloudnative-london-2018-slides])
+- Nov 2018 CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" ([slides][cncf-toc-presentation])
+- Sept 2018 blog post; "[What is Cortex?][what-is-cortex]"
+- Aug 2018 PromCon panel; "[Prometheus Long-Term Storage Approaches][promcon-2018-panel]" ([video][promcon-2018-video])
+- Aug 2017 PromCon talk; "[Cortex: Prometheus as a Service, One Year On][promcon-2017-talk]" ([videos][promcon-2017-video], [slides][promcon-2017-slides], write up [part 1][promcon-2017-writeup-1], [part 2][promcon-2017-writeup-2], [part 3][promcon-2017-writeup-3])
+- Jul 2017 design doc; "[Cortex Query Optimisations][cortex-query-optimisation-2017]"
+- Jun 2017 Prometheus London meetup talk; "Cortex: open-source, horizontally-scalable, distributed Prometheus" ([video][prometheus-london-2017-video])
+- Dec 2016 KubeCon talk; "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" ([video][kubecon-2016-video], [slides][kubecon-2016-slides])
+- Aug 2016 PromCon talk; "Project Frankenstein: Multitenant, Scale-Out Prometheus": ([video][promcon-2016-video], [slides][promcon-2016-slides])
+- Jun 2018 design document; "[Project Frankenstein: A Multi Tenant, Scale Out Prometheus](http://goo.gl/prdUYV)"
+
+[prometheus-scalability]: https://www.weave.works/blog/prometheus-scalability-with-bryan-boreham
+[prometheus-scalability-podcast]: https://softwareengineeringdaily.com/2019/01/21/prometheus-scalability-with-bryan-boreham/
+[aspen-mesh-2019]: https://www.weave.works/blog/how-aspen-mesh-runs-cortex-in-production
+[kubecon-2018-talk]: https://kccna18.sched.com/event/GrXL/cortex-infinitely-scalable-prometheus-bryan-boreham-weaveworks
+[kubecon-2018-video]: https://www.youtube.com/watch?v=iyN40FsRQEo
+[kubecon-2018-slides]: https://static.sched.com/hosted_files/kccna18/9b/Cortex%20CloudNativeCon%202018.pdf
+[cloudnative-london-2018-slides]: https://www.slideshare.net/grafana/cortex-horizontally-scalable-highly-available-prometheus
+[cncf-2018-blog]: https://www.cncf.io/blog/2018/12/18/cortex-a-multi-tenant-horizontally-scalable-prometheus-as-a-service/
+[cncf-toc-presentation]: https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6
+[what-is-cortex]: https://medium.com/weaveworks/what-is-cortex-2c30bcbd247d
+[promcon-2018-panel]: https://promcon.io/2018-munich/talks/panel-discussion-prometheus-long-term-storage-approaches/
+[promcon-2018-video]: https://www.youtube.com/watch?v=3pTG_N8yGSU
+[prometheus-london-2017-video]: https://www.youtube.com/watch?v=Xi4jq2IUbLs
+[promcon-2017-talk]: https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/
+[promcon-2017-video]: https://www.youtube.com/watch?v=_8DmPW4iQBQ
+[promcon-2017-slides]: https://promcon.io/2017-munich/slides/cortex-prometheus-as-a-service-one-year-on.pdf
+[promcon-2017-writeup-1]: https://kausal.co/blog/cortex-prometheus-aas-promcon-1/
+[promcon-2017-writeup-2]: https://kausal.co/blog/cortex-prometheus-aas-promcon-2/
+[promcon-2017-writeup-3]: https://kausal.co/blog/cortex-prometheus-aas-promcon-3/
+[cortex-query-optimisation-2017]: https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY
+[kubecon-2016-video]: https://www.youtube.com/watch?v=9Uctgnazfwk
+[kubecon-2016-slides]: http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service
+[promcon-2016-video]: https://youtu.be/3Tb4Wc0kfCM
+[promcon-2016-slides]: http://www.slideshare.net/weaveworks/project-frankenstein-a-multitenant-horizontally-scalable-prometheus-as-a-service
+
+## <a name="help"></a>Getting Help
+
+If you have any questions about Cortex:
+
+- Ask a question on the [Cortex Slack channel](https://cloud-native.slack.com/messages/cortex/). To invite yourself to the CNCF Slack, visit http://slack.cncf.io/.
+- <a href="https://github.com/cortexproject/cortex/issues/new">File an issue.</a>
+- Send an email to <a href="mailto:[email protected]">[email protected]</a>
+
+Your feedback is always welcome.
 
 ## Hosted Cortex (Prometheus as a service)
 
@@ -33,98 +94,10 @@ container-based applications. Sign up at https://cloud.weave.works
 and follow the instructions there. Additional help can also be found
 in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/overview/).
 
+[Instrumenting Your App: Best Practices](https://www.weave.works/docs/cloud/latest/tasks/monitor/best-instrumenting/)
+
 ### GrafanaCloud
 
 To use Cortex as part of Grafana Cloud, sign up for [GrafanaCloud](https://grafana.com/cloud)
 by clicking "Log In" in the top right and then "Sign Up Now".  Cortex is included
 as part of the Starter and Basic Hosted Grafana plans.
-
-## Contributing
-
-For a guide to contributing to Cortex, see the [contributor guidelines](CONTRIBUTING.md).
-
-## For Developers
-
-To build:
-```
-make
-```
-
-(By default, the build runs in a Docker container, using an image built
-with all the tools required. The source code is mounted from where you
-run `make` into the build container as a Docker volume.)
-
-To run the test suite:
-```
-make test
-```
-
-## Playing in `minikube`
-
-First, start `minikube`.
-
-You may need to load the Docker images into your minikube environment. There is
-a convenience rule in the Makefile to do this:
-
-```
-make prime-minikube
-```
-
-Then run Cortex in minikube:
-```
-kubectl apply -f ./k8s
-```
-
-(these manifests use `latest` tags, i.e. this will work if you have
-just built the images and they are available on the node(s) in your
-Kubernetes cluster)
-
-Cortex will sit behind an nginx instance exposed on port 30080.  A job is deployed to scrape itself.  Try it:
-
-http://192.168.99.100:30080/api/prom/api/v1/query?query=up
-
-If that doesn't work, your Minikube might be using a different ip address. Check with `minikube status`.
-
-### Dependency management
-
-We uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages.
-This requires a working Go environment with version 1.11 or greater, git and [bzr](http://wiki.bazaar.canonical.com/Download) installed.
-
-To add or update a new dependency, use the `go get` command:
-
-```bash
-# Pick the latest tagged release.
-go get example.com/some/module/pkg
-
-# Pick a specific version.
-go get example.com/some/module/[email protected]
-```
-
-Tidy up the `go.mod` and `go.sum` files:
-
-```bash
-go mod tidy
-git add go.mod go.sum
-git commit
-```
-
-You have to commit the changes to `go.mod` and `go.sum` before submitting the pull request.
-
-## Further reading
-
-To learn more about Cortex, consult the following documents/talks:
-
-- [Original design document for Project Frankenstein](http://goo.gl/prdUYV)
-- PromCon 2016 Talk: "Project Frankenstein: Multitenant, Scale-Out Prometheus": [video](https://youtu.be/3Tb4Wc0kfCM), [slides](http://www.slideshare.net/weaveworks/project-frankenstein-a-multitenant-horizontally-scalable-prometheus-as-a-service)
-- KubeCon Prometheus Day talk "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" [slides](http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service)
-- CNCF TOC Presentation; "Horizontally Scalable, Multi-tenant Prometheus" [slides](https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6)
-
-## <a name="help"></a>Getting Help
-
-If you have any questions about Cortex:
-
-- Ask a question on the [Cortex Slack channel](https://cloud-native.slack.com/messages/cortex/). To invite yourself to the CNCF Slack, visit http://slack.cncf.io/.
-- <a href="https://github.com/cortexproject/cortex/issues/new">File an issue.</a>
-- Send an email to <a href="mailto:[email protected]">[email protected]</a>
-
-Your feedback is always welcome.

docs/architecture.md

@@ -128,16 +128,4 @@ The interface works somewhat differently across the supported databases:
 
 A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Cortex has evolved, mainly in an attempt to better load balance writes and improve query performance.
 
-> The current schema recommendation is the **v6 schema**.
-
-## See also
-
-For more details on Cortex's architecture, we recommend reading/watching:
-
-- "[Project Frankenstein: A multi tenant, scale out Prometheus](https://docs.google.com/document/d/1C7yhMnb1x2sfeoe45f4mnnKConvroWhJ8KQZwIHJOuw/edit#heading=h.nimsq29kl184)" (the original design doc)
-- "[Multitenant, Scale-Out Prometheus](https://promcon.io/2016-berlin/talks/multitenant-scale-out-prometheus/)" (PromCon 2016 talk)
-- "Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service" (KubeCon Prometheus Day talk) [slides](http://www.slideshare.net/weaveworks/weave-cortex-multitenant-horizontally-scalable-prometheus-as-a-service) [video](https://www.youtube.com/watch?v=9Uctgnazfwk)
-- "[Cortex: Prometheus as a Service, One Year On](https://promcon.io/2017-munich/talks/cortex-prometheus-as-a-service-one-year-on/)" (PromCon 2017 Talk)
-- "Horizontally Scalable, Multi-tenant Prometheus" [slides](https://docs.google.com/presentation/d/190oIFgujktVYxWZLhLYN4q8p9dtQYoe4sxHgn4deBSI/edit#slide=id.g3b8e2d6f7e_0_6) (CNCF TOC Presentation)
-- [Cortex Query Woes](https://docs.google.com/document/d/1lsvSkv0tiAMPQv-V8vI2LZ8f4i9JuTRsuPI_i-XcAqY) provides details on Cortex query optimizations.
-- [Cortex: Horizontally Scalable, Highly Available Prometheus Monitoring](https://speakerdeck.com/grafana/cortex-horizontally-scalable-highly-available-prometheus-monitoring) (Presentation at Cloud Native London Meetup, Nov 2018)
+> The current schema recommendation is the **v10 schema**.