Skip to content

Conversation

@iksaif
Copy link
Member

@iksaif iksaif commented Feb 22, 2018

Ironically, graphite-web exposes no metrics at all, which makes it hard to
monitor things (people can implement things in their webservers but this
won't have the same level of details)

This takes advantage of Prometheus's client library and django integration
to add django-related metrics and allow for additional metrics to be added
later. Bonus point, we get Prometheus support too.

@codecov-io
Copy link

Codecov Report

Merging #2240 into master will decrease coverage by 0.02%.
The diff coverage is 71.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2240      +/-   ##
==========================================
- Coverage    79.6%   79.58%   -0.03%     
==========================================
  Files          85       87       +2     
  Lines        8857     8889      +32     
  Branches     1897     1900       +3     
==========================================
+ Hits         7051     7074      +23     
- Misses       1545     1551       +6     
- Partials      261      264       +3
Impacted Files Coverage Δ
webapp/graphite/instrumentation/__init__.py 100% <100%> (ø)
webapp/graphite/urls.py 85.71% <100%> (+5.71%) ⬆️
webapp/graphite/settings.py 77.61% <100%> (+0.33%) ⬆️
webapp/graphite/app_settings.py 80% <50%> (-10.91%) ⬇️
webapp/graphite/instrumentation/apps.py 68.75% <68.75%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6e45fa...b2328a9. Read the comment docs.


try:
graphite_urls.append(
url('^prometheus/', include('django_prometheus.urls')),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be something like prometeus-metrics/ ? Thinking about possible (logical) clash with #2195

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prometheus-metrics/metrics isn't nice. Maybe having prometheus/metrics, prometheus/write, prometheus/read isn't really an issue ? I could just create a prometheus apps that will later be extended to add /read and redirect its /metrics to django-prometheus

@deniszh deniszh requested a review from DanCech February 22, 2018 10:34
@DanCech
Copy link
Member

DanCech commented Feb 22, 2018

I like the idea a lot, but I'm much less keen on coupling ever more tightly with django, and I would like to see instrumentation of at least the basic steps in the find, render and processing pipeline so we can get some of the same stats that right now are just in the logs.

@iksaif
Copy link
Member Author

iksaif commented Feb 22, 2018

but I'm much less keen on coupling ever more tightly with django,

Per view/method/service stats really need to be done at the framework level. If it ever moves to another framework the metrics will change, but that's fine. Using django-prometheus gives us most of the metrics we want for free

find, render and processing pipeline

Sure, I'll add something that doesn't depend directly on the view. I'll probably instrument find(), fetch(), and the associated "multi" versions. Probably some tags methods too. Likely using the database name as a label/node.

FYI, here is how it currently looks like: metrics.txt

@deniszh
Copy link
Member

deniszh commented Feb 22, 2018

I also not really like more Django, but it's better to have some metrics then no metrics, right?
Maybe default prefiux should be switched to graphite.django. ?

@iksaif
Copy link
Member Author

iksaif commented Feb 22, 2018 via email

Ironically, graphite-web exposes no metrics at all, which makes it hard to
monitor things (people can implement things in their webservers but this
won't have the same level of details)

This takes advantage of Prometheus's client library and django integration
to add django-related metrics and allow for additional metrics to be added
later. Bonus point, we get Prometheus support too.

Also adds three basic metrics:
- storage_find_latency_seconds (per finder + global)
- storage_fetch_latency_seconds (per finder + global)
- prefetched_series (number of prefetched series per expression)
@iksaif
Copy link
Member Author

iksaif commented Feb 22, 2018

Here it is with a few basic metrics on top of what django provides for http endpoints

@deniszh
Copy link
Member

deniszh commented Feb 22, 2018

Yes, but now we have hard dependency on prometheus_client which make me sad.

@DanCech
Copy link
Member

DanCech commented Feb 23, 2018

Yeah, those metric names are horrible also. I think that it makes more sense for us to build this ourselves or to use a more graphite-friendly library.

@iksaif
Copy link
Member Author

iksaif commented Feb 24, 2018

Maybe.. I'll experiment with django-prometheus insternally then, it's doable to integrate it using local_settings.py only

@deniszh deniszh added this to the 1.2.0 milestone Aug 15, 2018
@iksaif iksaif closed this Aug 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants