Add ability to turn off _created metrics

[csmarchbanks]

As per the June 24th 2021 Prometheus Dev Summit there is consensus that all first party clients should have a parameter to turn off _created metrics.

[baryluk]

Any update on this?

_created metrics are not very useful (I personally never used them), and add to resource consumption through the monitoring stack.

Thanks.

[csmarchbanks]

Hello, no progress has been made but I would be happy to accept a contribution to provide a flag.

[nidemidovich]

May be I don't understand this feature clearly, but seems like it doesn't work as expected. I created few metrics in my Python+FastAPI+Gunicorn application

from prometheus_client import CollectorRegistry, Counter

multiproc_registry = CollectorRegistry()

PROFILES_OK = Counter(
    "profiles_ok",
    "Number of successfully received profiles",
    ["request", "method", "platform"],
    registry=multiproc_registry,
)

And then I added multiproc_registry to the instance of FastAPI

def make_metrics_app():
    multiprocess.MultiProcessCollector(
        multiproc_registry, path=settings.PROMETHEUS_MULTIPROC_DIR
    )
    return make_asgi_app(registry=multiproc_registry)
    
application.mount("/metrics", make_metrics_app())

But still I facing _created gauge

[csmarchbanks]

Did you set PROMETHEUS_DISABLE_CREATED_SERIES=True as well? This library tries to match as many SHOULD clauses as possible in OpenMetrics so the created series are on by default.

[nidemidovich]

Yes, I did. I even tried set the variable before import prometheus_client

[nidemidovich]

Any updates on this? @csmarchbanks Sorry for mentioning

[csmarchbanks]

I haven't had time yet, I am hoping to try to reproduce today, but it might have to wait until next week.

[baryluk]

I was using this flag for some time, and sometimes it does not work, even if I set it in-process before importing prometheus_client, it does not work as expected. I could envvar set it on a process startup, but it just clutters startup scripts, or hand started stuff. I do not have anything that imports it transitively before my import.

So, I hacked and just set internal variable:

#!/usr/bin/env python3

# ...
import os
# ...

os.environ["PROMETHEUS_DISABLE_CREATED_SERIES"] = "True"

import prometheus_client

prometheus_client.metrics._use_created = False

...

[csmarchbanks]

Yeah, we need to make that variable settable in code. See #933.

[nidemidovich]

I'll try @baryluk hack for now. But hope you figure out how to solve the issue in better way.

May be reopen the issue?

[csmarchbanks]

I spent a bit of time trying to reproduce this, and if I set the environment variable before running the script it seems to consistently work. My guess is that setting the environment variable isn't being propagated to child processes, similar to if you try to set PROMETHEUS_MULTIPROC_DIR inside of python. The readme calls this out in multiprocess mode, but maybe we should explicitly call it out for PROMETHEUS_DISABLE_CREATED_SERIES too?

Also, if you are using multi-process mode _created metrics are actually not created at all. Which made me realize you are specifying registry= when defining your metrics. This is not supported and can cause duplicate/wrong metrics to be exported. See the note in https://github.com/prometheus/client_python/#multiprocess-mode-eg-gunicorn about how registries cannot be used like normal.

I don't think this issue should be reopened, the work for disabling _created from code is already captured in the other issue.

[baryluk]

That is interesting, because I do not use child subprocesses. I do use some threading, but not subprocesses. And if I do use subprocesses, this is to run non-python auxilary programs (subprocess.run/Popen).

Not sure why setting env before program start, and modying it in the main thread, before starting any other sub-processes, should have any differences. Maybe os.environ["..."] does not C environment, but just some internal dict?