Metrics Exporters: Include `Resource` Data in Labels

[Aaronontheweb]

Feature Request

Using the latest OTel RCs on NuGet (1.2.0-RC3) I have the following configuration for OTel metrics in a simple ASP.NET Core + Akka.NET app:

services.AddOpenTelemetryMetrics(builder =>
{
    builder
        .SetResourceBuilder(ResourceBuilder.CreateDefault()
    .AddService(Assembly.GetEntryAssembly().GetName().Name, serviceInstanceId: $"{Dns.GetHostName()}"))
        .AddPhobosInstrumentation()
        .AddHttpClientInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddPrometheusExporter(opt =>
        {
        });
});

And here's the data I see from the /metrics route in one instance of that application:

# HELP http_client_duration_milliseconds measure the duration of the outbound HTTP request
# TYPE http_client_duration_milliseconds histogram
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="0"} 0 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="5"} 0 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="10"} 0 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="25"} 0 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="50"} 1 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="75"} 351 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="100"} 406 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="250"} 446 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="500"} 446 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="1000"} 447 1648128438222
http_client_duration_milliseconds_bucket{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201",le="+Inf"} 447 1648128438222
http_client_duration_milliseconds_sum{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201"} 32986.69470000003 1648128438222
http_client_duration_milliseconds_count{http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="201"} 447 1648128438222

In both the /metrics output and in the Prometheus UI itself, I don't see any of the Resource metadata associated with my OTel metrics - not the app name or instance id. This isn't the case for OTel tracing, which includes this data helpfully inside each individual span.

It would be tremendously useful if I could have this Resource data associated with the metrics produced by each process.

It looks like this is addressed, to some extent, in the OTel Metrics Data model and more specifically, in the Prometheus compatibility guide: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/datamodel.md#resource-attributes

Describe the solution you'd like:

Per the OTel metrics data model I referenced earlier, it would be ideal if Resource serviceName / instanceId values could be associated with the metrics stream from each producer in my environment.

Jaeger does this inside its OTel exporter via the following:

opentelemetry-dotnet/src/OpenTelemetry.Exporter.Jaeger/JaegerExporter.cs

Lines 81 to 82 in f531391

	string serviceName = (string)this.ParentProvider.GetDefaultResource().Attributes.FirstOrDefault(
	pair => pair.Key == ResourceSemanticConventions.AttributeServiceName).Value;

However, the OTEL metrics data model muddies the waters a bit - it looks like it explicitly says that Resource attributes should not be added as Metric attributes:

The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes ....

I'm not sure if this refers to the internal data model of OTel or how the exporters are supposed to expose those metrics are both. Either way - I would love a solution for my Resource metadata to appear as queryable dimensions when building dashboards, alerts, and reports for Prometheus and any other metrics exporter.

Describe alternatives you've considered.

Manually adding my own service name / instanceId metrics via Prometheus scrape configuration or custom attributes / TagLists that i add to each of my metrics.

[cijothomas]

Prometheus Exporter spec is still experimental, and people are actively working on it. We can start incorporating the changes to the .NET Prometheus Exporter, as specs land.

(Note that Prometheus Exporter package is not stable, and will not be released as stable when we release the rest of the SDK as 1.2, because the Prometheus Exporter spec is not stable. (Metric SDK spec just reached stable yesterday))

[cijothomas]

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk_exporters/prometheus.md The Prometheus Exporter spec we'll be need to follow.

[dashpole]

An update on the Prometheus exporter spec front. From the spec for handling resource in prometheus exporters:

In SDK Prometheus (pull) exporters, resource attributes SHOULD be converted to a single target_info metric; otherwise, they MUST be dropped, and MUST NOT be attached as labels to other metric families. The target_info metric MUST be an info-typed metric whose labels MUST include the resource attributes, and MUST NOT include any other labels. There MUST be at most one target_info metric exposed on an SDK Prometheus endpoint.

[nicomoya123]

Any update on this ?

[robertcoltheart]

I'd like to see this feature included in the Prom exporter, @cijothomas are you happy to accept a PR for this?

[cijothomas]

I'd like to see this feature included in the Prom exporter, @cijothomas are you happy to accept a PR for this?

yes!