Optimize OpenTelemetry Metric and Tracing autoconfiguration

[FranPregernik]

Hi!
I would like to propose an improvement in the way the OpenTelemetry metrics and tracing are auto-configured.
This is related to the issue: #30156 and my comment at the end.

As a starting point, the tracing OpenTelemetryAutoConfiguration sets up an instance of the OpenTelemetry and fetches a list of tracing providers. It is a great way to specify a non-supported exporter (e.g. OtlpGrpcSpanExporter instead of brave/zipkin/wavefront).

But the OtlpMetricsExportAutoConfiguration does not use the OpenTelemetry instance at all and registers the OtlpMeterRegistry based on a configuration. Currently I have no way of specifying a OtlpGrpcMetricExporter that it want to use.

My proposition is to have separate basic OpenTelemetry auto-configuration, a OpenTelemetry tracing auto-configuration and finally a OpenTelemetry metrics auto-configuration.

The main OpenTeleletry autoconfiguration would have the following:

    @Bean
    @ConditionalOnMissingBean
    // puts the sevice name based on the spring.application.name, additionally reads in additional attributes like OtlpMetricsExportAutoConfiguration does
    fun otelResource(environment: Environment): Resource {
        val applicationName = environment.getProperty("spring.application.name", "application")
        return Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, applicationName))
    }

    @Bean
    @ConditionalOnMissingBean
    fun openTelemetry(
        sdkTracerProvider: ObjectProvider<SdkTracerProvider>,
        sdkMeterProvider: ObjectProvider<SdkMeterProvider>,  // This is the new bit - custom SdkMeterProvider that is set up like SdkTracerProvider allowing me to create a OtlpGrpcMetricExporter bean
        contextPropagators: ObjectProvider<ContextPropagators>
    ): OpenTelemetry {
        val builder = OpenTelemetrySdk.builder()
        contextPropagators.ifUnique {
            builder.setPropagators(it)
        }
        sdkMeterProvider.ifUnique {
            builder.setMeterProvider(it)
        }
        sdkTracerProvider.ifUnique {
            // set by org.springframework.boot.actuate.autoconfigure.tracing.OpenTelemetryAutoConfiguration.otelSdkTracerProvider
            builder.setTracerProvider(it)
        }
        return builder.build()
    }

The otelResource is a consistent way to specify the service name and other parameters for both metrics and tracing. The OpenTelemetry accepts the additional sdkMeterProvider and allows a setup of a custom MetricExporter.

The OtlpMetricsExportAutoConfiguration would create the SdkMeterProvider, Exporter and Registry:

    @Bean
    fun sdkMeterProvider(
        metricExportersProvider: ObjectProvider<List<MetricExporter>>,
        otelResource: Resource
    ): SdkMeterProvider {
        // from https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/OpenTelemetryAutoConfiguration.java
        val meterProviderBuilder = SdkMeterProvider.builder()
        val interval = properties.metrics.interval
        metricExportersProvider.getIfAvailable { emptyList() }
            .map { metricExporter ->
                val metricReaderBuilder = PeriodicMetricReader.builder(metricExporter)
                if (interval != null) {
                    metricReaderBuilder.setInterval(interval)
                }
                metricReaderBuilder.build()
            }
            .forEach { reader ->
                meterProviderBuilder.registerMetricReader(reader)
            }
        return meterProviderBuilder.setResource(otelResource).build()
    }

    @Bean
    fun otelMeterRegistry(openTelemetry: OpenTelemetry): MeterRegistry {
        return OpenTelemetryMeterRegistry.create(openTelemetry)
    }
    
    @Bean
    @ConditionalOnMissingBean(MetricExporter::class)
    fun metricExporter(...) : OtlpHttpMetricExporter {  
       // ...
    }

So now both tracing and metrics use the same OpenTelemetry instance that is set up with a single OpenTelemetry Resource instance. The default metric exporter is the OtlpHttpMetricExporter and can be overridden with the GRPC version.

[wilkinsona]

Thanks for the suggestion, @FranPregernik. Unfortunately, I'm struggling to follow exactly what you mean. For example, you seem to be proposing a change to OtlpMetricsExportAutoConfiguration such that it would no longer define a OtlpMeterRegistry and would define an OpenTelemetryMeterRegistry instead. From what I can tell, that appears to be part of https://github.com/open-telemetry/opentelemetry-java-instrumentation which Spring Boot does not use.

I think it may be easier if we take a step back and focus on what you'd like to be able to do rather than describing code changes that you believe will make it possible. Alternatively, seeing Java code (rather than Kotlin) complete with imports may help us to better understand your goals.

[jimbogithub]

If I've understood @FranPregernik correctly and from an examination of the code:

1. OtlpMeterRegistry does its own direct publishing and supports HTTP only. It would be better if it bridged to OpenTelemetry and allowed us to supply either OtlpGrpcMetricExporter or OtlpHttpMetricExporter (or supported the standard OTEL_... envs (https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/) to choose).
2. Resource config is inconsistent. OtlpConfig provides the necessary settings for metrics but there is no such provision for traces. Traces get theirs from OpenTelemetryAutoConfiguration.otelSdkTracerProvider with no means of adding attributes other than to completely replace that bean. A common overridable Resource bean would be useful and by default reuse the OtlpConfig for traces as well.
3. (adding a new one of my own) OpenTelemetryAutoConfiguration builds an OpenTelemetry instance. Other OpenTelemetry libraries do not see this instance, e.g. opentelemetry-jdbc uses GlobalOpenTelemetry.get to obtain it's instance. There needs to be a way of sharing. I'm currently doing it as discussed in https://stackoverflow.com/questions/75293292/spring-micrometer-opentelemetry-exporter-otlp

[FranPregernik]

@wilkinsona I am sorry, I should have checked the libraries of the classes I mentioned. My bad.

@jimbogithub Thank you for stepping in. That is it exactly. However, one small sidenote on the last point, I have a PR open-telemetry/opentelemetry-java-instrumentation#7697 that allows JDBC instrumentation to accept an external OpenTelemetry instance.

[braunsonm]

Also interested in #2 in particular. Currently there is no way to identify a traced application's environment which makes it difficult to use in workplaces that have dev, staging, prod, etc.

Ideally you can provide a configuration property like metrics, but for now the whole SdkTracerProvider needs to be replaced.

[zeitlinger]

I think this could work:

• use OpenTelemetry micrometer bridge instead of OtlpMeterRegistry, which is also used in the java agent (https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/micrometer/micrometer-1.5)
• use SDK auto config, so you can use all config parameters that you can use in java agent (https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure)

We would just need to pass all spring properties with otel. prefix to SDK autoconfigure by setting them as system properties. WDYT?

[chicobento]

We would just need to pass all spring properties with otel. prefix to SDK autoconfigure by setting them as system properties. WDYT?

We are using this strategy so we can support all otel configuration flags - and provide an additional means of configuration - via spring boot properties. I was going to propose this for Spring, but one drawback I see for now is that otel-sdk-autoconfigure version has -alpha suffix: see their releases https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-sdk-extension-autoconfigure. Not sure when they'll promote it to stable.

ps: instead of using system properties you can provide a properties supplier and provide a map that was generated from spring boot properties. E.g:

Map<String, String> otelProperties = readPropertiesStartingWith("otel", env);
final AutoConfiguredOpenTelemetrySdkBuilder sdkBuilder = AutoConfiguredOpenTelemetrySdk.builder();
sdkBuilder.addPropertiesSupplier(() -> otelProperties);