Provide Dynatrace Exporter (#170)

* Provide Dynatrace Exporter

Adds a DynatraceMetricsExporter to the provided exporters. This exporter will parse CF
bindings for a Dynatrace service binding. If found, it will export metrics to the API endpoint.
For this to work, the API token field in the service credentials needs to be provided with
`otel.javaagent.extension.sap.cf.binding.dynatrace.metrics.token-name`.

Signed-off-by: Karsten Schnitter <[email protected]>

Author: KarstenSchnitter

cf-java-logging-support-opentelemetry-agent-extension/README.md

@@ -1,13 +1,14 @@
-# OpenTelemetry Java Agent Extension for SAP Cloud Logging
+# OpenTelemetry Java Agent Extension for SAP BTP Observability
 
 This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
-The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
-If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
+The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) and [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring).
+If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to those services.
 Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.
 
 The extension provides the following main features:
 
 * additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* additional exporter for metrics for [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring)
 * auto-configuration of the generic OpenTelemetry connection to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
 * adding resource attributes describing the CF application
 
@@ -40,17 +41,17 @@ See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), h
 
 Once the agent is attached to the JVM with the extension in place, there are two ways, which can be used to send data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging):
 
-1. Use the `cloud-logging` exporters explicitly as provided by the extension.
+1. Use the `cloud-logging` and/or `dynatrace` exporters explicitly as provided by the extension.
 This can be achieved via system properties or environment variables:
 ```sh
 -Dotel.logs.exporter=cloud-logging \
--Dotel.metrics.exporter=cloud-logging \
+-Dotel.metrics.exporter=cloud-logging,dynatrace \
 -Dotel.traces.exporter=cloud-logging
 
 #or
 
 export OTEL_LOGS_EXPORTER=cloud-logging
-export OTEL_METRICS_EXPORTER=cloud-logging
+export OTEL_METRICS_EXPORTER=cloud-logging,dynatrace
 export OTEL_TRACES_EXPORTER=cloud-logging
 java #...
 ```
@@ -59,7 +60,7 @@ java #...
 
 ```sh
 -Dotel.logs.exporter=otlp \
--Dotel.metrics.exporter=otlp # default value \
+-Dotel.metrics.exporter=otlp \ # default value 
 -Dotel.traces.exporter=otlp # default value
 
 #or
@@ -75,8 +76,8 @@ That means, without any configuration the agent with the extension will forward
 The difference between `cloud-logging` and `otlp` exporters are explained in an own [section](#implementation-differences-between-cloud-logging-and-otlp-exporter).
 The benefit of the `cloud-logging` exporter is, that it can be combined with a different configuration of the `otlp` exporter.
 
-For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging), the application needs to be bound to a corresponding service instance.
-The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
+For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) or [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring), the application needs to be bound to a corresponding service instances.
+The service instances can be either managed or [user-provided](#using-user-provided-service-instances).
 
 ## Configuration
 
@@ -103,16 +104,19 @@ The extension itself can be configured by specifying the following system proper
 |----------|---------------|---------|
 | `otel.javaagent.extension.sap.cf.binding.cloud-logging.label` or `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
 | `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` or `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.dynatrace.label` | `dynatrace` | The label of the managed service binding to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.dynatrace.tag` | `dynatrace` | The tag of any service binding (managed or user-provided) to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.dynatrace.metrics.token-name` | | The name of the field containing the Dynatrace API token within the service binding credentials. This is required to send metrics to Dynatrace. |
 | `otel.javaagent.extension.sap.cf.binding.user-provided.label` | `user-provided` | The label of a user-provided service binding to bind to. Note, this label is defined by the Cloud Foundry instance. |
 | `otel.javaagent.extension.sap.cf.resource.enabled` | `true` | Whether to add CF resource attributes to all events. |
 
 > The `otel.javaagent.extension.sap.*` properties are preferred over the `com.sap.otel.extension.*` properties, which are kept for compatibility. 
 Each `otel.javaagent.extension.sap.*` property can also be provided as environment variable `OTEL_JAVAAGENT_EXTENSION_SAP_*`.
 
 The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
-User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
-All matching bindings are filtered for the configured tag ("Cloud Logging" by default).
-The first binding will be taken for configuration for the OpenTelemetry exporter.
+User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" or "dynatrace" by default).
+All matching bindings are filtered for the configured tag ("Cloud Logging" od "dynatrace" by default).
+The first Cloud Logging binding will be taken for configuration for the standard OpenTelemetry (otlp) exporter.
 Preferring user-provided services over managed service instances allows better control of the binding properties, e.g. syslog drains.
 
 ### Recommended Agent Configuration
@@ -140,14 +144,16 @@ The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemet
 
 ## Using User-Provided Service Instances
 
+### SAP Cloud Logging
+
 The extension provides support not only for managed service instance of [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) but also for user-provided service instances.
 This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
 Furthermore, this helps on sharing service instances across CF orgs or landscapes.
 
-The extension requires four fields in the user-provided service credentials and needs to be tagged with the `com.sap.otel.extension.cloud-logging.tag` (default: `Cloud Logging`) documented in section [Configuration](#configuration).
+The extension requires four fields in the user-provided service credentials and needs to be tagged with the `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` (default: `Cloud Logging`) documented in section [Configuration](#configuration).
 
 | Field name | Contents |
-|------------|---------|
+|------------|----------|
 | `ingest-otlp-endpoint` | The OTLP endpoint including port. It will be prefixed with `https://`. |
 | `ingest-otlp-key` | The mTLS client key in PCKS#8 format. Line breaks as `\n`. |
 | `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
@@ -172,6 +178,31 @@ Note, that you can easily feed arbitrary credentials to the extension.
 It does not need to be [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 You can even change the tag using the configuration parameters of the extension.
 
+### Dynatrace
+
+SAP BTP internally offers a managed Dynatrace service, that is recognized by the extension.
+Externally, user-provided service instances need to be created.
+The [Dynatrace documentation](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring) explains, how to generate the necessary access url and tokens.
+The extension requires two fields in the user-provided service credentials and needs to be tagged with the `otel.javaagent.extension.sap.cf.binding.dynatrace.tag` (default: `dynatrace`) documented in section [Configuration](#configuration).
+
+| Field name | Contents |
+|------------|----------|
+| `apiurl` | The Dynatrace API endpoint, e.g. `https://apm.example.com/e/<some-uuid>/api`. This url will be appended with `/v2/otlp/v1/metrics` to create the full endpoint url. |
+| `<your_token_field>` | The API token to be used with the above endpoint. Ensure, that it has the required permissions to ingest data over the endpoint. |
+
+Do not forget to configure the name chosen for `<your_token_field>` via the respective configuration property:
+
+```sh
+java #... \
+-Dotel.javaagent.extension.sap.cf.binding.dynatrace.metrics.token-name=<your_token_field> \
+# ...
+
+# or
+
+OTEL_JAVAAGENT_EXTENSION_SAP_CF_BINDING_DYNATRACE_METRICS_TOKEN-NAME=<your_token_field>
+java #...
+```
+
 ## Implementation Differences between Cloud-Logging and OTLP Exporter
 
 The `cloud-logging` exporter provided by this extension is a facade for the `OtlpGrpcExporter` provided by the OpenTelemetry Java Agent, just like the `otlp` exporter.

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/CloudLoggingConfigurationCustomizerProvider.java

@@ -12,7 +12,7 @@ public class CloudLoggingConfigurationCustomizerProvider implements AutoConfigur
     @Override
     public void customize(AutoConfigurationCustomizer autoConfiguration) {
         autoConfiguration
-                .addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier(cfEnv));
+                .addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier());
 
         // ConfigurableLogRecordExporterProvider
     }

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudFoundryServicesAdapter.java

@@ -0,0 +1,39 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
+
+import io.pivotal.cfenv.core.CfEnv;
+import io.pivotal.cfenv.core.CfService;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+class CloudFoundryServicesAdapter {
+
+    private final CfEnv cfEnv;
+
+    CloudFoundryServicesAdapter(CfEnv cfEnv) {
+        this.cfEnv = cfEnv;
+    }
+
+    /**
+     * Stream CfServices, that match the provided properties. Empty or null values are interpreted as not applicable. No
+     * check will be performed during search. User-provided service instances will be preferred unless the
+     * {@code userProvidedLabel is null or empty. Provided only null values will return all service instances.
+     *
+     * @param serviceLabels the labels of services
+     * @param serviceTags   the tags of services
+     * @return a stream of service instances present in the CloudFoundry environment variable VCAP_SERVICES
+     */
+    Stream<CfService> stream(List<String> serviceLabels, List<String> serviceTags) {
+        Stream<CfService> services;
+        if (serviceLabels == null || serviceLabels.isEmpty())
+            services = cfEnv.findAllServices().stream();
+        else {
+            services = serviceLabels.stream().flatMap(l -> cfEnv.findServicesByLabel(l).stream());
+        }
+        if (serviceTags == null || serviceTags.isEmpty()) {
+            return services;
+        }
+        return services.filter(svc -> svc.existsByTagIgnoreCase(serviceTags.toArray(new String[0])));
+    }
+
+}

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingBindingPropertiesSupplier.java

@@ -14,7 +14,6 @@
 import java.util.function.Supplier;
 import java.util.logging.Level;
 import java.util.logging.Logger;
-import java.util.stream.Stream;
 
 public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<String, String>> {
 
@@ -26,13 +25,20 @@ public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<Strin
 
     private final CloudLoggingServicesProvider cloudLoggingServicesProvider;
 
-    public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
+    public CloudLoggingBindingPropertiesSupplier() {
+        this(new CloudLoggingServicesProvider(getDefaultProperties(), new CloudFoundryServicesAdapter(new CfEnv())));
+    }
+
+    CloudLoggingBindingPropertiesSupplier(CloudLoggingServicesProvider cloudLoggingServicesProvider) {
+        this.cloudLoggingServicesProvider = cloudLoggingServicesProvider;
+    }
+
+    private static ConfigProperties getDefaultProperties() {
         Map<String, String> defaults = new HashMap<>();
         defaults.put("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
         defaults.put("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
         defaults.put("otel.javaagent.extension.sap.cf.binding.user-provided.label", "user-provided");
-        ConfigProperties configProperties = DefaultConfigProperties.create(defaults);
-        this.cloudLoggingServicesProvider = new CloudLoggingServicesProvider(configProperties, cfEnv);
+        return DefaultConfigProperties.create(defaults);
     }
 
     private static boolean isBlank(String text) {

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingServicesProvider.java

@@ -6,9 +6,12 @@
 
 import java.util.List;
 import java.util.function.Supplier;
-import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static java.util.stream.Collectors.toList;
+
 public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>> {
 
     private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
@@ -17,15 +20,14 @@ public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>>
 
     private final List<CfService> services;
 
-    public CloudLoggingServicesProvider(ConfigProperties config, CfEnv cfEnv) {
-        String userProvidedLabel = getUserProvidedLabel(config);
-        String cloudLoggingLabel = getCloudLoggingLabel(config);
-        String cloudLoggingTag = getCloudLoggingTag(config);
-        List<CfService> userProvided = cfEnv.findServicesByLabel(userProvidedLabel);
-        List<CfService> managed = cfEnv.findServicesByLabel(cloudLoggingLabel);
-        this.services = Stream.concat(userProvided.stream(), managed.stream())
-                .filter(svc -> svc.existsByTagIgnoreCase(cloudLoggingTag))
-                .collect(Collectors.toList());
+    public CloudLoggingServicesProvider(ConfigProperties config) {
+        this(config, new CloudFoundryServicesAdapter(new CfEnv()));
+    }
+
+    CloudLoggingServicesProvider(ConfigProperties config, CloudFoundryServicesAdapter adapter) {
+        List<String> serviceLabels = asList(getUserProvidedLabel(config), getCloudLoggingLabel(config));
+        List<String> serviceTags = singletonList(getCloudLoggingTag(config));
+        this.services = adapter.stream(serviceLabels, serviceTags).collect(toList());
     }
 
     private String getUserProvidedLabel(ConfigProperties config) {

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/DynatraceServiceProvider.java

@@ -0,0 +1,47 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
+
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+import io.pivotal.cfenv.core.CfEnv;
+import io.pivotal.cfenv.core.CfService;
+
+import java.util.List;
+import java.util.function.Supplier;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+
+public class DynatraceServiceProvider implements Supplier<CfService> {
+
+    private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
+    private static final String DEFAULT_DYNATRACE_LABEL = "dynatrace";
+    private static final String DEFAULT_DYNATRACE_TAG = "dynatrace";
+
+    private final CfService service;
+
+    public DynatraceServiceProvider(ConfigProperties config) {
+        this(config, new CloudFoundryServicesAdapter(new CfEnv()));
+    }
+
+    DynatraceServiceProvider(ConfigProperties config, CloudFoundryServicesAdapter adapter) {
+        List<String> serviceLabels = asList(getUserProvidedLabel(config), getDynatraceLabel(config));
+        List<String> serviceTags = singletonList(getDynatraceTag(config));
+        this.service = adapter.stream(serviceLabels, serviceTags).findFirst().orElse(null);
+    }
+
+    private String getUserProvidedLabel(ConfigProperties config) {
+        return config.getString("otel.javaagent.extension.sap.cf.binding.user-provided.label", DEFAULT_USER_PROVIDED_LABEL);
+    }
+
+    private String getDynatraceLabel(ConfigProperties config) {
+        return config.getString("otel.javaagent.extension.sap.cf.binding.dynatrace.label", DEFAULT_DYNATRACE_LABEL);
+    }
+
+    private String getDynatraceTag(ConfigProperties config) {
+        return config.getString("otel.javaagent.extension.sap.cf.binding.dynatrace.tag", DEFAULT_DYNATRACE_TAG);
+    }
+
+    @Override
+    public CfService get() {
+        return service;
+    }
+}

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/CloudLoggingLogsExporterProvider.java

@@ -7,7 +7,6 @@
 import io.opentelemetry.sdk.autoconfigure.spi.logs.ConfigurableLogRecordExporterProvider;
 import io.opentelemetry.sdk.common.export.RetryPolicy;
 import io.opentelemetry.sdk.logs.export.LogRecordExporter;
-import io.pivotal.cfenv.core.CfEnv;
 import io.pivotal.cfenv.core.CfService;
 
 import java.time.Duration;
@@ -25,7 +24,7 @@ public class CloudLoggingLogsExporterProvider implements ConfigurableLogRecordEx
     private final CloudLoggingCredentials.Parser credentialParser;
 
     public CloudLoggingLogsExporterProvider() {
-        this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
+        this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
     }
 
     CloudLoggingLogsExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/CloudLoggingMetricsExporterProvider.java

@@ -14,7 +14,6 @@
 import io.opentelemetry.sdk.metrics.export.MetricExporter;
 import io.opentelemetry.sdk.metrics.internal.aggregator.AggregationUtil;
 import io.pivotal.cfenv.core.CfCredentials;
-import io.pivotal.cfenv.core.CfEnv;
 import io.pivotal.cfenv.core.CfService;
 
 import java.time.Duration;
@@ -35,7 +34,7 @@ public class CloudLoggingMetricsExporterProvider implements ConfigurableMetricEx
     private final CloudLoggingCredentials.Parser credentialParser;
 
     public CloudLoggingMetricsExporterProvider() {
-        this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
+        this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
     }
 
     CloudLoggingMetricsExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/CloudLoggingSpanExporterProvider.java

@@ -8,7 +8,6 @@
 import io.opentelemetry.sdk.common.export.RetryPolicy;
 import io.opentelemetry.sdk.trace.export.SpanExporter;
 import io.pivotal.cfenv.core.CfCredentials;
-import io.pivotal.cfenv.core.CfEnv;
 import io.pivotal.cfenv.core.CfService;
 
 import java.time.Duration;
@@ -26,7 +25,7 @@ public class CloudLoggingSpanExporterProvider implements ConfigurableSpanExporte
     private final CloudLoggingCredentials.Parser credentialParser;
 
     public CloudLoggingSpanExporterProvider() {
-        this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
+        this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
     }
 
     CloudLoggingSpanExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {