Skip to content

Provide Dynatrace Exporter #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 45 additions & 14 deletions cf-java-logging-support-opentelemetry-agent-extension/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# OpenTelemetry Java Agent Extension for SAP Cloud Logging
# OpenTelemetry Java Agent Extension for SAP BTP Observability

This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) and [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring).
If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to those services.
Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.

The extension provides the following main features:

* additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
* additional exporter for metrics for [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring)
* auto-configuration of the generic OpenTelemetry connection to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
* adding resource attributes describing the CF application

Expand Down Expand Up @@ -40,17 +41,17 @@ See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), h

Once the agent is attached to the JVM with the extension in place, there are two ways, which can be used to send data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging):

1. Use the `cloud-logging` exporters explicitly as provided by the extension.
1. Use the `cloud-logging` and/or `dynatrace` exporters explicitly as provided by the extension.
This can be achieved via system properties or environment variables:
```sh
-Dotel.logs.exporter=cloud-logging \
-Dotel.metrics.exporter=cloud-logging \
-Dotel.metrics.exporter=cloud-logging,dynatrace \
-Dotel.traces.exporter=cloud-logging

#or

export OTEL_LOGS_EXPORTER=cloud-logging
export OTEL_METRICS_EXPORTER=cloud-logging
export OTEL_METRICS_EXPORTER=cloud-logging,dynatrace
export OTEL_TRACES_EXPORTER=cloud-logging
java #...
```
Expand All @@ -59,7 +60,7 @@ java #...

```sh
-Dotel.logs.exporter=otlp \
-Dotel.metrics.exporter=otlp # default value \
-Dotel.metrics.exporter=otlp \ # default value
-Dotel.traces.exporter=otlp # default value

#or
Expand All @@ -75,8 +76,8 @@ That means, without any configuration the agent with the extension will forward
The difference between `cloud-logging` and `otlp` exporters are explained in an own [section](#implementation-differences-between-cloud-logging-and-otlp-exporter).
The benefit of the `cloud-logging` exporter is, that it can be combined with a different configuration of the `otlp` exporter.
Comment on lines 76 to 77
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would assume this also applies for Dynatrace, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there is no default configuration for the otlp exporter and Dynatrace. You need to use the dynatrace exporter explicitly. Note, that otlp is considered to be a legacy/fallback approach. Since Cloud Logging is always available, it is used as the default.


For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging), the application needs to be bound to a corresponding service instance.
The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) or [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring), the application needs to be bound to a corresponding service instances.
The service instances can be either managed or [user-provided](#using-user-provided-service-instances).

## Configuration

Expand All @@ -103,16 +104,19 @@ The extension itself can be configured by specifying the following system proper
|----------|---------------|---------|
| `otel.javaagent.extension.sap.cf.binding.cloud-logging.label` or `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
| `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` or `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
| `otel.javaagent.extension.sap.cf.binding.dynatrace.label` | `dynatrace` | The label of the managed service binding to bind to. |
| `otel.javaagent.extension.sap.cf.binding.dynatrace.tag` | `dynatrace` | The tag of any service binding (managed or user-provided) to bind to. |
| `otel.javaagent.extension.sap.cf.binding.dynatrace.metrics.token-name` | | The name of the field containing the Dynatrace API token within the service binding credentials. This is required to send metrics to Dynatrace. |
| `otel.javaagent.extension.sap.cf.binding.user-provided.label` | `user-provided` | The label of a user-provided service binding to bind to. Note, this label is defined by the Cloud Foundry instance. |
| `otel.javaagent.extension.sap.cf.resource.enabled` | `true` | Whether to add CF resource attributes to all events. |

> The `otel.javaagent.extension.sap.*` properties are preferred over the `com.sap.otel.extension.*` properties, which are kept for compatibility.
Each `otel.javaagent.extension.sap.*` property can also be provided as environment variable `OTEL_JAVAAGENT_EXTENSION_SAP_*`.

The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
All matching bindings are filtered for the configured tag ("Cloud Logging" by default).
The first binding will be taken for configuration for the OpenTelemetry exporter.
User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" or "dynatrace" by default).
All matching bindings are filtered for the configured tag ("Cloud Logging" od "dynatrace" by default).
The first Cloud Logging binding will be taken for configuration for the standard OpenTelemetry (otlp) exporter.
Preferring user-provided services over managed service instances allows better control of the binding properties, e.g. syslog drains.

### Recommended Agent Configuration
Expand Down Expand Up @@ -140,14 +144,16 @@ The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemet

## Using User-Provided Service Instances

### SAP Cloud Logging

The extension provides support not only for managed service instance of [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) but also for user-provided service instances.
This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
Furthermore, this helps on sharing service instances across CF orgs or landscapes.

The extension requires four fields in the user-provided service credentials and needs to be tagged with the `com.sap.otel.extension.cloud-logging.tag` (default: `Cloud Logging`) documented in section [Configuration](#configuration).
The extension requires four fields in the user-provided service credentials and needs to be tagged with the `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` (default: `Cloud Logging`) documented in section [Configuration](#configuration).

| Field name | Contents |
|------------|---------|
|------------|----------|
| `ingest-otlp-endpoint` | The OTLP endpoint including port. It will be prefixed with `https://`. |
| `ingest-otlp-key` | The mTLS client key in PCKS#8 format. Line breaks as `\n`. |
| `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
Expand All @@ -172,6 +178,31 @@ Note, that you can easily feed arbitrary credentials to the extension.
It does not need to be [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
You can even change the tag using the configuration parameters of the extension.

### Dynatrace

SAP BTP internally offers a managed Dynatrace service, that is recognized by the extension.
Externally, user-provided service instances need to be created.
The [Dynatrace documentation](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring) explains, how to generate the necessary access url and tokens.
The extension requires two fields in the user-provided service credentials and needs to be tagged with the `otel.javaagent.extension.sap.cf.binding.dynatrace.tag` (default: `dynatrace`) documented in section [Configuration](#configuration).

| Field name | Contents |
|------------|----------|
| `apiurl` | The Dynatrace API endpoint, e.g. `https://apm.example.com/e/<some-uuid>/api`. This url will be appended with `/v2/otlp/v1/metrics` to create the full endpoint url. |
| `<your_token_field>` | The API token to be used with the above endpoint. Ensure, that it has the required permissions to ingest data over the endpoint. |

Do not forget to configure the name chosen for `<your_token_field>` via the respective configuration property:

```sh
java #... \
-Dotel.javaagent.extension.sap.cf.binding.dynatrace.metrics.token-name=<your_token_field> \
# ...

# or

OTEL_JAVAAGENT_EXTENSION_SAP_CF_BINDING_DYNATRACE_METRICS_TOKEN-NAME=<your_token_field>
java #...
```

## Implementation Differences between Cloud-Logging and OTLP Exporter

The `cloud-logging` exporter provided by this extension is a facade for the `OtlpGrpcExporter` provided by the OpenTelemetry Java Agent, just like the `otlp` exporter.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ public class CloudLoggingConfigurationCustomizerProvider implements AutoConfigur
@Override
public void customize(AutoConfigurationCustomizer autoConfiguration) {
autoConfiguration
.addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier(cfEnv));
.addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier());

// ConfigurableLogRecordExporterProvider
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;

import io.pivotal.cfenv.core.CfEnv;
import io.pivotal.cfenv.core.CfService;

import java.util.List;
import java.util.stream.Stream;

class CloudFoundryServicesAdapter {

private final CfEnv cfEnv;

CloudFoundryServicesAdapter(CfEnv cfEnv) {
this.cfEnv = cfEnv;
}

/**
* Stream CfServices, that match the provided properties. Empty or null values are interpreted as not applicable. No
* check will be performed during search. User-provided service instances will be preferred unless the
* {@code userProvidedLabel is null or empty. Provided only null values will return all service instances.
*
* @param serviceLabels the labels of services
* @param serviceTags the tags of services
* @return a stream of service instances present in the CloudFoundry environment variable VCAP_SERVICES
*/
Stream<CfService> stream(List<String> serviceLabels, List<String> serviceTags) {
Stream<CfService> services;
if (serviceLabels == null || serviceLabels.isEmpty())
services = cfEnv.findAllServices().stream();
else {
services = serviceLabels.stream().flatMap(l -> cfEnv.findServicesByLabel(l).stream());
}
if (serviceTags == null || serviceTags.isEmpty()) {
return services;
}
return services.filter(svc -> svc.existsByTagIgnoreCase(serviceTags.toArray(new String[0])));
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import java.util.function.Supplier;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.stream.Stream;

public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<String, String>> {

Expand All @@ -26,13 +25,20 @@ public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<Strin

private final CloudLoggingServicesProvider cloudLoggingServicesProvider;

public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
public CloudLoggingBindingPropertiesSupplier() {
this(new CloudLoggingServicesProvider(getDefaultProperties(), new CloudFoundryServicesAdapter(new CfEnv())));
}

CloudLoggingBindingPropertiesSupplier(CloudLoggingServicesProvider cloudLoggingServicesProvider) {
this.cloudLoggingServicesProvider = cloudLoggingServicesProvider;
}

private static ConfigProperties getDefaultProperties() {
Map<String, String> defaults = new HashMap<>();
defaults.put("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
defaults.put("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
defaults.put("otel.javaagent.extension.sap.cf.binding.user-provided.label", "user-provided");
ConfigProperties configProperties = DefaultConfigProperties.create(defaults);
this.cloudLoggingServicesProvider = new CloudLoggingServicesProvider(configProperties, cfEnv);
return DefaultConfigProperties.create(defaults);
}

private static boolean isBlank(String text) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@

import java.util.List;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.Stream;

import static java.util.Arrays.asList;
import static java.util.Collections.singletonList;
import static java.util.stream.Collectors.toList;

public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>> {

private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
Expand All @@ -17,15 +20,14 @@ public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>>

private final List<CfService> services;

public CloudLoggingServicesProvider(ConfigProperties config, CfEnv cfEnv) {
String userProvidedLabel = getUserProvidedLabel(config);
String cloudLoggingLabel = getCloudLoggingLabel(config);
String cloudLoggingTag = getCloudLoggingTag(config);
List<CfService> userProvided = cfEnv.findServicesByLabel(userProvidedLabel);
List<CfService> managed = cfEnv.findServicesByLabel(cloudLoggingLabel);
this.services = Stream.concat(userProvided.stream(), managed.stream())
.filter(svc -> svc.existsByTagIgnoreCase(cloudLoggingTag))
.collect(Collectors.toList());
public CloudLoggingServicesProvider(ConfigProperties config) {
this(config, new CloudFoundryServicesAdapter(new CfEnv()));
}
Comment on lines +23 to +25
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like the simplification that comes from the refactoring


CloudLoggingServicesProvider(ConfigProperties config, CloudFoundryServicesAdapter adapter) {
List<String> serviceLabels = asList(getUserProvidedLabel(config), getCloudLoggingLabel(config));
List<String> serviceTags = singletonList(getCloudLoggingTag(config));
this.services = adapter.stream(serviceLabels, serviceTags).collect(toList());
}

private String getUserProvidedLabel(ConfigProperties config) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;

import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.pivotal.cfenv.core.CfEnv;
import io.pivotal.cfenv.core.CfService;

import java.util.List;
import java.util.function.Supplier;

import static java.util.Arrays.asList;
import static java.util.Collections.singletonList;

public class DynatraceServiceProvider implements Supplier<CfService> {

private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
private static final String DEFAULT_DYNATRACE_LABEL = "dynatrace";
private static final String DEFAULT_DYNATRACE_TAG = "dynatrace";

private final CfService service;

public DynatraceServiceProvider(ConfigProperties config) {
this(config, new CloudFoundryServicesAdapter(new CfEnv()));
}

DynatraceServiceProvider(ConfigProperties config, CloudFoundryServicesAdapter adapter) {
List<String> serviceLabels = asList(getUserProvidedLabel(config), getDynatraceLabel(config));
List<String> serviceTags = singletonList(getDynatraceTag(config));
this.service = adapter.stream(serviceLabels, serviceTags).findFirst().orElse(null);
}

private String getUserProvidedLabel(ConfigProperties config) {
return config.getString("otel.javaagent.extension.sap.cf.binding.user-provided.label", DEFAULT_USER_PROVIDED_LABEL);
}

private String getDynatraceLabel(ConfigProperties config) {
return config.getString("otel.javaagent.extension.sap.cf.binding.dynatrace.label", DEFAULT_DYNATRACE_LABEL);
}

private String getDynatraceTag(ConfigProperties config) {
return config.getString("otel.javaagent.extension.sap.cf.binding.dynatrace.tag", DEFAULT_DYNATRACE_TAG);
}

@Override
public CfService get() {
return service;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import io.opentelemetry.sdk.autoconfigure.spi.logs.ConfigurableLogRecordExporterProvider;
import io.opentelemetry.sdk.common.export.RetryPolicy;
import io.opentelemetry.sdk.logs.export.LogRecordExporter;
import io.pivotal.cfenv.core.CfEnv;
import io.pivotal.cfenv.core.CfService;

import java.time.Duration;
Expand All @@ -25,7 +24,7 @@ public class CloudLoggingLogsExporterProvider implements ConfigurableLogRecordEx
private final CloudLoggingCredentials.Parser credentialParser;

public CloudLoggingLogsExporterProvider() {
this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
}

CloudLoggingLogsExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import io.opentelemetry.sdk.metrics.export.MetricExporter;
import io.opentelemetry.sdk.metrics.internal.aggregator.AggregationUtil;
import io.pivotal.cfenv.core.CfCredentials;
import io.pivotal.cfenv.core.CfEnv;
import io.pivotal.cfenv.core.CfService;

import java.time.Duration;
Expand All @@ -35,7 +34,7 @@ public class CloudLoggingMetricsExporterProvider implements ConfigurableMetricEx
private final CloudLoggingCredentials.Parser credentialParser;

public CloudLoggingMetricsExporterProvider() {
this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
}

CloudLoggingMetricsExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
import io.opentelemetry.sdk.common.export.RetryPolicy;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.pivotal.cfenv.core.CfCredentials;
import io.pivotal.cfenv.core.CfEnv;
import io.pivotal.cfenv.core.CfService;

import java.time.Duration;
Expand All @@ -26,7 +25,7 @@ public class CloudLoggingSpanExporterProvider implements ConfigurableSpanExporte
private final CloudLoggingCredentials.Parser credentialParser;

public CloudLoggingSpanExporterProvider() {
this(config -> new CloudLoggingServicesProvider(config, new CfEnv()).get(), CloudLoggingCredentials.parser());
this(config -> new CloudLoggingServicesProvider(config).get(), CloudLoggingCredentials.parser());
}

CloudLoggingSpanExporterProvider(Function<ConfigProperties, Stream<CfService>> serviceProvider, CloudLoggingCredentials.Parser credentialParser) {
Expand Down
Loading