Summary
When exporting telemetry to a back-end/collector over gRPC or HTTP using OpenTelemetry Protocol format (OTLP), if the request results in a unsuccessful request (i.e. HTTP 4xx or 5xx), the response is read into memory with no upper-bound on the number of bytes consumed.
This could cause memory exhaustion in the consuming application if the configured back-end/collector endpoint is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response.
Details
open-telemetry/opentelemetry-dotnet#6564 introduced a change to read the response body when a non-200 HTTP status code is received when exporting telemetry to aid debugging by operators so that the error response is included in the logs emitted by the exporter for both gRPC and HTTP/protobuf.
An unintended consequence of this change is that the response body is fully read into memory when received with no upper-bound.
This vulnerability was surfaced during the investigation of GHSA-w8rr-5gcm-pp58.
Impact
If an application using the OTLP exporter is configured to use a back-end/collector endpoint that is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response the application could have its memory exhausted and create a denial-of-service condition.
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7017 updates the OTLP exporter for both gRPC and HTTP to:
- Limit the number of bytes read from the response body in an error condition to 4MiB (see open-telemetry/opentelemetry-proto#781);
- Only attempt to read the response body if OpenTelemetry error logging is enabled.
References
martincostello Remediation developer
1seal Reporter
Kielek Remediation reviewer
arminru Coordinator
Summary
When exporting telemetry to a back-end/collector over gRPC or HTTP using OpenTelemetry Protocol format (OTLP), if the request results in a unsuccessful request (i.e. HTTP 4xx or 5xx), the response is read into memory with no upper-bound on the number of bytes consumed.
This could cause memory exhaustion in the consuming application if the configured back-end/collector endpoint is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response.
Details
open-telemetry/opentelemetry-dotnet#6564 introduced a change to read the response body when a non-200 HTTP status code is received when exporting telemetry to aid debugging by operators so that the error response is included in the logs emitted by the exporter for both gRPC and HTTP/protobuf.
An unintended consequence of this change is that the response body is fully read into memory when received with no upper-bound.
This vulnerability was surfaced during the investigation of GHSA-w8rr-5gcm-pp58.
Impact
If an application using the OTLP exporter is configured to use a back-end/collector endpoint that is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response the application could have its memory exhausted and create a denial-of-service condition.
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7017 updates the OTLP exporter for both gRPC and HTTP to:
References