exclude_otel_instrumented_services: port fallback over-suppresses — a traces-only (or metrics-only) service loses the other signal entirely

[bourbonkk]

Summary

Follow-up to #2711 / #2712. #2712 fixed the PeerPort → HostPort comparison so the gRPC port fallback in exclude_otel_instrumented_services actually fires. That fix is correct, but it activates a pre-existing design problem that #2711 raised and that was never resolved (the issue was auto-closed by the PR merge):

When the OTLP :path cannot be decoded and detection falls back to port matching, a single OTLP export flags the service for BOTH traces AND metrics — because traces and metrics share the same OTLP port (4317). As a result, a service that exports only one signal via its SDK has OBI suppress the other signal too, which nothing else produces → that signal is lost entirely (over-suppression).

Since #2712 this is now live behavior, not dead code.

Root cause (code level)

1. One export span satisfies both predicates

IsExportMetricsSpan and IsExportTracesSpan each accept a span by path OR port (pkg/appolly/app/request/span.go:2143-2159):

func (s *Span) IsExportMetricsSpan(...) bool {
    ... return s.isMetricsExportURL() || s.sendsMetricsOnOtelPort(...)
}
func (s *Span) IsExportTracesSpan(...) bool {
    ... return s.isTracesExportURL()  || s.sendsTracesOnOtelPort(...)
}

When :path == "*" (see §3), the URL matchers return false and both fall through to the port heuristic. But both heuristics converge on the same port (span.go:2069-2124):

sendsMetricsOnGrpcOtelPort → (no METRICS_ENDPOINT) → sendsOnDefaultGrpcOtelPort → OTLP_ENDPOINT / 4317
sendsTracesOnGrpcOtelPort  → (no TRACES_ENDPOINT)  → sendsOnDefaultGrpcOtelPort → OTLP_ENDPOINT / 4317

With a single generic OTEL_EXPORTER_OTLP_ENDPOINT, HostPort == 4317 makes both IsExportMetricsSpan and IsExportTracesSpan return true for the same span.

2. Sticky flags accumulate across spans

checkIfExportsOTel (pkg/ebpf/common/pids.go:225) sets one flag per span via if / else if, but the flags are set-once/sticky (EnsureExportsOTel* in pkg/appolly/discover/exec/file.go:220, "returns true if this call flipped the flag"):

if span.IsExportMetricsSpan(...) && fi.EnsureExportsOTelMetrics() {
    pf.reportAvoidedService(fi, "metrics")
} else if span.IsExportTracesSpan(...) && fi.EnsureExportsOTelTraces() {
    pf.reportAvoidedService(fi, "traces")
}

Because every export span matches both predicates, the 1st span flips the metrics flag and the 2nd span (metrics already set → short-circuits to the else-if) flips the traces flag. SDKs export both signals periodically, so within seconds both flags are set.

3. Drop points then suppress both

// RED metrics — pkg/export/otel/metrics.go:896 (and pkg/export/prom/prom.go:938)
return span.Service.Features.AppRED() && !span.Service.ExportsOTelMetrics()
// traces — pkg/export/otel/tracesgen/tracesgen.go:244
return request.IgnoreTraces(span) || span.Service.ExportsOTelTraces() || !acceptSpan(...)

Once both flags are set, OBI's own RED metrics and traces for that service are dropped.

Why :path is "*" (the trigger)

pkg/ebpf/common/http2grpc_transform.go:409:

if path == "" || !validPath.MatchString(path) {
    path = "*"
}

"*" is a sentinel for "OBI could not decode the path", not malformed traffic. OTLP exporters hold a single long-lived HTTP/2 connection and send an identical :path every export, so HPACK compresses it to a dynamic-table index after the first request. OBI attaching mid-stream lacks that dynamic-table state and cannot resolve the index → "*". This is common (and especially frequent with Python grpcio exporters), which is exactly why the port fallback exists.

Reproduction

• Service currency exports traces only via OTLP gRPC to a generic endpoint OTEL_EXPORTER_OTLP_ENDPOINT=collector:4317 (no metrics SDK).
• Its trace-export client spans arrive with Path="*", HostPort=4317, status UNSET.
• IsExportTracesSpan → true (port) ✅ correct.
• IsExportMetricsSpan → also true (same port) ❌ misclassification.
• ExportsOTelMetrics gets set → metrics.go:896 drops currency's RED metrics.
• Net result: currency has no RED metrics anywhere — the SDK never sent them and OBI is now suppressing its own. (Symmetric case: a metrics-only service loses traces.)

Proposed directions

I'd suggest starting with (1) as a near-term safeguard against the over-suppression that #2712 activated, and pursuing (3) as the long-term fix.

1. Conservative fallback default (near-term). When detection relies on the port-only signal (opaque :path), do not set both flags from a single match; prefer not suppressing when the signal type is ambiguous. In observability, silent data loss is worse than duplication, so the ambiguous case should fail open. This directly contains the regression introduced by Fix OTLP export detection: compare server port (HostPort) not client ephemeral port (PeerPort) #2712 without waiting for a full decoding fix.
2. Granular suppression. Suppress only overlapping metric names rather than the whole signal type.
3. Improve :path decoding reliability (long-term). Track per-connection HPACK dynamic-table state so an indexed :path can be resolved, making the port fallback unnecessary and removing the ambiguity at its source.

Happy to bring this to the OBI SIG.

Environment

• OBI version: main (post-Fix OTLP export detection: compare server port (HostPort) not client ephemeral port (PeerPort) #2712)
• Affects: any OTLP gRPC exporter whose :path is HPACK-indexed (notably Python grpcio)