Proposal: Network-Layer Sub-Spans for Request Latency Decomposition

[NameHaibinZhang]

Component

OBI (eBPF instrumentation)

Problem

1. Summary

This proposal introduces network-layer sub-spans to OBI-generated traces, decomposing
request latency into discrete phases: DNS resolution, TCP connection establishment, TLS
handshake, and application processing. By surfacing these phases as child spans (or span
links) beneath existing HTTP/gRPC request spans, users gain immediate visibility into
whether observed latency originates in the network stack or in application logic — without
any application code changes.

2. Motivation

Problem Statement

When a service-to-service request is slow, operators today see a single HTTP or gRPC span
whose duration conflates network overhead with application processing time. Distinguishing
"DNS took 200 ms" from "the upstream handler was slow" requires manual packet captures or
separate monitoring tools, breaking the trace-centric workflow.

Real-World Context

This proposal is motivated by recurring issues observed in production cloud environments.
When supporting customers running microservice workloads on cloud infrastructure, a
frequently reported problem is: "requests between services are slow, but we cannot
determine whether it is the network or the application." Today, resolving such cases
requires combining distributed traces with separate network diagnostic tools (tcpdump,
dig, packet captures), which is time-consuming, requires elevated privileges, and is
often impractical in containerized/serverless environments. Native network-layer
visibility within OBI traces would dramatically reduce mean-time-to-resolution (MTTR)
for these cases.

Use Cases

Scenario	Value
DNS resolution delays (e.g., stale cache, slow resolver)	Identify misconfigured resolvers or cache TTL issues
TCP connection establishment overhead (SYN retransmits, connection pool exhaustion)	Detect capacity issues before they cascade
TLS handshake latency (certificate chain validation, OCSP stapling)	Surface crypto overhead that disproportionately affects short requests
Cloud provider network variability	Correlate cross-AZ latency spikes with specific network phases

Current Gap

OBI generates flat HTTP/gRPC spans that cover the entire request lifecycle. Network-layer
timing exists in fragments (DNS events, TCP RTT metrics, SSL uprobe data) but is not
correlated to the parent request span — making root-cause analysis a manual, multi-tool
exercise.

3. Current State

OBI already captures several network-layer signals, but they are disconnected from request
traces:

Signal	Location	Limitation
DNS detection	bpf/generictracer/dns.h	Produces independent events; no parent span linkage
TCP connect entry	bpf/generictracer/k_tracer.c (kprobe/tcp_connect)	Records PID→socket mapping for later correlation, but does not capture connect completion time
TCP RTT at close	bpf/statsolly/k_tcp.c	Emits k_event_stat_tcp_rtt at connection close — a metric, not per-request, not a span
Network flows	bpf/netolly/flows.c (TC classifier)	Captures packet-level flow metadata; no application-layer trace context
SSL read/write	bpf/generictracer/libssl.c	Instruments SSL_read/SSL_write for data extraction; does not time handshake
Trace context propagation	HTTP header parsing + TCP Option kind=25	Complete — parent context is available at probe time
SpanLink infrastructure	pkg/ebpf/common/ + pkg/export/otel/tracesgen/	Used for Go channel tracking; reusable for network event correlation

Key insight: The eBPF probes and userspace infrastructure are largely in place. The
missing piece is timing correlation — capturing start/end timestamps for each network
phase and associating them with the owning HTTP/gRPC span.

4. Proposed Design

4.1 Target Span Hierarchy

HTTP/gRPC Request Span (existing parent)
│
├── dns.lookup (child span)
│     start: query sent    end: response received
│     Attributes:
│       dns.question.name
│       dns.response_code
│       network.peer.address (resolved IP)
│       network.transport = "udp" | "tcp"
│
├── tcp.connect (child span)
│     start: connect() entry    end: connection established
│     Attributes:
│       network.peer.address
│       network.peer.port
│       network.transport = "tcp"
│
├── tls.handshake (child span, when TLS is used)
│     start: SSL_do_handshake entry    end: handshake complete
│     Attributes:
│       tls.protocol.version
│       tls.cipher_suite (if extractable)
│
└── [application processing — remainder of existing span body]

For connection-pooled requests (where TCP/TLS are already established), only the
application-processing portion remains — no sub-spans are generated for reused connections.

4.2 eBPF Layer Changes

TCP Connect Timing

The existing kprobe/tcp_connect in k_tracer.c already fires at connection initiation.
To capture completion:

• Option A: Add kretprobe/inet_csk_complete_hashdance or instrument socket state
  transition to TCP_ESTABLISHED via tracepoint/sock/inet_sock_set_state.
• Option B: Add kprobe/tcp_finish_connect (called when SYN-ACK is received).

Store {pid_tgid, sock*} → start_ns in a new BPF map at kprobe/tcp_connect entry;
look it up and emit a ring buffer event with (start_ns, end_ns) at completion.

// New map: track connect start time per socket
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 4096);
    __type(key, u64);        // sock pointer
    __type(value, u64);      // start timestamp (ktime_ns)
    __uint(pinning, OBI_PIN_INTERNAL);
} tcp_connect_start SEC(".maps");

DNS Correlation

The existing DNS parser in dns.h already extracts query/response data. Extend it to:

1. Capture bpf_ktime_get_ns() at DNS query send and response receive.
2. Store {pid, dns_transaction_id} → (trace_id, span_id, start_ns) so that userspace
   can construct a child span with the correct parent.

This requires a small map addition (bounded by max concurrent DNS queries per host).

TLS Handshake Timing

Add uprobes for SSL_do_handshake (or SSL_connect for client-side):

SEC("uprobe/libssl.so:SSL_do_handshake")
int BPF_UPROBE(obi_uprobe_ssl_do_handshake, void *ssl) { ... }

SEC("uretprobe/libssl.so:SSL_do_handshake")
int BPF_URETPROBE(obi_uretprobe_ssl_do_handshake, int ret) { ... }

Store entry timestamp keyed by (pid_tgid, ssl*), emit timing event on return.

New Ring Buffer Event Types

enum {
    k_event_tcp_connect = ...,   // TCP connect start + end timestamps
    k_event_tls_handshake = ..., // TLS handshake start + end timestamps
    k_event_dns_correlated = ..., // DNS query/response with parent trace context
};

4.3 Userspace Changes

Event Processing (pkg/internal/ebpf/generictracer/)

• Register handlers for the new event types from the ring buffer.
• Store pending network events in a short-lived in-memory buffer keyed by
  (pid, socket/connection).

Correlation Logic (pkg/ebpf/common/ or new package)

• When an HTTP/gRPC span completes, look up any pending DNS/TCP/TLS events that:
  • Share the same PID
  • Occurred within the span's time window
  • Match the destination address/port
• Attach matched events as child spans (preferred) or SpanLinks.

Export (pkg/export/otel/tracesgen/)

• Extend tracesgen.go to emit child spans for correlated network events.
• Reuse existing appendSpanLinks infrastructure as a fallback relationship model.

4.4 Semantic Conventions

All attributes follow OpenTelemetry semantic conventions:

• DNS: dns.question.name, dns.response_code — DNS Registry
• Network: network.peer.address, network.peer.port, network.transport — Network Registry
• TLS: tls.protocol.version, tls.cipher_suite — TLS Registry

Span names follow the {protocol}.{operation} pattern: dns.lookup, tcp.connect,
tls.handshake.

5. Implementation Phases

Phase 1: DNS Sub-Span Correlation (Low risk, high value)

Scope: Link existing DNS events to HTTP parent spans via PID + time window matching.

Changes:

• Minimal eBPF modification: add timestamp and PID context to DNS events
• Primarily Go-layer correlation logic in pkg/ebpf/common/
• New child span emission in tracesgen.go

Effort: ~1 week
Risk: Low — DNS detection already works; this is correlation-only.

Phase 2: TCP Connect Sub-Span (Medium risk, high value)

Scope: Capture connection establishment duration as a child span.

Changes:

• New kprobe/tracepoint for connect completion
• New BPF map (tcp_connect_start) for socket → timestamp correlation
• New ring buffer event type
• Userspace handler and correlation logic

Effort: ~2 weeks
Risk: Medium — requires careful handling of connect timeout/failure paths and map
cleanup.

Phase 3: TLS Handshake Sub-Span (Medium risk, medium value)

Scope: Time SSL_do_handshake / SSL_connect as a child span.

Changes:

• New uprobe/uretprobe pair on SSL_do_handshake
• Timestamp storage map keyed by (pid_tgid, ssl*)
• Ring buffer event emission on handshake completion

Effort: ~2 weeks
Risk: Medium — library function naming varies across TLS implementations (OpenSSL,
BoringSSL, LibreSSL). May need multiple probe targets.

Phase 4: Unified Network Latency View (Higher risk)

Scope: Aggregate all sub-spans into a coherent network breakdown with configuration
controls.

Changes:

• Configuration schema for enabling/disabling individual phases
• End-to-end integration tests
• Documentation and examples

Effort: ~1 week
Risk: Higher — integration complexity across all phases; performance tuning.

6. Configuration

network_spans:
  enabled: true          # Master switch for all network sub-spans
  dns: true              # Emit dns.lookup child spans
  tcp_connect: true      # Emit tcp.connect child spans
  tls_handshake: true    # Emit tls.handshake child spans

When enabled: false, no additional eBPF probes are attached and no correlation logic
runs — zero overhead.

Individual toggles allow operators to selectively enable phases based on their
observability needs and performance budget.

7. Performance Considerations

Factor	Impact	Mitigation
Additional kprobes/uprobes	~50–100 ns per probe hit	Feature is opt-in; probes only attached when enabled
New BPF maps	Memory proportional to max concurrent connections	Bounded max_entries; aggressive cleanup on timeout
Userspace correlation	CPU proportional to connection rate	Time-window-based eviction; bounded buffer size
Ring buffer throughput	Additional events per connection	Only fires on new connections, not pooled reuse
High-traffic scenarios	Potential event storm during connection bursts	Configurable rate limiting; early-exit for pooled sockets

Connection pooling: For services using persistent connections (HTTP/2, gRPC streams,
database pools), TCP/TLS sub-spans are generated only on initial connection establishment.
Subsequent requests on the same connection skip network sub-span generation entirely.

8. Compatibility

• Minimum kernel: 5.8 (existing OBI requirement) — all proposed kprobes and
  tracepoints (inet_sock_set_state, tcp_connect) are available.
• RHEL 4.18 backport: tcp_finish_connect and inet_sock_set_state tracepoint need
  verification on RHEL8/CentOS8. Fallback: use kretprobe/tcp_v4_connect which is
  available on 4.18.
• No breaking changes: Existing span format is unchanged. New sub-spans are additive.
• Backward compatible: Backends that do not render child spans gracefully ignore them.
• Feature gating: Disabled by default in initial release; opt-in via configuration.

9. Alternatives Considered

A. Metrics-Only Approach

Extend existing TCP RTT metrics with per-request granularity and add DNS/TLS timing
metrics.

Rejected: Metrics lack trace correlation. Users cannot pinpoint which specific
request suffered from DNS delay without switching to a separate metrics view and
manually correlating timestamps.

B. SpanLinks Instead of Parent-Child

Use SpanLinks to loosely associate network events with HTTP spans, avoiding strict
temporal nesting requirements.

Considered as fallback: If strict parent-child proves too complex (e.g., DNS query
spans a different goroutine), SpanLinks provide a viable alternative. The existing
SpanLink infrastructure in OBI makes this low-cost to implement.

C. Userspace-Only Correlation (No New eBPF Probes)

Correlate existing independent DNS/TCP events purely by time proximity in Go code.

Rejected: In high-concurrency scenarios (many parallel connections from the same PID),
time-proximity matching produces false correlations. eBPF-level context (socket pointer,
connection tuple) is essential for accuracy.

D. Integration with netolly Pipeline

Route network sub-span data through the existing netolly TC-based flow pipeline.

Rejected for initial implementation: The netolly pipeline operates at L3/L4 without
application context (no trace IDs). Bridging the two would require significant
architectural changes. Future work could unify them.

10. Open Questions

1. Connection pool handling: Should TCP connect spans be generated only for the first
   connection to a destination, or also for pool expansion events?

2. DNS caching: Should the DNS sub-span be suppressed when the resolution is served
   from OS/library cache (near-zero latency)?

3. Relationship model: Is strict parent-child the right default, or should SpanLinks
   be preferred to avoid issues with concurrent/out-of-order network operations?

4. Pipeline placement: Should this integrate within generictracer (sharing PID
   validation and connection tracking) or be a separate eBPF program with its own
   lifecycle?

5. Span naming: Should span names include the target (e.g., dns.lookup api.example.com)
   or keep them generic (dns.lookup) with the target as an attribute only?

6. Minimum duration threshold: Should sub-spans below a configurable threshold
   (e.g., < 1 ms) be suppressed to reduce noise?

Community feedback on these questions is welcome before implementation begins.

11. References

OBI Source References

• DNS detection: bpf/generictracer/dns.h
• TCP connect probe: bpf/generictracer/k_tracer.c (line 206: kprobe/tcp_connect)
• TCP RTT metric: bpf/statsolly/k_tcp.c
• SSL uprobes: bpf/generictracer/libssl.c
• Network flows: bpf/netolly/flows.c
• SpanLink infrastructure: pkg/ebpf/common/
• Trace export: pkg/export/otel/tracesgen/tracesgen.go
• TCP connection map: bpf/generictracer/maps/tcp_connection_map.h
• SSL connection tracking: bpf/generictracer/ssl_defs.h

OpenTelemetry Specifications

• DNS Semantic Conventions
• Network Semantic Conventions
• TLS Semantic Conventions
• General Span Conventions

Related Work

• netobserv-ebpf-agent — eBPF-based network flow observability (TC classifier approach)
• OpenTelemetry HTTP Client Span Spec — defines the parent span this proposal extends

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response