[RFC - do not merge] custom spans: user-declared spans from USDT probes and function symbols#2552
Draft
mmat11 wants to merge 1 commit into
Draft
[RFC - do not merge] custom spans: user-declared spans from USDT probes and function symbols#2552mmat11 wants to merge 1 commit into
mmat11 wants to merge 1 commit into
Conversation
Adds a new custom_spans config block letting users declare OpenTelemetry spans without touching application code — point OBI at a USDT marker (one-shot or paired _start/_end) or an ELF symbol (entry-only or entry + uretprobe) and get matching OTLP spans out. Includes: - config surface (pkg/config/custom_span.go) with a validator that rejects mixed target shapes, duplicate names, duplicate probes; - BPF machinery (bpf/generictracer/custom_span.c) with one emit body shared by four programs (start/end/event/func_ret), per-attach cookie spec lookup on kernel ≥ 5.15 with an (pid, ns, ip) map fallback on older kernels including RHEL 8 / 4.18; - match-value filter applied in BPF so high-frequency probes like python:function__entry only surface events of interest; - cross-tracer dispatch: gotracer routes EVENT_CUSTOM_SPAN records back to generictracer via EBPFEventContext.CustomSpanHandler so Go binaries emit custom spans without duplicating the tracer machinery; - Go function_span pairing on the goroutine pointer (r14 on amd64 / x28 on arm64) so pairs survive the scheduler moving a goroutine between OS threads across a blocking call; - automatic argument extraction for Go function-mode targets: walks the runtime type tables (.typelink → _type → funcType) via a new pkg/internal/gometa package and emits one BPF spec entry per primitive argument, including primitive fields reachable through a struct-pointer argument. Works on stripped binaries built with -ldflags="-s -w". A manual attrs: block can layer on top, overriding auto slots on collision; - integration coverage for C, C++ (folly SDT), Rust, Go (salp), Python (python-stapsdt), Ruby (ruby-stapsdt), Java (JNI over libstapsdt), Node.js (Node-API over libstapsdt); per-language recipes and cheat sheet in devdocs/custom-spans.md.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2552 +/- ##
==========================================
+ Coverage 69.23% 69.57% +0.34%
==========================================
Files 350 362 +12
Lines 47637 49374 +1737
==========================================
+ Hits 32980 34354 +1374
- Misses 12577 12835 +258
- Partials 2080 2185 +105
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Super excited for this @mmat11 !!! 🥇 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DEMO - Not intended to merge as-is.
This PR is a demo / discussion driver for a feature. I'm opening it to gather feedback on the shape of the config, the UX and next steps.
What this adds
A new top-level
custom_spansconfig block that lets users declareotel spans without touching application code or writing native
instrumentation. Four target shapes:
function_span: "sym"— entry uprobe + return-site uprobes; span =entry-to-return.
function_noret: "sym"— attach an entry uprobe at an ELF symbol.usdt_span: "provider:base"— attach at<base>_startand<base>_end;emit a span whose duration is the interval between them.
usdt_noret: "provider:probe"— attach one uprobe at a USDT marker;each fire emits a zero-duration span.
For Go
function_spantargets, OBI also auto-extracts every scalarand string argument (including primitive fields of
*Request-stylestruct pointers) by walking the runtime type metadata — no
attrs:block needed. Works on stripped binaries.
What needs to be reviewed
Mostly
devdocs/custom-spans.md.Specifically I'd like feedback on:
rate limiting, service selectors, ...?
follow-up PRs?
Additional features / ideas welcome.
Example
This test was done on https://opentelemetry.io/docs/demo/services/checkout/ - https://github.com/open-telemetry/opentelemetry-demo/tree/main/src/checkout
place_orderspan (https://github.com/open-telemetry/opentelemetry-demo/blob/main/src/checkout/genproto/oteldemo/demo.pb.go#L1456):