Skip to content

[RFC - do not merge] custom spans: user-declared spans from USDT probes and function symbols#2552

Draft
mmat11 wants to merge 1 commit into
open-telemetry:mainfrom
coralogix:matt/custom-spans
Draft

[RFC - do not merge] custom spans: user-declared spans from USDT probes and function symbols#2552
mmat11 wants to merge 1 commit into
open-telemetry:mainfrom
coralogix:matt/custom-spans

Conversation

@mmat11

@mmat11 mmat11 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

DEMO - Not intended to merge as-is.
This PR is a demo / discussion driver for a feature. I'm opening it to gather feedback on the shape of the config, the UX and next steps.

What this adds

A new top-level custom_spans config block that lets users declare
otel spans without touching application code or writing native
instrumentation. Four target shapes:

  • function_span: "sym" — entry uprobe + return-site uprobes; span =
    entry-to-return.
  • function_noret: "sym" — attach an entry uprobe at an ELF symbol.
  • usdt_span: "provider:base" — attach at <base>_start and <base>_end;
    emit a span whose duration is the interval between them.
  • usdt_noret: "provider:probe" — attach one uprobe at a USDT marker;
    each fire emits a zero-duration span.

For Go function_span targets, OBI also auto-extracts every scalar
and string argument (including primitive fields of *Request-style
struct pointers) by walking the runtime type metadata — no attrs:
block needed. Works on stripped binaries.

ebpf:
  custom_spans:
    ttl: 1m
    spans:
      - name: order.place
        on: { function_span: "main.(*Order).Place" }

What needs to be reviewed

Mostly devdocs/custom-spans.md.

Specifically I'd like feedback on:

  • Config structure — is the current shape ok?
  • Feature set — is what's here enough? Should we add sampling,
    rate limiting, service selectors, ...?
  • Language coverage — start with Go and add more languages in
    follow-up PRs?

Additional features / ideas welcome.

Example

This test was done on https://opentelemetry.io/docs/demo/services/checkout/ - https://github.com/open-telemetry/opentelemetry-demo/tree/main/src/checkout

custom_spans:
    ttl: 30s
    spans:
      - name: checkout.place_order
        on:
          function_span: "main.(*checkout).PlaceOrder"
      - name: checkout.charge_card
        on:
          function_span: "main.(*checkout).chargeCard"
      - name: checkout.ship_order
        on:
          function_span: "main.(*checkout).shipOrder"
      - name: checkout.send_confirmation
        on:
          function_span: "main.(*checkout).sendOrderConfirmation"
      - name: checkout.prep_order_items
        on:
          function_span: "main.(*checkout).prepOrderItems"
Screenshot 2026-06-29 at 19 55 17

place_order span (https://github.com/open-telemetry/opentelemetry-demo/blob/main/src/checkout/genproto/oteldemo/demo.pb.go#L1456):

Screenshot 2026-06-30 at 13 47 09

Adds a new custom_spans config block letting users declare OpenTelemetry
spans without touching application code — point OBI at a USDT marker
(one-shot or paired _start/_end) or an ELF symbol (entry-only or entry +
uretprobe) and get matching OTLP spans out. Includes:

- config surface (pkg/config/custom_span.go) with a validator that
  rejects mixed target shapes, duplicate names, duplicate probes;
- BPF machinery (bpf/generictracer/custom_span.c) with one emit body
  shared by four programs (start/end/event/func_ret), per-attach cookie
  spec lookup on kernel ≥ 5.15 with an (pid, ns, ip) map fallback on
  older kernels including RHEL 8 / 4.18;
- match-value filter applied in BPF so high-frequency probes like
  python:function__entry only surface events of interest;
- cross-tracer dispatch: gotracer routes EVENT_CUSTOM_SPAN records back
  to generictracer via EBPFEventContext.CustomSpanHandler so Go
  binaries emit custom spans without duplicating the tracer machinery;
- Go function_span pairing on the goroutine pointer (r14 on amd64 /
  x28 on arm64) so pairs survive the scheduler moving a goroutine
  between OS threads across a blocking call;
- automatic argument extraction for Go function-mode targets: walks
  the runtime type tables (.typelink → _type → funcType) via a new
  pkg/internal/gometa package and emits one BPF spec entry per
  primitive argument, including primitive fields reachable through a
  struct-pointer argument. Works on stripped binaries built with
  -ldflags="-s -w". A manual attrs: block can layer on top, overriding
  auto slots on collision;
- integration coverage for C, C++ (folly SDT), Rust, Go (salp),
  Python (python-stapsdt), Ruby (ruby-stapsdt), Java (JNI over
  libstapsdt), Node.js (Node-API over libstapsdt); per-language recipes
  and cheat sheet in devdocs/custom-spans.md.
@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.09963% with 580 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.57%. Comparing base (3ba13ca) to head (706f480).
⚠️ Report is 169 commits behind head on main.

Files with missing lines Patch % Lines
pkg/ebpf/instrumenter.go 54.88% 92 Missing and 28 partials ⚠️
pkg/internal/ebpf/generictracer/generictracer.go 72.93% 45 Missing and 14 partials ⚠️
pkg/ebpf/custom_span.go 69.83% 44 Missing and 10 partials ⚠️
pkg/internal/gometa/recipe.go 66.19% 38 Missing and 10 partials ⚠️
pkg/internal/gometa/walker.go 66.94% 30 Missing and 9 partials ⚠️
.../internal/ebpf/generictracer/custom_span_reader.go 79.23% 33 Missing and 5 partials ⚠️
pkg/internal/gometa/method.go 54.87% 26 Missing and 11 partials ⚠️
pkg/internal/gometa/type.go 62.36% 28 Missing and 7 partials ⚠️
pkg/internal/gometa/field.go 57.33% 23 Missing and 9 partials ⚠️
pkg/ebpf/auto_attrs.go 77.34% 23 Missing and 6 partials ⚠️
... and 8 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2552      +/-   ##
==========================================
+ Coverage   69.23%   69.57%   +0.34%     
==========================================
  Files         350      362      +12     
  Lines       47637    49374    +1737     
==========================================
+ Hits        32980    34354    +1374     
- Misses      12577    12835     +258     
- Partials     2080     2185     +105     
Flag Coverage Δ
integration-test 50.52% <42.84%> (-0.40%) ⬇️
integration-test-arm 29.35% <40.80%> (+2.43%) ⬆️
integration-test-vm-5.15-lts 29.72% <42.28%> (+0.38%) ⬆️
integration-test-vm-6.18-lts 30.23% <42.28%> (+2.30%) ⬆️
k8s-integration-test 34.51% <1.76%> (-1.13%) ⬇️
oats-test 33.95% <1.76%> (-1.77%) ⬇️
unittests 63.14% <49.46%> (-0.54%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@grcevski

grcevski commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Super excited for this @mmat11 !!! 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants