feat: add distributed tracing support #4745

siavashs · 2025-11-15T12:24:12Z

Add tracing support using otel to the the following components:

api: extract trace and span IDs from request context
provider: mem put
dispatch: split logic and use better naming
inhibit: source and target traces, mutes, etc. drop metrics
silence: query, expire, mutes
notify: add distributed tracing support to stages and all http requests

Note: inhibitor metrics are dropped since we have tracing now and they
are not needed. We have not released any version with these metrics so
we can drop them safely, this is not a breaking change.

This change borrows part of the implementation from #3673
Fixes #3670

Signed-off-by: Dave Henderson [email protected]
Signed-off-by: Siavash Safi [email protected]

TODO list

Demo

Alertmanager receiving alerts from a "sender" and tracing it all the way to an Aggregation Group inside Dispatcher

Note that Sender here is a custom Cloudflare proxy which acts as a limiter, it is included as an example to show distributed tracing in action

Trace of Dispatcher flushing alerts and notifications failing to be sent to PagerDuty

siavashs · 2025-11-15T12:43:17Z

I'll implement the Prometheus notifier changes in a new draft PR based on prometheus/prometheus#16355
So if the above PR got merged then it would be much easier to rebase the tracing changes.

thompson-tomo

Some feedback on attribute naming based on Open telemetry exposure in semconv. Might be worthwhile to add an alerting.platform.name attribute to the spans which stores alert manager.

Is there interest in having these signals defined in the open telemetry signal registry (semantic conventions)?

dispatch/dispatch.go

notify/notify.go

dispatch/dispatch.go

SuperQ · 2025-11-15T14:39:25Z

config/config.go

 	return json.Marshal(result)
 }
+
+// TODO: probably move these into prometheus/common since they're copied from


This is probably a good idea, what do you think @ArthurSens?

I can prepare a PR for common later this week.

I would not be against it :)

api/v2/api.go

dispatch/dispatch.go

notify/notify.go

dispatch/dispatch.go

thompson-tomo · 2025-11-18T02:13:08Z

notify/notify.go

+		trace.WithAttributes(attribute.String("alerting.notify.integration.name", i.name)),
+		trace.WithAttributes(attribute.Int("alerting.alerts.count", len(alerts))),
+		trace.WithSpanKind(trace.SpanKindClient),


Is there an easy way to add additional integration details potential some from https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client-span in particular server.*, note I assume this span represents the outbound call to a notification system hence the client span kind.

otelhttp generates a bunch of spans for each http request so I think it is redundant to add more info to the parent span, unless if we want to disable those and construct only one custom span:

If we are going to also have a http span then perhaps this should be internal.

thompson-tomo · 2025-11-18T02:25:34Z

dispatch/dispatch.go

+			attribute.String("alerting.alert.name", alert.Name()),
+			attribute.String("alerting.alert.fingerprint", alert.Fingerprint().String()),
+		),
+		trace.WithSpanKind(trace.SpanKindConsumer),


check this one

Suggested change

trace.WithSpanKind(trace.SpanKindConsumer),

trace.WithSpanKind(trace.SpanKindInternal),

Producer is here https://github.com/prometheus/alertmanager/pull/4745/files#diff-dfa3fcb58b56a2b2a5e995bf37d4f4cd18e64db2835e51a2b077fdc8ec26e7b4R253

Missed that one, are these outgoing calls as per the test mentioned in #4745 (comment)

No, it is this case:

deferred execution (PRODUCER and CONSUMER spans).

Yes but it still needs to be out going based on other information in the spec. Anyway I have raised open-telemetry/opentelemetry-specification#4758 to get further clarification.

siavashs · 2025-11-21T16:11:02Z

CI failures will be fixed after #4761 is merged.

OGKevin

Some minor ✨comments

provider/provider.go

examples/webhook/echo.go

tracing/tracing.go

ultrotter

Do we have before-and-after benchmark measurements with tracing enabled, and disabled?

ultrotter · 2025-12-05T14:43:58Z

dispatch/dispatch.go

 // notifyFunc is a function that performs notification for the alert
 // with the given fingerprint. It aborts on context cancelation.
-// Returns false iff notifying failed.
+// Returns false if notifying failed.


I don't think that was a typo :)

doc/examples/simple.yml

examples/webhook/echo.go

siavashs · 2025-12-05T15:37:13Z

Do we have before-and-after benchmark measurements with tracing enabled, and disabled?

No, tracing is disabled by default, I have to check how we can enable tracing in benchmarks and then generate a diff.

siavashs · 2025-12-05T15:48:06Z

Just did a quick diff, for silences there is a 6% overhead with tracing rate of 1.0:

goos: darwin
goarch: arm64
pkg: github.com/prometheus/alertmanager/silence
cpu: Apple M3 Pro
                                                                 │ bench-notrace.txt │           bench-trace.txt            │
                                                                 │      sec/op       │    sec/op     vs base                │
Mutes/0_total,_0_matching-12                                            383.6n ± ∞ ¹   387.5n ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/1_total,_1_matching-12                                            916.6n ± ∞ ¹   924.4n ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/100_total,_10_matching-12                                         1.581µ ± ∞ ¹   1.620µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/1000_total,_1_matching-12                                         1.069µ ± ∞ ¹   1.138µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/1000_total,_10_matching-12                                        1.926µ ± ∞ ¹   1.939µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/1000_total,_100_matching-12                                       10.83µ ± ∞ ¹   10.60µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/10000_total,_0_matching-12                                        1.881µ ± ∞ ¹   1.881µ ± ∞ ¹       ~ (p=1.000 n=1) ³
Mutes/10000_total,_10_matching-12                                       1.846µ ± ∞ ¹   1.898µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Mutes/10000_total,_1000_matching-12                                     105.6µ ± ∞ ¹   105.6µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesIncremental/1000_base_silences-12                                  105.2µ ± ∞ ¹   105.2µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesIncremental/3000_base_silences-12                                  104.2µ ± ∞ ¹   104.1µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesIncremental/7000_base_silences-12                                  104.6µ ± ∞ ¹   109.0µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesIncremental/10000_base_silences-12                                 103.1µ ± ∞ ¹   113.0µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Query/100_silences-12                                                   16.43µ ± ∞ ¹   17.36µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Query/1000_silences-12                                                  171.6µ ± ∞ ¹   182.1µ ± ∞ ¹       ~ (p=1.000 n=1) ²
Query/10000_silences-12                                                 2.155m ± ∞ ¹   3.462m ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryParallel/100_silences-12                                           3.749µ ± ∞ ¹   4.416µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryParallel/1000_silences-12                                          29.15µ ± ∞ ¹   37.42µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryParallel/10000_silences-12                                         465.5µ ± ∞ ¹   625.2µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryWithConcurrentAdds/1000_initial_silences,_10%_add_rate-12          109.5µ ± ∞ ¹   100.9µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryWithConcurrentAdds/1000_initial_silences,_1%_add_rate-12           60.16µ ± ∞ ¹   64.85µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryWithConcurrentAdds/1000_initial_silences,_0.1%_add_rate-12         33.90µ ± ∞ ¹   48.51µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryWithConcurrentAdds/10000_initial_silences,_1%_add_rate-12          485.8µ ± ∞ ¹   534.1µ ± ∞ ¹       ~ (p=1.000 n=1) ²
QueryWithConcurrentAdds/10000_initial_silences,_0.1%_add_rate-12        470.9µ ± ∞ ¹   477.7µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesParallel/100_silences-12                                           7.420µ ± ∞ ¹   7.221µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesParallel/1000_silences-12                                          70.76µ ± ∞ ¹   64.93µ ± ∞ ¹       ~ (p=1.000 n=1) ²
MutesParallel/10000_silences-12                                         436.6µ ± ∞ ¹   446.5µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/1000_silences,_0%_expired-12                                         39.93µ ± ∞ ¹   40.24µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/1000_silences,_30%_expired-12                                        58.45µ ± ∞ ¹   57.40µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/1000_silences,_80%_expired-12                                        81.02µ ± ∞ ¹   81.08µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/10000_silences,_0%_expired-12                                        367.2µ ± ∞ ¹   374.6µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/10000_silences,_10%_expired-12                                       423.0µ ± ∞ ¹   444.7µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/10000_silences,_50%_expired-12                                       625.0µ ± ∞ ¹   661.0µ ± ∞ ¹       ~ (p=1.000 n=1) ²
GC/10000_silences,_80%_expired-12                                       756.8µ ± ∞ ¹   764.5µ ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                                 42.69µ         45.34µ        +6.19%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

I think this is acceptable and expected.

Add tracing support using otel to the the following components: - api: extract trace and span IDs from request context - provider: mem put - dispatch: split logic and use better naming - inhibit: source and target traces, mutes, etc. drop metrics - silence: query, expire, mutes - notify: add distributed tracing support to stages and all http requests Note: inhibitor metrics are dropped since we have tracing now and they are not needed. We have not released any version with these metrics so we can drop them safely, this is not a breaking change. This change borrows part of the implementation from prometheus#3673 Fixes prometheus#3670 Signed-off-by: Dave Henderson <[email protected]> Signed-off-by: Siavash Safi <[email protected]>

Adds changes that have been merged after 2025-11-21 into main. The entry prometheus#4629 is removed from the log, since it is superseded by prometheus#4745.

siavashs force-pushed the feat/tracing branch 2 times, most recently from 178f982 to 22bfc0c Compare November 15, 2025 12:34

siavashs added component/notify component/api component/dispatch labels Nov 15, 2025

siavashs self-assigned this Nov 15, 2025

siavashs force-pushed the feat/tracing branch from 22bfc0c to c1d9a3c Compare November 15, 2025 12:37

siavashs mentioned this pull request Nov 15, 2025

Instrument with OTel tracing #3673

Closed

thompson-tomo reviewed Nov 15, 2025

View reviewed changes

SuperQ reviewed Nov 15, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch from c1d9a3c to 7b93382 Compare November 15, 2025 15:39

thompson-tomo reviewed Nov 16, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch 4 times, most recently from 14b79bd to 47747ad Compare November 17, 2025 21:16

thompson-tomo reviewed Nov 18, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch 9 times, most recently from c1482c7 to 5882778 Compare November 21, 2025 16:00

siavashs marked this pull request as ready for review November 21, 2025 16:28

siavashs requested a review from SuperQ November 21, 2025 16:31

siavashs requested review from Spaceman1701, grobinson-grafana, thompson-tomo and ultrotter November 21, 2025 16:32

siavashs force-pushed the feat/tracing branch 3 times, most recently from 6ba9be4 to 37231f9 Compare November 22, 2025 12:17

siavashs mentioned this pull request Nov 23, 2025

Add AlertLifeCycleObserver that allows consumers to hook into Alert life cycle #3461

Open

OGKevin reviewed Nov 25, 2025

View reviewed changes

provider/provider.go Show resolved Hide resolved

examples/webhook/echo.go Outdated Show resolved Hide resolved

tracing/tracing.go Show resolved Hide resolved

siavashs force-pushed the feat/tracing branch 2 times, most recently from bea0dba to b65bdbf Compare November 27, 2025 08:37

siavashs mentioned this pull request Dec 5, 2025

Release v0.30.0-rc.0 #4770

Merged

ultrotter approved these changes Dec 5, 2025

View reviewed changes

Spaceman1701 approved these changes Dec 5, 2025

View reviewed changes

siavashs force-pushed the feat/tracing branch 3 times, most recently from f2706f6 to 87706af Compare December 5, 2025 15:30

siavashs force-pushed the feat/tracing branch from 87706af to 4688a8e Compare December 5, 2025 21:34

SuperQ merged commit 18939ce into prometheus:main Dec 5, 2025
7 checks passed

siavashs deleted the feat/tracing branch December 8, 2025 09:41

This was referenced Dec 10, 2025

Race conditions in the clustering code? #2033

Closed

Proposal: Alert State Analytics for Alertmanager #4732

Open

dongjiang1989 mentioned this pull request Dec 16, 2025

feat: Support add distributed tracing for alertmanager prometheus-operator/prometheus-operator#8181

Open

5 tasks

cxdy mentioned this pull request Dec 26, 2025

fix(bug): memory leak in tracing client #4828

Open

	trace.WithSpanKind(trace.SpanKindConsumer),
	trace.WithSpanKind(trace.SpanKindInternal),

feat: add distributed tracing support #4745

feat: add distributed tracing support #4745

Uh oh!

Conversation

siavashs commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO list

Demo

Alertmanager receiving alerts from a "sender" and tracing it all the way to an Aggregation Group inside Dispatcher

Trace of Dispatcher flushing alerts and notifications failing to be sent to PagerDuty

Uh oh!

siavashs commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thompson-tomo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siavashs Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siavashs commented Nov 21, 2025

Uh oh!

OGKevin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ultrotter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

siavashs commented Dec 5, 2025

Uh oh!

siavashs commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

siavashs commented Nov 15, 2025 •

edited

Loading

siavashs commented Nov 15, 2025 •

edited

Loading

siavashs Nov 18, 2025 •

edited

Loading

siavashs commented Dec 5, 2025 •

edited

Loading