Prometheus/OpenTelemetry Native Export

[Ronitsabhaya75]

Currently, FoundationDB exposes metrics via its JSON status output (fdbcli --exec "status json"). To integrate with modern observability platforms like Prometheus, Grafana, or Datadog, users must:

• Periodically scrape the JSON status endpoint
• Parse the complex nested JSON structure
• Transform metrics into Prometheus format (labels, counters, gauges)
• Run a separate exporter sidecar process

This creates operational overhead, potential data inconsistency, and monitoring gaps between scrapes.

[Ronitsabhaya75]

i was thinking to build a native metrics exporter directly into FoundationDB that exposes metrics in:

• Prometheus format (/metrics HTTP endpoint with OpenMetrics-compatible output)-
• OpenTelemetry Protocol (OTLP) for push-based telemetry

approach:

• Add HTTP server capability (or extend existing internal HTTP endpoints)
• Create metric registry in fdbserver
• Instrument key code paths (transaction processing, storage engine, etc.)
• Expose /metrics endpoint on configurable port

[saintstack]

Would we then be composing and emitting metrics in two forms? Status JSON and then prometheus format on /metrics endpoint? All processes would run an http server? Thanks.

[Ronitsabhaya75]

@saintstack Yes, this would add Prometheus format alongside the existing status JSON they serve different use cases (time-series monitoring vs. point-in-time diagnostics).

The initial scope I was thinking to target was the simulation HTTP infrastructure only.

let me know if this sounds good

[saintstack]

@saintstack Yes, this would add Prometheus format alongside the existing status JSON they serve different use cases (time-series monitoring vs. point-in-time diagnostics).

Thanks. Are you just trying to avoid this 'Run a separate exporter sidecar process'? Its a common-enough pattern, no? Sure, it creates 'operational overhead' but prometheus transforms inside fdb does to, just of a different form. What you thinking for folks who don't want the prometheus view on metrics emission?

Say more on what you are thinking about the http endpoint? How it would work. Currently the fdbserver has the fdb endpoints. GRPC is coming and then there would be HTTP also? Thanks.

[saintstack]

(Oh, might be good to do this discussion up on the forums or over on the discord channel)

[Ronitsabhaya75]

@saintstack Great questions! Let me address each:

Sidecar exporter overhead

Yes, one goal i was thinking was avoiding the operational complexity of running separate Prometheus exporters.
the main implementation should be data consistency and latency

current approaches scrape status json, which requires:

• External parser/transformer
• Potential data staleness between scrape and status generation
• Loss of counter granularity (status JSON aggregates some metrics)
• Native Prometheus export gives direct access to the raw SimpleCounter metrics already being tracked internally.

Optional format for non-Prometheus users

The /metrics endpoint would be purely opt-in - it only outputs data when explicitly scraped. Users who don't use Prometheus are unaffected:

• Status JSON remains unchanged
• No performance impact
• No additional process overhead

HTTP endpoint architecture

For draft PR, I'm targeting simulation infrastructure only seeing if the concept work in infra.

Production deployment is definitely worth deeper discussion:

Which processes expose metrics? (cluster controller? all fdbservers? one per machine?)
Port configuration/security model
How this fits with upcoming gRPC endpoints

lemme know if this make sense.

[gxglass]

@Ronitsabhaya75 serving these metrics lightweight over an in-built HTTP server makes sense to me in principle.

I think a nice bonus would be if you are prepared to test this end to end with a prometheus environment of some kind, if only running locally, as a proof of concept -- that would be great. (We do not have anything handy here nor time to mess with this given other priorities.)

[Ronitsabhaya75]

@gxglass yeah i was doing end to end. for CI i was thinking to add tests which spins github actions.

[johscheuer]

Yes, one goal i was thinking was avoiding the operational complexity of running separate Prometheus exporters.
the main implementation should be data consistency and latency

I think the operational complexity of running an additional prometheus exporter per cluster (or one for multiple clusters) is not that high and a fairly common pattern in Kubernetes deployments. Depending on how you look at those things there could also be benefits in running an "external" exporter.

Potential data staleness between scrape and status generation

Could you explain a bit more what you mean by that? In most production environments you would scrape the prometheus endpoints every 60s, so the status generation shouldn't be a huge issue I guess.

I think one of the biggest benefits of exposing those metrics in a standard format like OTEL is that from an operational perspective you are not blind if the database is in a bad state, e.g. unavailable. In those scenarios you have to use the logging system with the trace events right now because the machine-readable status might not be available or only with partial data.