Add latency breakdown to UserBotLatencyObserver

[markbackman]

Context

The goal is to measure the user to bot latency and to display the contributing factors to that latency number. The possible contributors are:

• VAD stop_secs
• STT latency
• Turn detection
• LLM latency
• Pre-TTS text aggregation
• TTS latency
• Function call execution time

The sequence of events looks something like this:

We are breaking the measurement down into the following categories:

• User turn secs: This measures from when the user actually stops speaking (VADStoppedSpeakingFrame - VADParams stop_secs) to when the UserStoppedSpeakingFrame occurs. There are two main contributors to this phase: STT latency and turn detection. When an STT latency is available, we display that correspond TTFB value. We do not explicitly display the turn detection latency because it is difficult to characterize, but it is generally going to be the main contributor to the long user_turn_secs values.
• TTFB latencies: STT, LLM, TTS. The LLM and TTS portions directly impact the response time. The STT is a function of more complex logic, which I mention above.
• Text aggregation latency: This is the time to build a sentence from the LLM streamed token. If sentence aggregation is in place, we measure it to help the developer understand the impact.
• Function call latency: Time spent executing tool/function calls between LLM roundtrips. Measured from FunctionCallInProgressFrame to FunctionCallResultFrame, making previously invisible API call time visible in the waterfall.

Note: we do not provide a measurement for the turn result because it does not have a direct impact on response time. The result itself is used by the LLMUserAggregator logic to determine how to respond, when the user stop strategy includes a turn analyzer. The turn result latency is available in a MetricsFrame for those that want access, perhaps for troubleshooting or debugging.

Summary

• Added on_latency_breakdown event to UserBotLatencyObserver with a LatencyBreakdown model providing per-service TTFB, text aggregation, user turn duration, and function call latency metrics
• Added on_first_bot_speech_latency event measuring time from ClientConnectedFrame to first BotStartedSpeakingFrame, with a latency breakdown including per-service metrics. Skipped when user speaks first (only meaningful for greetings).
• Added FunctionCallMetrics tracking to LatencyBreakdown, measuring execution time of each function call via FunctionCallInProgressFrame to FunctionCallResultFrame
• Measures user_turn_secs from actual user silence (VAD timestamp minus stop_secs) to turn release, capturing VAD silence detection, STT finalization, and turn analyzer wait time
• Resets metric accumulators on InterruptionFrame to discard stale metrics from cancelled LLM/TTS cycles
• All breakdown data models are Pydantic BaseModel with start_time (Unix timestamp) and duration_secs fields, enabling .model_dump() serialization and timeline plotting
• Updated example 29 with waterfall-style latency breakdown display including function calls

Testing

uv run pytest tests/test_user_bot_latency_observer.py -v

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.14815% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/pipecat/observers/user_bot_latency_observer.py	98.14%	2 Missing ⚠️

Files with missing lines	Coverage Δ
src/pipecat/observers/user_bot_latency_observer.py	97.67% <98.14%> (+5.67%)	⬆️

... and 15 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aconchillo]

Since this might be common. I'm wondering if we could have a function that we could call that returns us a chronological list of strings. Like:

events = observer.chronological_events()

So people could just iterate and print.

[markbackman]

Good idea!

[markbackman]

Ok, pushing a commit with this function.

[aconchillo]

Approving! This is great. Just added a comment to simplify examples/apps a bit.