Context Missing Few Words , on the AssistantAggregator Level #3627
Description
pipecat version
0.0.100
Python version
3.12.12
Operating System
mac os
Issue description
We have been observing few words are missing from the Context for the Assistant Role , upon deeper analysis most likely it's the LLMAssistantAggregator , the main cause of this behavior is the LLMAssistantAggregator decides whether to add TTS Text Frames based on flag _is_started which turn is set by LLMResponseStartFrame , so in scenarios where a TTSTextFrame arrives before the LLMResponseStartFrame we are missing those words in the context.
refer to this Section Of Code.
We are using ElevenLabs as a TTS service .
Reproduction steps
Hmm I think there is no clear steps to reproduce this , it just happens once in a While , probably Place a FrameLogger Between the transport.output() and the AssistantAggregator , observe the scenarios where the TTSFrame Arrived before the LLMResponseStartFrame, that TTSFrame will not be a part of the context
Expected behavior
Hmm I think an Approach Would be to Buffer that Kind of the Frames and add it to the context , so that word is not missed from the context.
Actual behavior
The Word is Missed From the Context Refer to The Lines Of Logs attached.
Logs
2026-02-02 18:05:07.762 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: MetricsFrame#73
2026-02-02 18:05:07.874 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#112(pts: 0:01:01.210050, text: [That's])
2026-02-02 18:05:08.035 | DEBUG | pipecat.transports.base_output:_bot_started_speaking:608 - Bot started speaking
2026-02-02 18:05:08.036 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: BotStartedSpeakingFrame#16
2026-02-02 18:05:08.037 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: LLMFullResponseStartFrame#7
2026-02-02 18:05:08.038 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: AggregatedTextFrame#18(pts: None, text: [That's great!])
2026-02-02 18:05:08.038 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSStartedFrame#7
2026-02-02 18:05:08.038 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: AggregatedTextFrame#19(pts: None, text: [Would you like me to schedule a consultation call to discuss how we can help handle those calls?])
2026-02-02 18:05:08.039 | DEBUG | pipecat.processors.logger:process_frame:71 - < LLM Output: BotStartedSpeakingFrame#17
2026-02-02 18:05:08.163 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#113(pts: 0:01:01.500050, text: [great!])
2026-02-02 18:05:08.756 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#114(pts: 0:01:02.092050, text: [Would])
2026-02-02 18:05:08.987 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#115(pts: 0:01:02.324050, text: [you])
2026-02-02 18:05:09.093 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#116(pts: 0:01:02.429050, text: [like])
2026-02-02 18:05:09.277 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#117(pts: 0:01:02.614050, text: [me])
2026-02-02 18:05:09.382 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#118(pts: 0:01:02.719050, text: [to])
2026-02-02 18:05:09.521 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#119(pts: 0:01:02.858050, text: [schedule])
2026-02-02 18:05:09.939 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#120(pts: 0:01:03.276050, text: [a])
2026-02-02 18:05:10.010 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#121(pts: 0:01:03.346050, text: [consultation])
2026-02-02 18:05:10.636 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#122(pts: 0:01:03.973050, text: [call])
2026-02-02 18:05:10.915 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#123(pts: 0:01:04.251050, text: [to])
2026-02-02 18:05:11.032 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#124(pts: 0:01:04.368050, text: [discuss])
2026-02-02 18:05:11.415 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#125(pts: 0:01:04.751050, text: [how])
2026-02-02 18:05:11.531 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#126(pts: 0:01:04.867050, text: [we])
2026-02-02 18:05:11.612 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#127(pts: 0:01:04.948050, text: [can])
2026-02-02 18:05:11.751 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#128(pts: 0:01:05.087050, text: [help])
2026-02-02 18:05:11.937 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#129(pts: 0:01:05.273050, text: [handle])
2026-02-02 18:05:12.251 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#130(pts: 0:01:05.587050, text: [those])
2026-02-02 18:05:12.460 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSTextFrame#131(pts: 0:01:05.796050, text: [calls?])
2026-02-02 18:05:12.461 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: LLMFullResponseEndFrame#15
2026-02-02 18:05:13.395 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: TTSStoppedFrame#7
2026-02-02 18:05:13.881 | DEBUG | pipecat.processors.aggregators.llm_response_universal:_on_user_turn_started:582 - LLMUserAggregator#0: User started speaking (strategy: VADUserTurnStartStrategy#0)
2026-02-02 18:05:13.882 | DEBUG | pipecat.utils.tracing.turn_trace_observer:_handle_turn_ended:213 - Ended tracing for Turn 8
2026-02-02 18:05:13.882 | DEBUG | pipecat.utils.tracing.turn_trace_observer:_handle_turn_started:192 - Started tracing for Turn 9
2026-02-02 18:05:13.884 | DEBUG | pipecat.pipeline.task:_source_push_frame:772 - PipelineTask#0: received interruption task frame InterruptionTaskFrame#7
2026-02-02 18:05:13.886 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: VADUserStartedSpeakingFrame#7
2026-02-02 18:05:13.886 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: UserStartedSpeakingFrame#14
2026-02-02 18:05:13.887 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: VADUserStartedSpeakingFrame#7
2026-02-02 18:05:13.887 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: UserStartedSpeakingFrame#14
2026-02-02 18:05:13.891 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: InterruptionFrame#7
2026-02-02 18:05:13.893 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: InterruptionFrame#7
2026-02-02 18:05:13.894 | DEBUG | pipecat.transports.base_output:_bot_stopped_speaking:630 - Bot stopped speaking
2026-02-02 18:05:13.894 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: BotStoppedSpeakingFrame#16
2026-02-02 18:05:13.895 | DEBUG | pipecat.processors.logger:process_frame:71 - < LLM Output: BotStoppedSpeakingFrame#17
2026-02-02 18:05:14.226 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: InterimTranscriptionFrame#9(user: df4a3320-2862-4a65-b9be-f350ee2203e8, text: [She], language: en, timestamp: 2026-02-03T02:05:14.209Z)
2026-02-02 18:05:14.235 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: InterimTranscriptionFrame#9(user: df4a3320-2862-4a65-b9be-f350ee2203e8, text: [She], language: en, timestamp: 2026-02-03T02:05:14.209Z)
2026-02-02 18:05:14.708 | DEBUG | pipecat.transports.daily.transport:_on_transcription_message:2666 - Transcription (from: df4a3320-2862-4a65-b9be-f350ee2203e8): [Sure.]
2026-02-02 18:05:14.709 | INFO | language.detector:process_frame:51 - [Lang-Detect] lang=en, len=5, text="Sure."
2026-02-02 18:05:15.104 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: VADUserStoppedSpeakingFrame#7
2026-02-02 18:05:15.112 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: VADUserStoppedSpeakingFrame#7
2026-02-02 18:05:15.202 | DEBUG | pipecat.audio.turn.smart_turn.base_smart_turn:analyze_end_of_turn:162 - End of Turn result: EndOfTurnState.COMPLETE
2026-02-02 18:05:15.203 | DEBUG | pipecat.processors.logger:process_frame:71 - > LLM Output: MetricsFrame#74
2026-02-02 18:05:15.204 | DEBUG | pipecat.processors.logger:process_frame:71 - > After Transport Output: MetricsFrame#74
2026-02-02 18:05:15.402 | DEBUG | pipecat.processors.aggregators.llm_response_universal:_on_user_turn_stopped:600 - LLMUserAggregator#0: User stopped speaking (strategy: TurnAnalyzerUserTurnStopStrategy#0)
2026-02-02 18:05:15.405 | DEBUG | pipecat.adapters.base_llm_adapter:from_standard_tools:127 - Retrieving the tools using the adapter: <class 'pipecat.adapters.services.open_ai_adapter.OpenAILLMAdapter'>
2026-02-02 18:05:15.406 | DEBUG | pipecat.services.openai.base_llm:_stream_chat_completions_universal_context:337 - GroqLLMService#0: Generating chat from universal context [{'role': 'system', 'content': '## Identity:\nYou Very Long System Prompt.....'}, {'role': 'assistant', 'content': 'how can I help you today?'}, {'role': 'user', 'content': 'Can you hear me?'}, {'role': 'assistant', 'content': 'Yes, I can hear you perfectly! What brings you to BuzzWisely today?'}, {'role': 'user', 'content': 'Well, I would like to know more about your business.'}, {'role': 'assistant', 'content': ' provides AI-powered phone answering services to help businesses never miss a call. We handle incoming'}, {'role': 'user', 'content': 'Mhmm.'}, {'role': 'assistant', 'content': "Basically, we answer your phones when you can't. Would you like"}, {'role': 'user', 'content': 'Got it.'}, {'role': 'assistant', 'content': 'Perfect! Do you have a business that misses calls?'}, {'role': 'user', 'content': 'Yes. We do.'}, {'role': 'assistant', 'content': "That's frustrating! What's your business?"}, {'role': 'user', 'content': 'We manufacture IoT sensors.'}, {'role': 'assistant', 'content': 'Nice! IoT sensors are huge right now. Do you get a lot of customer calls?'}, {'role': 'user', 'content': 'Yes. We do.'}, {'role': 'assistant', 'content': 'great! Would you like me to schedule a consultation call to discuss how we can help handle those calls?'}, {'role': 'user', 'content': 'Sure.'}]