Should UserIdleController respect user mute and user turn stopped turn ?

[kedar389]

pipecat version

0.0.101

Python version

No response

Operating System

No response

Question

Currently, the UserIdleController only responds to UserSpeakingFrame and BotSpeakingFrame. There is also special handling for FunctionCallsStartedFrame to prevent timeouts while waiting for the LLM. However, this feels like a band-aid solution, as there are gaps between these frames. For example, network latency or CPU overload could cause the UserIdleController to trigger a timeout unexpectedly.

I think that once we detect that a user turn has stopped, we should no longer expect user speech. This should also apply when a user is muted.

Should UserIdleController stop waiting for user speech once a user turn ends or the user is muted?

What I've tried

No response

Context

I encountered this issue while trying to implement a DTMF authentication processor. Consider the following scenario:
A user is muted from the start. They remain muted until they are authenticated via a set of DTMF tones. This is collected in custom Frame processor.

In the current version of Pipecat: This workflow is not possible without encountering the bugs described above, or without using non-standard approaches.

Normally, DTMF tones from DTMF aggregator would be filtered because of mute, I created a custom DTMF aggregator to handle this, but UserIdleController still triggers even when the user is muted because it does not register any user activity but also it does not know about user's mute.

Regardless of what i was trying to achieve I’d love to hear your thoughts on this matter.

[markbackman]

Thanks for the context. It sounds like you have different phases of the conversation: times where the user doesn't talk and times where they do. It sounds like you need a way to dynamically change the user_idle_timeout value to reflect those different phases. That seems like a logical addition for Pipecat.

Also, question: how are you muting the user?

[kedar389]

Through custom mute strategy with custom frame that is emitted when user is authenticated, as that is the standard way to go about user muting (from what i understood). I think user idle timeout should not change dynamically at least not because of this case, nor i would like to do that for now as that results in behavioral changes of the pipeline and makes the code more complex. Also I think changing user idle timeout does not address the issues of User Idle Controller that I mentioned and would just be another band aid fix, even though someone could benefit from such feature (for example when the changes would be based on some kind of expectations from user answer).

[markbackman]

I see what you're saying. After thinking about this, I think you're right. It would be easier to trigger a timeout after solely the BotStoppedSpeakingFrame, waiting for speaking activity. That should eliminate and waiting.

I think this works out for the most part today because user to bot latency is low and user idle timeouts are long. But, if the LLM has a very long response time, it can trigger the idle timeout, which is correct per today's logic but wrong in the spirit of what the user idle controller is trying to provide.

I'll look into improvements in the code.

Separately, I do think an adjustable timeout is the right thing. There are definitely cases where you want to adjust the behavior of the user idle logic. I'll look into adding that.

Regarding the user muting, for both the STTMuteFilter and user_mute_strategies in the LLMUserAggregator, it only mutes the user while the bot is talking. Because the user muting is always coincident with the bot speaking, it's irrelevant to the user idle logic as the bot speaking would reset any user idle timeouts. If the user goes a long time without speech input, you should be adjusting the user idle timeout, or writing your own custom user idle frame processor.

[markbackman]

@kedar389 check out the change in #3744 which makes the change we're discussing. This should be more resilient to non-happy path cases.

[kedar389]

"Regarding the user muting, for both the STTMuteFilter and user_mute_strategies in the LLMUserAggregator, it only mutes the user while the bot is talking. Because the user muting is always coincident with the bot speaking, it's irrelevant to the user idle logic as the bot speaking would reset any user idle timeouts. If the user goes a long time without speech input, you should be adjusting the user idle timeout, or writing your own custom user idle frame processor."

I’ve been thinking about this and wondering: when would we want the UserIdle controller to remain active while the user is muted?

Pipecat isn’t just an audio pipeline framework, and I can imagine cases where the user is muted but we’re not expecting speech input from user at all.

In the BaseUserStrategy docs it says:

"Typical heuristics include:

The bot is currently speaking, so user should be muted

A function call or tool execution is in progress

The system is otherwise not ready to accept user input"

The first two are essentially bot-turn markers, as you described. However, “The system is otherwise not ready to accept user input” is exactly the use case I was implementing.

Because of that, it feels like UserMute should be related to UserIdleProcessor. From a naming perspective, I think many people would assume that user idle detection is inactive while the user is muted. Because if user is muted how can we even get input from him ? Unless input counts something beside user's speech. (but that may be just my personal opinion) . That seems like the more intuitive behavior no ?

[kedar389]

I will look into the other PRs and continue discussion about them there. Thank you for the work.

[markbackman]

The system is otherwise not ready to accept user input
I believe this represents the MuteUntilFirstBotCompleteUserMuteStrategy, which is also muting when the bot is speaking. There is no way to use the mute strategies when the bot is not speaking.

Given this, user mute and user idle are not related. User idle is about detecting when the user could speak but isn't (i.e. they're not muted). This is already implemented as expected.

I've added dynamic controls to turn on/off this feature and change the idle timeout. That will help to adjust to varying circumstances.

[kedar389]

I still think its anti logical to the concept of that class. Just because there is set of current implement strategies it does not mean that there are no use cases that could work with.

Thank you for the work.