Allow passing timeout to LLM services (for FAILING, not for retry)

[trycatchal]

Problem Statement

Use Case

We just noticed the new'ish LLMSwitcher and started to use it - awesome addition! So we wanted to have timeouts (~2s or so) trigger an error that we could handle in our code to call upon the LLM Switcher to do its thing, but then we noticed that the only way to do this was to either:

1. Make changes to the private _client member of the LLM processor
2. Add a bunch of code to monitor and actively interrupt the LLM if it takes too long

Request

We'd like to be able to pass the timeout to the LLM service constructor.

Thanks!
-Al

Proposed Solution

We only use OpenAI-like LLMs so I'm not sure how hard this is to do across all LLM services.

But for OpenAI-like's, we also noticed the timeout can have more granular settings because of the underlying HTTPX timeout settings - bonus points if we can even specify those too!

But honestly, even being able to pass a single timeout that will make it fail at any point in the response if there's >timeout seconds of inactivity would be great.

Alternative Solutions

If it's too tricky to make this work for all LLM Services in a standard way, just exposing the underlying client on a per-llm-service basis

Additional Context

No response

Would you be willing to help implement this feature?

• Yes, I'd like to contribute
• No, I'm just suggesting

[lukepayyapilli]

Hi @trycatchal, thanks for the feature request!

I've submitted a PR with a fix: #3529

The Fix

You can now use retry_timeout_secs to enable fail-on-timeout behavior for LLMSwitcher integration:

llm = OpenAILLMService(
    model="gpt-4",
    retry_timeout_secs=2.0,  # Fail after 2 seconds
    retry_on_timeout=False,   # Don't retry, let LLMSwitcher handle it
)

When the timeout is exceeded, asyncio.TimeoutError propagates, which LLMSwitcher can catch to trigger a fallback.

Behavior Matrix

retry_timeout_secs	retry_on_timeout	Behavior
None (default)	False (default)	No timeout
None	True	5.0s timeout + retry
Explicit (e.g., 5.0)	False	Timeout + fail (NEW)
Explicit (e.g., 5.0)	True	Timeout + retry

Design Note

The parameter name retry_timeout_secs is admittedly not ideal since it now controls non-retry behavior too. I considered renaming to timeout_secs but opted for backwards compatibility. The PR includes a note about potential future cleanup to improve the naming.

Let me know if this addresses your use case - thanks!

[trycatchal]

Thanks Luke and thanks Mark! This is perfect! Super stoked to to use it!