Skip to content

Add Inworld Realtime Service#4140

Merged
filipi87 merged 9 commits intopipecat-ai:mainfrom
cshape:cale/inworld-realtime
Apr 9, 2026
Merged

Add Inworld Realtime Service#4140
filipi87 merged 9 commits intopipecat-ai:mainfrom
cshape:cale/inworld-realtime

Conversation

@cshape
Copy link
Copy Markdown
Contributor

@cshape cshape commented Mar 25, 2026 •
edited
Loading

Add Inworld's Realtime Speech to Speech API as a Pipecat service.

@cshape cshape force-pushed the cale/inworld-realtime branch from c1911c0 to 3ab7e18 Compare March 25, 2026 21:46
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 25, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 49.81504% with 407 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/services/inworld/realtime/llm.py 30.67% 348 Missing ⚠️
...ecat/adapters/services/inworld_realtime_adapter.py 37.80% 51 Missing ⚠️
src/pipecat/services/inworld/realtime/events.py 96.47% 8 Missing ⚠️
Files with missing lines Coverage Δ
src/pipecat/services/inworld/realtime/events.py 96.47% <96.47%> (ø)
...ecat/adapters/services/inworld_realtime_adapter.py 37.80% <37.80%> (ø)
src/pipecat/services/inworld/realtime/llm.py 30.67% <30.67%> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cshape cshape force-pushed the cale/inworld-realtime branch from 03d1054 to 3960c79 Compare March 27, 2026 19:19
@filipi87 filipi87 self-requested a review April 8, 2026 12:14
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread examples/realtime/realtime-inworld.py
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread examples/foundational/19zb-inworld-realtime.py Outdated
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread examples/foundational/19zb-inworld-realtime.py Outdated
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread src/pipecat/services/inworld/realtime/llm.py
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread src/pipecat/adapters/services/inworld_realtime_adapter.py
filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Comment thread src/pipecat/services/inworld/realtime/llm.py Outdated
@filipi87
Copy link
Copy Markdown
Contributor

filipi87 commented Apr 8, 2026

Hi @cshape, when trying to run the example examples/foundational/19zb-inworld-realtime.py, I’m encountering the following error:

2026-04-08 10:22:58.909 | DEBUG    | pipecat.services.inworld.realtime.llm:_send_user_audio:1017 - Streaming audio to Inworld: 24000Hz, 1ch, 960B/frame
2026-04-08 10:22:59.123 | ERROR    | pipecat.processors.frame_processor:push_error_frame:744 - InworldRealtimeLLMService#0 error: Inworld Realtime Error: Failed to read content stream. INTERNAL: HTTP error: 400
2026-04-08 10:22:59.124 | WARNING  | pipecat.pipeline.task:_source_push_frame:903 - PipelineTask#0: Something went wrong: ErrorFrame#0(error: Inworld Realtime Error: Failed to read content stream. INTERNAL: HTTP error: 400, fatal: False)

Could you double check whether everything is working as expected ?

filipi87
filipi87 reviewed Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@filipi87 filipi87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pushing this PR, @cshape.

Overall, it looks good. I’ve left a few comments that we should address.

I noticed that we’re reusing a lot of code from OpenAIRealtime. But the same pattern exists with XAI (Grok), so once this lands, I’ll work on a broader refactor to introduce a shared base class that all three can extend.

For now, it’s probably best to handle that in a follow-up so we don’t block this PR.

Let me know once you’ve addressed the comments so I can review it again 👍.

cshape added 7 commits April 8, 2026 11:04
@cshape
Add Inworld Realtime LLM service
604f37c 
Adds a WebSocket-based realtime service for Inworld's cascade
STT/LLM/TTS API with semantic VAD, function calling, and streaming
transcription support.

New files:
- src/pipecat/services/inworld/realtime/ (service, events)
- src/pipecat/adapters/services/inworld_realtime_adapter.py
- examples/foundational/19zb-inworld-realtime.py

Also includes:
- websockets dependency for inworld extra in pyproject.toml
- Adapter and settings tests matching OpenAI/Grok realtime patterns
- Fix for double-response when server-side VAD is enabled
@cshape
Prefer init-provided system instruction in Inworld Realtime
1ac86b1 
Adopt _resolve_system_instruction() from BaseLLMAdapter, matching the
pattern applied to OpenAI Realtime, Grok Realtime, Gemini Live, and
Nova Sonic in the pk/realtime-services-init-v-context-system-instructions-cleanup
branch.
@cshape
Update changelog entry with PR number
3632d8f
@cshape
Fix changelog format to use bullet point
7eb456b
@cshape
Polish PR: default model, example cleanup, changelog update
89cb1e1 
- Change default model from gpt-4.1-nano to gpt-4.1-mini
- Add function calling demo to example
- Remove demo-testing artifact from system instruction
- Mention Router support in changelog
@cshape
Address PR review feedback for Inworld Realtime
dbb8787 
- Move example to examples/realtime/realtime-inworld.py
- Change initial context role from "user" to "developer"
- Remove explicit sample rates from example; sync them in
  _ensure_audio_config so Inworld gets the transport's actual rates
- Add audio race condition guard in _handle_evt_audio_delta (matches
  OpenAI realtime pattern)
- Convert remaining "system"/"developer" messages to "user" in adapter
- Add clarifying comment for local-VAD vs server-VAD metrics paths
@cshape
Simplify example, add provider tracking, remove local VAD path
3f47a1e 
- Remove function calling from example, switch model to xai/grok-4-1-fast-non-reasoning
- Add pipecat-realtime session key prefix and provider_data metadata
  for Inworld traffic attribution
- Remove local VAD code path (Inworld only supports server-side VAD)
- Use typed InputAudioBufferAppendEvent for audio sends
@cshape cshape force-pushed the cale/inworld-realtime branch from 3960c79 to 3f47a1e Compare April 8, 2026 18:34
filipi87
filipi87 reviewed Apr 9, 2026
View reviewed changes
Comment thread src/pipecat/adapters/services/inworld_realtime_adapter.py Outdated
@cshape
Remove dead shimmed tools code, set STT/VAD defaults
780ca09 
- Remove non-functional AdapterType.SHIM custom tools code from adapter
- Default STT model to assemblyai/u3-rt-pro
- Default VAD eagerness to low
filipi87
filipi87 approved these changes Apr 9, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@filipi87 filipi87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for putting this together, @cshape 🚀

@filipi87 filipi87 merged commit ec574ed into pipecat-ai:main Apr 9, 2026
10 checks passed
@markbackman markbackman mentioned this pull request Apr 9, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@filipi87 filipi87 filipi87 approved these changes

+1 more reviewer

@oleksa-source oleksa-source oleksa-source approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cshape @filipi87 @oleksa-source