Merge main: keep upstream tooling & security, add discord_voice, Wyoming, Polly/Edge/Deepgram, fallbacks (#11)

* feat(tts): add Kokoro TTS provider

- Add `kokoro-js` dependency
- Implement `KokoroTTSProvider` in `src/tts.ts` for offline/local TTS
- Update configuration schema in `src/config.ts` to support Kokoro options
- Update documentation (README, SKILL, CHANGELOG) with usage instructions

Co-Authored-By: Claude (gemini-3-pro-low) <noreply@anthropic.com>

* feat(stt): add Local Whisper STT provider

- Add `@xenova/transformers` and `wavefile` dependencies
- Implement `LocalWhisperSTT` in `src/stt.ts` using Transformers.js
- Update configuration in `src/config.ts` to support `local-whisper` options
- Update documentation with usage instructions for offline STT

Co-Authored-By: Claude (gemini-3-pro-low) <noreply@anthropic.com>

* feat: add local whisper STT support and update docs

* refactor(stt): fix race condition in local model init and use wavefile for wav generation

* fix(voice): prevent stream.push() after EOF crash by handling stream lifecycle manually

* feat: add logging interface and type checking script

- Introduced a `StreamingSTTLogger` interface for improved logging capabilities in the streaming STT classes.
- Updated `DeepgramStreamingSTT` and `StreamingSTTManager` to accept a logger instance for better error and info logging.
- Added a new script in `package.json` for TypeScript type checking.

* fix: improve audio handling during streaming STT sessions

- Implemented a mechanism to buffer audio chunks while the WebSocket connection is being established, preventing audio loss during the handshake.
- Introduced a `waitForReady()` method to ensure callers can await the connection before sending audio.
- Updated `sendAudio` to buffer audio instead of dropping it when the connection is not yet ready.

* fix: enhance audio buffering and connection handling during streaming STT

- Improved the audio buffering mechanism to handle scenarios where the WebSocket connection is delayed.
- Ensured that audio chunks are stored until the connection is ready, preventing loss during the initial handshake.
- Updated the `sendAudio` method to utilize the buffering system effectively.

* feat: OpenClaw compatibility, core-bridge, manifest, smoke test

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add support for GPT-4o mini transcription and enhance STT provider options

- Introduced `gpt4o-mini` as a new Speech-to-Text provider for higher quality transcriptions.
- Updated configuration files and documentation to reflect the new STT provider options.
- Enhanced ElevenLabs model ID configuration to support v2, v3, and turbo models.
- Implemented a new `Gpt4oMiniTranscribeSTT` class for handling transcriptions using OpenAI's GPT-4o mini model.

* feat: enhance STT provider options and add thinking sound configuration

- Expanded Speech-to-Text provider options to include `gpt4o-transcribe` and `gpt4o-transcribe-diarize` for improved transcription capabilities.
- Introduced a new `thinkingSound` configuration to play a sound while processing, with customizable options for enabling, file path, and volume.
- Updated documentation and configuration files to reflect these changes, ensuring clarity on new features and usage.

* feat: enhance OpenClaw integration and troubleshooting documentation

- Added a fallback mechanism to resolve the OpenClaw root directory using `require`, improving compatibility with the OpenClaw gateway process.
- Updated README.md to include troubleshooting steps for the "Unable to resolve OpenClaw root" error, detailing how to set the `OPENCLAW_ROOT` environment variable.
- Documented the `OPENCLAW_ROOT` variable in the troubleshooting section for clarity on its usage.

* feat: enhance OpenClaw integration with configurable root path and thinking sound

- Added `openclawRoot` configuration option to specify the OpenClaw package root directory, improving compatibility with the OpenClaw gateway.
- Updated `thinkingSound` documentation in README.md to clarify configuration options for enabling, file path, and volume.
- Refactored core functions to utilize the new `openclawRoot` parameter for loading core dependencies.
- Enhanced JSON schema in plugin manifests to include `openclawRoot` description for better user guidance.

* fix: update README and package.json for troubleshooting and dependency management

- Added troubleshooting steps in README.md for the "Cannot find module structures/ClientUser" error, including commands to reinstall dependencies.
- Updated package.json to include the @sinclair/typebox dependency in the main dependencies section, ensuring it is available for runtime use.

* Phase 0: voice pipeline latency, config safeguards, English docs

- Kokoro: return null from createStreamingTTSProvider (direct batch fallback)
- silenceThresholdMs default 800ms
- thinkingSound.stopDelayMs configurable (default 50ms)
- Validate timeout values (silenceThresholdMs, minAudioMs, etc.) >= 0
- REFACTORING.md: translate to English, update LOC, mark Phase 0 done
- README: stopDelayMs, silenceThresholdMs defaults

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: configurable no-emoji hint for TTS voice prompt

- noEmojiHint: true | false | string
- true (default): inject default 'no emojis' text
- false: do not inject
- string: use custom hint text
- Export DEFAULT_NO_EMOJI_HINT from config

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: TTS fallback provider for quota/rate limit errors

- ttsFallbackProvider config (openai | elevenlabs | kokoro)
- Detect retryable errors: quota_exceeded, 401, 429, 503
- Try fallback when primary fails (streaming + batch)
- Kokoro as free local fallback option
- README: fallback config, Kokoro marked as free

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: convert Kokoro PCM to WAV for Discord playback

Kokoro returns raw PCM; Discord needs a format FFmpeg can decode.
Wrap PCM in WAV header via wavefile before createAudioResource.
Shared createResourceFromTTSResult() for main and fallback flows.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: add kokoro and local-whisper to plugin config schema

- ttsProvider enum: add kokoro (was missing, caused validation error)
- sttProvider enum: add local-whisper
- openclaw.plugin.json + clawdbot.plugin.json

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(kokoro): convert PCM Buffer to Int16Array for wavefile WAV creation

wavefile.fromScratch() expects sample values (-32768..32767), not raw bytes.
Passing a Buffer caused corrupt WAV output and no audio playback.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: session fallback, emoji strip, provider voices, ElevenLabs default

- TTS fallback: stay on fallback provider for rest of session once switched
- Emoji stripping: remove emojis before TTS when noEmojiHint is set (Kokoro)
- Provider-specific voices: openai.voice (nova), elevenlabs.voiceId, kokoro.voice
- OpenAI: validate voice, resolve to nova if invalid
- ElevenLabs: default modelId eleven_turbo_v2_5
- README: docs for new options

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add Deepgram & Polly TTS, ordered fallback providers

- Add DeepgramTTS (REST API, Opus/OGG)
- Add PollyTTS (AWS SDK, MP3)
- Add ttsFallbackProviders array for ordered fallback chain
- Backward compat: ttsFallbackProvider (single) still supported
- Session stores which fallback succeeded for rest of channel

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: use deepgram model for TTS when it's an Aura voice (e.g. aura-2-kara-de)

When model is set to an Aura TTS model (aura-*), use it for TTS instead
of defaulting to aura-asteria-en. STT model stays nova-2.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add Wyoming Faster Whisper STT provider (remote over TCP)

- WyomingWhisperSTT connects to wyoming-faster-whisper server via TCP
- Config: wyomingWhisper.host, port, uri, language, connectTimeoutMs
- Wyoming protocol: transcribe -> audio-start/chunk/stop -> transcript
- Default: 127.0.0.1:10300

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add STT fallback providers (quota, rate limit, Wyoming unreachable)

- sttFallbackProviders array (sttFallbackProvider legacy)
- isRetryableSttError: quota, 401/429/503, ECONNREFUSED, timeout, wyoming
- Session stores fallbackSttProvider on success
- tryTranscribeWithProvider for fallback transcription

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: increase Wyoming Whisper default timeout to 60s

faster-whisper inference can take 15-30s; 10s was too short

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: Wyoming protocol - parse data_length/payload_length correctly

Wyoming sends transcript text in separate data bytes, not in header.
Implement proper Wyoming message parsing (header + data_length + payload_length).
Default timeout back to 10s.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(stt): lazy-load Xenova to prevent OpenClaw crash when local-whisper not used

Top-level import of @xenova/transformers could crash OpenClaw/Electron
on startup. Now Xenova is loaded only when LocalWhisperSTT is actually
used (dynamic import in ensureInitialized).

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(voice): prevent duplicate speech and unheard TTS when agent uses discord_voice speak

- Add spokeViaToolThisRun flag to skip processRecording speak when agent
  already spoke via discord_voice tool
- Inject system prompt: 'do not use discord_voice speak tool; just return
  reply as text' for voice transcript flow
- Stop thinking sound and re-subscribe main player before speak() when
  agent calls speak tool during run (connection was on thinkingPlayer,
  so TTS was not heard)

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(voice): prevent duplicate transcript processing and double speak

- Clear previous silence timer on each 'end' event to avoid multiple
  processRecording calls from duplicate Discord speaking.end events
- Add transcript dedupe: skip same transcript within 5s window
- Use current session from map for spokeViaToolThisRun check (agent
  auto-join may replace session during run)
- Clear silence timer handle when callback fires

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: prevent duplicate plugin init when loaded via both openclaw and clawdbot extensions

Add discordVoiceRegistered singleton guard - skip duplicate register() to avoid
two Discord clients and double voice event processing. Reset on stop/enabled=false.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(stt): Wyoming always retried, other primaries stick to fallback

- When primary is wyoming-whisper: try primary each time (service may be
  temporarily down), never persist fallback
- When primary is other (gpt4o-mini, etc.): stick to fallback for rest
  of session once switched

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: runtime provider/model/think overrides via set-stt, set-tts, set-model, set-think

- Session overrides for STT/TTS provider, model, thinkLevel
- TTS speak() uses effective provider (override or config)
- handleTranscript uses session model/think overrides for agent run
- Gateway: discord-voice.set-stt, set-tts, set-model, set-think, models
- Agent tool: set-stt, set-tts, set-model, set-think actions
- CLI: voice set-stt, set-tts, set-model, set-think
- getAvailableModels() from agents.list/defaults for suggestions
- Status includes model, thinkLevel, availableModels
- README: document new commands

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: add wyomingWhisper and polly.accessKeyId/secretAccessKey to clawdbot schema

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor: use discord_voice prefix for slash commands and CLI to avoid overlap with TTS/voice commands

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: allow additional config properties, add localWhisper and thinkingSound.stopDelayMs to schema

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: exclude .claude/ and REFACTORING.md from repo

Co-authored-by: Cursor <cursoragent@cursor.com>

* Allow additional config properties in plugin schema

Set additionalProperties: true so OpenClaw config validation
accepts config keys not explicitly listed (fixes 'must NOT have
additional properties' error when loading plugin).

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: add hono@4.11.9 to satisfy peer dep and fix npm ci

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Minoa <altgithub@minoa.cat>
Co-authored-by: Claude (gemini-3-pro-low) <noreply@anthropic.com>
Co-authored-by: doxy <doxycomp@gmx.net>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 5 people

CHANGELOG.md

@@ -19,7 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Streaming TTS for real-time audio playback
 - Barge-in support to interrupt bot responses
 - Auto-reconnect with heartbeat monitoring
-- Discord slash commands: `/voice join`, `/voice leave`, `/voice status`
+- Discord slash commands: `/discord_voice join`, `/discord_voice leave`, `/discord_voice status`
 - CLI commands for voice management
 - Agent tool `discord_voice` for programmatic control
 - Configurable VAD sensitivity (low/medium/high)

README.md

@@ -184,21 +184,21 @@ Add these to your bot's OAuth2 URL or configure in Discord Developer Portal.
 
 Once registered with Discord, use these commands:
 
-- `/voice join <channel>` - Join a voice channel
-- `/voice leave` - Leave the current voice channel
-- `/voice status` - Show voice connection status
+- `/discord_voice join <channel>` - Join a voice channel
+- `/discord_voice leave` - Leave the current voice channel
+- `/discord_voice status` - Show voice connection status
 
 ### CLI Commands
 
 ```bash
 # Join a voice channel
-clawdbot voice join <channelId>
+clawdbot discord_voice join <channelId>
 
 # Leave voice
-clawdbot voice leave --guild <guildId>
+clawdbot discord_voice leave --guild <guildId>
 
 # Check status
-clawdbot voice status
+clawdbot discord_voice status
 ```
 
 ### Agent Tool

SKILL.md

@@ -27,32 +27,50 @@
   },
   "configSchema": {
     "type": "object",
-    "additionalProperties": false,
+    "additionalProperties": true,
     "properties": {
-      "enabled": {
-        "type": "boolean",
-        "default": true
-      },
+      "enabled": { "type": "boolean", "default": true },
       "sttProvider": {
         "type": "string",
-        "enum": ["whisper", "gpt4o-mini", "gpt4o-transcribe", "gpt4o-transcribe-diarize", "deepgram", "local-whisper"],
+        "enum": [
+          "whisper",
+          "gpt4o-mini",
+          "gpt4o-transcribe",
+          "gpt4o-transcribe-diarize",
+          "deepgram",
+          "local-whisper",
+          "wyoming-whisper"
+        ],
         "default": "whisper"
       },
+      "sttFallbackProviders": {
+        "type": "array",
+        "items": {
+          "type": "string",
+          "enum": [
+            "whisper",
+            "gpt4o-mini",
+            "gpt4o-transcribe",
+            "gpt4o-transcribe-diarize",
+            "deepgram",
+            "local-whisper",
+            "wyoming-whisper"
+          ]
+        },
+        "description": "Fallback STT when primary fails (quota, rate limit, Wyoming unreachable)"
+      },
       "streamingSTT": {
         "type": "boolean",
         "default": true,
         "description": "Use streaming STT for lower latency (Deepgram only)"
       },
       "ttsProvider": {
         "type": "string",
-        "enum": ["openai", "elevenlabs", "kokoro"],
+        "enum": ["openai", "elevenlabs", "deepgram", "polly", "edge", "kokoro"],
         "default": "openai",
-        "description": "openai, elevenlabs, or kokoro (free local)"
-      },
-      "ttsVoice": {
-        "type": "string",
-        "default": "nova"
+        "description": "openai, elevenlabs, edge (free), or kokoro (free local)"
       },
+      "ttsVoice": { "type": "string", "default": "nova" },
       "vadSensitivity": {
         "type": "string",
         "enum": ["low", "medium", "high"],
@@ -73,42 +91,24 @@
         "default": "off",
         "description": "Thinking level for voice responses (lower = faster)"
       },
-      "allowedUsers": {
-        "type": "array",
-        "items": { "type": "string" },
-        "default": []
-      },
-      "silenceThresholdMs": {
-        "type": "number",
-        "default": 1500
-      },
-      "minAudioMs": {
-        "type": "number",
-        "default": 500
-      },
-      "maxRecordingMs": {
-        "type": "number",
-        "default": 30000
-      },
+      "allowedUsers": { "type": "array", "items": { "type": "string" }, "default": [] },
+      "silenceThresholdMs": { "type": "number", "default": 1500 },
+      "minAudioMs": { "type": "number", "default": 500 },
+      "maxRecordingMs": { "type": "number", "default": 30000 },
       "heartbeatIntervalMs": {
         "type": "number",
         "default": 30000,
         "description": "Connection health check interval in ms"
       },
-      "autoJoinChannel": {
-        "type": "string",
-        "description": "Voice channel ID to auto-join on startup"
-      },
-      "openclawRoot": {
-        "type": "string",
-        "description": "OpenClaw package root if auto-detection fails"
-      },
+      "autoJoinChannel": { "type": "string", "description": "Voice channel ID to auto-join on startup" },
+      "openclawRoot": { "type": "string", "description": "OpenClaw package root if auto-detection fails" },
       "thinkingSound": {
         "type": "object",
         "properties": {
           "enabled": { "type": "boolean", "default": true },
           "path": { "type": "string", "default": "assets/thinking.mp3" },
-          "volume": { "type": "number", "default": 0.7, "minimum": 0, "maximum": 1 }
+          "volume": { "type": "number", "default": 0.7, "minimum": 0, "maximum": 1 },
+          "stopDelayMs": { "type": "number", "default": 50 }
         }
       },
       "openai": {
@@ -124,6 +124,29 @@
           }
         }
       },
+      "polly": {
+        "type": "object",
+        "properties": {
+          "region": { "type": "string", "default": "us-east-1" },
+          "voiceId": { "type": "string", "default": "Joanna" },
+          "engine": { "type": "string", "enum": ["standard", "neural", "long-form", "generative"] },
+          "accessKeyId": { "type": "string" },
+          "secretAccessKey": { "type": "string" }
+        }
+      },
+      "edge": {
+        "type": "object",
+        "properties": {
+          "voice": { "type": "string", "default": "de-DE-KatjaNeural" },
+          "lang": { "type": "string", "default": "de-DE" },
+          "outputFormat": { "type": "string", "default": "webm-24khz-16bit-mono-opus" },
+          "rate": { "type": "string" },
+          "pitch": { "type": "string" },
+          "volume": { "type": "string" },
+          "proxy": { "type": "string" },
+          "timeoutMs": { "type": "number" }
+        }
+      },
       "kokoro": {
         "type": "object",
         "properties": {
@@ -148,8 +171,25 @@
         "type": "object",
         "properties": {
           "apiKey": { "type": "string" },
-          "model": { "type": "string", "default": "nova-2" }
+          "model": { "type": "string", "default": "nova-2" },
+          "ttsModel": { "type": "string", "default": "aura-asteria-en" }
         }
+      },
+      "wyomingWhisper": {
+        "type": "object",
+        "properties": {
+          "host": { "type": "string", "default": "127.0.0.1" },
+          "port": { "type": "number", "default": 10300 },
+          "uri": { "type": "string" },
+          "language": { "type": "string" },
+          "connectTimeoutMs": { "type": "number", "default": 10000 }
+        },
+        "description": "Wyoming Faster Whisper (remote STT over TCP)"
+      },
+      "localWhisper": {
+        "type": "object",
+        "properties": { "model": { "type": "string" }, "quantized": { "type": "boolean" } },
+        "description": "Local Whisper STT (Xenova)"
       }
     }
   },
@@ -164,7 +204,7 @@
     },
     "ttsProvider": {
       "label": "Text-to-Speech Provider",
-      "help": "Use 'openai' or 'elevenlabs'"
+      "help": "openai, elevenlabs, deepgram, polly, edge, or kokoro"
     },
     "ttsVoice": {
       "label": "TTS Voice (deprecated)",
@@ -174,36 +214,15 @@
       "label": "OpenAI TTS Voice",
       "help": "nova, shimmer, echo, onyx, fable, alloy, ash, sage, coral"
     },
-    "kokoro.voice": {
-      "label": "Kokoro TTS Voice",
-      "help": "af_heart, af_bella, af_nicole, etc."
-    },
-    "vadSensitivity": {
-      "label": "VAD Sensitivity",
-      "help": "Voice activity detection sensitivity (low/medium/high)"
-    },
-    "allowedUsers": {
-      "label": "Allowed Users",
-      "help": "Discord user IDs allowed to use voice (empty = all allowed)"
-    },
-    "openai.apiKey": {
-      "label": "OpenAI API Key",
-      "sensitive": true
-    },
-    "elevenlabs.apiKey": {
-      "label": "ElevenLabs API Key",
-      "sensitive": true
-    },
-    "elevenlabs.modelId": {
-      "label": "ElevenLabs Model",
-      "help": "turbo, flash, v2, v3 (or full model ID)"
-    },
+    "kokoro.voice": { "label": "Kokoro TTS Voice", "help": "af_heart, af_bella, af_nicole, etc." },
+    "vadSensitivity": { "label": "VAD Sensitivity", "help": "Voice activity detection sensitivity (low/medium/high)" },
+    "allowedUsers": { "label": "Allowed Users", "help": "Discord user IDs allowed to use voice (empty = all allowed)" },
+    "openai.apiKey": { "label": "OpenAI API Key", "sensitive": true },
+    "elevenlabs.apiKey": { "label": "ElevenLabs API Key", "sensitive": true },
+    "elevenlabs.modelId": { "label": "ElevenLabs Model", "help": "turbo, flash, v2, v3 (or full model ID)" },
     "thinkingSound.enabled": { "label": "Thinking Sound", "help": "Play sound while processing" },
     "thinkingSound.path": { "label": "Thinking Sound File", "help": "Path to MP3" },
     "thinkingSound.volume": { "label": "Thinking Sound Volume", "help": "Volume 0-1" },
-    "deepgram.apiKey": {
-      "label": "Deepgram API Key",
-      "sensitive": true
-    }
+    "deepgram.apiKey": { "label": "Deepgram API Key", "sensitive": true }
   }
 }