speech-to-speech

The error occurs in TTS/chatTTS_handler.py because the streaming code assumes every ChatTTS audio chunk has shape (1, samples) and applies [0]. When ChatTTS returns a flat array with shape (samples,), [0] selects a single numpy.int16 sample, so len(audio_chunk) fails.

Fix the streaming branch by removing the trailing [0]:

# Before
audio_chunk = (audio_chunk * 32768).astype(np.int16)[0]

# After
audio_chunk = (audio_chunk * 32768).astype(np.int16)

To support both (samples,) and (1, samples) chunk shapes, use:

audio_chunk = (audio_chunk * 32768).astype(np.int16).squeeze()

Then rerun the pipeline. The relevant while len(audio_chunk) > self.chunk_size: loop will then receive an array instead of a single integer sample.

Update the speech-to-speech server to the latest version containing the Open WebUI compatibility fixes. The issue occurred because Open WebUI sends chat-history snapshots with IDs such as chat_context_1, deletes previous snapshots on refresh, and may send create_response: false followed by an explicit response.create; older server versions rejected or ignored these events and dropped the WebSocket session.

Steps:
1. Pull the latest speech-to-speech code/release that includes the fixes for Open WebUI Realtime chat-history refresh and create_response=false.
2. Rebuild and restart the server container, for example:

git pull
   docker compose up -d --build

The updated server accepts Open WebUI snapshot IDs, recognizes snapshot deletion events, and avoids generating an automatic duplicate response when audio.input.turn_detection.create_response is false. If you cannot update immediately, use the project's bundled web demo as a temporary workaround.

Upgrade to a build containing the fix for the audio.input.turn_detection.create_response flag. The defect is in RealtimeService._on_transcription_completed: it queues GenerateResponseRequest unconditionally after a nonempty transcript, even when the session was configured with create_response: false. Open WebUI then sends an explicit response.create, causing the same turn to be spoken twice.

When building from source, patch the transcription-completion and direct-audio paths so automatic generation is queued only when the effective create_response setting is enabled:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": {
          "type": "server_vad",
          "interrupt_response": true,
          "create_response": false
        }
      }
    }
  }
}

With the fix, create_response: false must still retain transcription, history, and VAD events, but must not enqueue an automatic response. The server should wait for the client's explicit response.create. Verify with the focused reproduction where dispatching TranscriptionCompletedEvent leaves the prompt queue empty:

assert queue.empty()

Temporary workaround if you cannot upgrade: use only one response mechanism—either leave automatic generation enabled and remove Open WebUI's explicit response.create, or keep create_response: false and prevent the server from auto-queuing the response. Upgrading or applying the patch is the recommended resolution.

Update to a build containing the PR "Preserve Kokoro session voices across language switches" (the fix for issue #659), or apply the source change below.

The overwrite happens in one process call: the client/response voice is assigned to self.voice, but when the language returns to the setup language, _lang_and_voice_for replaces it with self._initial_voice from the CLI.

Required change:

1. Pass the session/response-selected voice through to Kokoro processing instead of relying only on mutable self.voice:

yield from self._process_kokoro(text, language_code, pinned_voice=voice)

2. Make _lang_and_voice_for return that voice before applying CLI or per-language defaults:

def _lang_and_voice_for(..., pinned_voice=None):
    if pinned_voice is not None:
        return language_code, pinned_voice
    # existing language/default-voice resolution

3. Keep the existing CLI fallback: when no response or session voice is provided, continue using self._initial_voice. Resetting to _initial_voice in on_session_end is also correct.

4. On the MLX path, when the resolved voice changes, reload both the pipeline and voice, e.g. self.model._get_pipeline(new_lang_code) followed by load_voice(new_voice).

Verification: run with --tts kokoro --kokoro_lang_code a --kokoro_voice af_heart, set the client session voice to am_puck, send one Spanish reply, then English. The spoken voices should be ef_dora then am_puck, not ef_dora then af_heart.

Temporary workaround without patching: set --kokoro_voice to the same voice the client selects, or avoid switching away from the setup language; different per-session voices cannot be preserved by configuration alone.

The background CPU is caused by aiortc encoding and sending about 50 silent RTP audio packets per second even when the outgoing PCM buffer is empty. Upgrade to a Speech-to-Speech release that includes PR #570, or build/run from that PR if it is not released yet. The fix makes the outbound audio track's recv() wait on an event while the buffer is empty and wakes it when new PCM frames arrive or the track stops. This prevents continuous silent-frame encoding without stopping/recreating the RTCRtpSender and without WebRTC renegotiation; RTP timestamps remain continuous when audio resumes. As a temporary workaround, pause/disable the sender in src/speech_to_speech/api/openai_realtime/webrtc_session.py when the outgoing queue is empty and re-enable it before sending the next frame. replaceTrack(None) is less effective because aiortc still polls the track at 50 Hz, while stopping/recreating the sender sends RTCP BYE and may add restart overhead. If residual CPU remains, change _send_loop_for from 10 ms queue polling to an async await queue.get() style wait.