speech-to-speech

Overview

A low-latency, fully modular voice-agent pipeline (VAD->STT->LLM->TTS) that exposes an OpenAI Realtime-compatible WebSocket API. Every component is swappable, supporting local and hosted models. Ideal for building interactive voice applications.

README Preview

\n   \n  \n\n# Speech To Speech: Build voice agents with open-source models\n\n[](https://pypi.org/project/speech-to-speech/)\n[](https://pypi.org/project/speech-to-speech/)\n[](./LICENSE)\n\n\n\nA low-latency, fully modular voice-agent pipeline: **VAD -> STT -> LLM -> TTS**, exposed through an **OpenAI Realtime-compatible WebSocket API**. Every component is swappable. The LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at [HF Inference Providers](https://huggingface.co/inference-providers), or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack.\n\nThis pipeline runs in production as the conversation backend for thousands of [Reachy Mini](https://huggingface.co/blog/reachy-mini) robots.\n\n\n  \n    \n    \n    \n  \n\n\n## Quickstart\n\n```bash\npip install speech-to-speech\nexport OPENAI_API_KEY=...\nspeech-to-speech\n```\n\nThis starts an OpenAI Realtime-compatible server at `ws://localhost:8765/v1/realtime` using Parakeet TDT for local STT, an OpenAI-compatible LLM, and Qwen3-TTS for local speech output.\n\nFrom a source checkout, talk to it from a second terminal:\n\n```bash\npython scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765\n```\n\nPrefer to keep the LLM on your own machine? Serve Gemma 4 with llama.cpp:\n\n```bash\nllama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full\n```\n\nThen point the OpenAI-compatible LLM backend at it:\n\n```bash\nspeech-to-speech \\n    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \\n    --responses_api_base_url "http://127.0.0.1:8080/v1" \\n    --responses_api_api_key ""\n```\n\nAny OpenAI Realtime-compatible client can connect. See [Realtime API](#realtime-api) for the protocol and [LLM backends](#llm-backends) for provider and local-server options.\n\n## Index\n\n* [How it works](#how-it-works)\n* [Installation](#installation)\n* [Supported components](#supported-components)\n* [Run modes](#run-modes)\n* [R