huggingface/speech-to-speech Deep Dive: Building Local Voice Agents with Open-Source Models
huggingface/speech-to-speech Deep Dive: Building Local Voice Agents with Open-Source Models
In August 2024, Hugging Face released an unassuming repository: speech-to-speech. Two years later, as of July 30, 2026, it has amassed 8,101 stars, 1,022 forks, and is running in production as the conversation backend for thousands of Reachy Mini robots.
This isn't another voice model. It's a voice agent runtime — a framework that chains VAD (Voice Activity Detection), STT (Speech-to-Text), LLM (Large Language Model), and TTS (Text-to-Speech) into a low-latency pipeline, exposed through an OpenAI Realtime-compatible WebSocket API.
Why does this matter? Because today's production voice AI products — OpenAI's Advanced Voice Mode, Google's Gemini Live — treat their pipelines as a black box. speech-to-speech pries that box open, letting you replace every component independently: STT, LLM, TTS — all swapped for open-source alternatives.
1. Architecture: Four Threads, Four Channels
speech-to-speech's core runs four independent threads connected by queues:
Mic input → [VAD] → [STT] → [LLM] → [TTS] → audio output
↑ ↑ ↑ ↑
Each runs in its own thread, queue-decoupled
| Layer | Component | Default Implementation | Local? |
|---|---|---|---|
| VAD | Voice Activity Detection | Silero VAD v5 | ✅ Local |
| STT | Speech to Text | Parakeet TDT 0.6B | ✅ Local |
| LLM | Language Model | GPT-4.1-mini (OpenAI API) | ⚙️ Swappable to local |
| TTS | Text to Speech | Qwen3-TTS 1.7B | ✅ Local |
In the default configuration, STT and TTS both run locally — only the LLM uses the OpenAI API. But the LLM slot accepts any OpenAI-compatible endpoint: local llama.cpp, local vLLM, Hugging Face Inference Providers, DeepSeek, OpenRouter — you name it.
Four Run Modes
| Mode | Transport | Use When |
|---|---|---|
realtime (default) |
WebSocket, OpenAI Realtime protocol at /v1/realtime |
Building apps against a standard voice API |
local |
Machine mic and speakers | Talking to the pipeline directly in the terminal |
websocket |
Raw PCM over WebSocket | Building a minimal custom client |
socket |
Raw PCM over TCP | Models run on a remote server |
The realtime mode is the real deal — it implements the OpenAI Realtime API's core event set: input_audio_buffer.append, session.update, conversation.item.create, response.create, response.cancel, plus streaming transcription, tool calls, and interruption handling. Any OpenAI Realtime-compatible client connects without modification.
2. Component Matrix: Every Stage Is Swappable
This is where speech-to-speech gets genuinely impressive — it doesn't just offer "multiple choices," it maintains a full swappable backend matrix for every pipeline stage:
STT Backends
| Backend | Platform | Install | Notes |
|---|---|---|---|
| Parakeet TDT (default) | CUDA / CPU / Apple Silicon | Built-in | NVIDIA, 25 European languages |
| Whisper (Transformers) | CUDA / CPU | Built-in | Broad multilingual coverage |
| Faster Whisper | CUDA / CPU | faster-whisper |
CTranslate2 accelerated |
| Lightning Whisper MLX | Apple Silicon | whisper-mlx |
Fastest Whisper on macOS |
| MLX Audio Whisper | Apple Silicon | Built-in on macOS | Hugging Face MLX port |
| Paraformer | CUDA / CPU | paraformer |
Chinese-optimized, via FunASR |
LLM Backends
| Backend | Protocol | When to Use |
|---|---|---|
responses-api (default) |
/v1/responses |
OpenAI, HF Inference Providers, any compatible server |
chat-completions |
/v1/chat/completions |
Fallback when vLLM's Responses streaming tool-call path is unstable |
transformers |
In-process local | CUDA / CPU, load HF models directly |
mlx-lm |
In-process local | Apple Silicon with MLX acceleration |
A key design decision worth noting: chat-completions and responses-api share the same --responses_api_* connection flags, only routing to different endpoints. This isn't redundancy — the authors discovered that certain vLLM builds are unreliable on the Responses streaming tool-call path but solid on Chat Completions (see Issue #312). These two backends coexist not for feature parity but for production fault tolerance.
TTS Backends
| Backend | Platform | Install | Notes |
|---|---|---|---|
| Qwen3-TTS 1.7B (default) | Linux (GGML) / macOS (MLX) | Built-in | Multilingual, multi-voice, streaming |
| Kokoro-82M | CUDA / CPU / Apple Silicon | kokoro |
Lightweight, 82M parameters |
| Pocket TTS | CPU / CUDA | pocket |
Kyutai Labs, voice cloning support |
| ChatTTS | CUDA / CPU | chattts |
English + Chinese conversational style |
| MMS TTS | CUDA / CPU | facebook-mms |
Meta, broad multilingual coverage |
Qwen3-TTS deployment details reveal thoughtful engineering: on Linux it uses GGML (via faster-qwen3-tts), on macOS it auto-switches to mlx-audio. GGML defaults to CUDA 12.8, but the project provides pre-built wheels for CUDA 12.4, 13.x, and CPU. This isn't a simple "pip install" story — hardware coverage at this level determines how many people can actually use the tool.
3. Fully Local: From Zero to Dialogue in Two Commands
The fully local deployment path is speech-to-speech's killer feature. No API keys required:
# Terminal 1: Start llama.cpp serving Gemma 4
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full
# Terminal 2: Start speech-to-speech, point LLM at local llama.cpp
speech-to-speech \
--model_name "ggml-org/gemma-4-E4B-it-GGUF" \
--responses_api_base_url "http://127.0.0.1:8080/v1" \
--responses_api_api_key ""
Production deployment is even cleaner — the project includes a Docker Compose file that spins up llama.cpp + speech-to-speech in one shot.
macOS Optimization: --local_mac_optimal_settings
Apple Silicon users get a one-flag optimal setup:
speech-to-speech --local_mac_optimal_settings
This flag auto-configures four things: MPS acceleration, Parakeet TDT for STT, MLX LM as the LLM backend, and Qwen3-TTS with 6-bit MLX quantization. It even supports comparing quantization levels for TTS performance on macOS:
python scripts/benchmark_tts.py \
--handlers qwen3 \
--iterations 3 \
--qwen3_mlx_quantizations bf16 4bit 6bit 8bit
This level of hardware care is quintessentially Hugging Face — not "we support Apple Silicon," but "we've tuned the optimal defaults for Apple Silicon."
4. Speech-to-Speech vs. VibeVoice: Two Voice AI Philosophies
As of July 2026, the two most-watched open-source voice AI projects are speech-to-speech and Microsoft VibeVoice. But their purposes are fundamentally different:
| Dimension | speech-to-speech | Microsoft VibeVoice |
|---|---|---|
| Category | Voice agent runtime framework | Voice AI model family |
| Core capability | Pipeline orchestration: VAD→STT→LLM→TTS | Standalone models: TTS + ASR |
| Stars (as of 2026-07) | 8,101 | 51,503 |
| Created | 2024-08 | 2025-08 |
| License | Apache 2.0 | MIT |
| Model count | 0 (orchestrates, doesn't train) | 4 (ASR-7B / TTS-1.5B / Realtime-0.5B / BitNet) |
| Max processing length | Limited by LLM context window | ASR 60 min / TTS 90 min |
| Real-time | ✅ Streaming with interruption | TTS streaming (~300ms latency), ASR non-streaming |
| Modularity | ✅ Every component swappable | ❌ Models are fixed |
| Fully local | ✅ Complete local stack | ✅ (ASR-BitNet runs on CPU) |
| Tool calling | ✅ LLM tool calling supported | ❌ Not applicable |
| Production use | Reachy Mini robots | Azure AI Foundry Labs |
The core difference: speech-to-speech answers "how do I assemble existing models into a conversational agent?", while VibeVoice answers "how do I build better voice models?" The former is an orchestration layer, the latter is a model layer.
This also explains VibeVoice's 6× higher star count — models naturally attract more attention than frameworks. But in practice, building a voice agent likely requires both: VibeVoice's ASR/TTS + speech-to-speech's pipeline + your own LLM.
5. Multi-Dimensional Scoring
| Dimension | Score | Rationale |
|---|---|---|
| Architecture | 9/10 | Four-thread decoupling, queue communication, hot-swappable components — clean and practical |
| Hardware coverage | 9/10 | CUDA/CPU/MPS, Linux/macOS, GGML/MLX dual track, multi-version CUDA support |
| Ecosystem compatibility | 9/10 | OpenAI Realtime protocol compatibility, any client connects directly |
| Fully local capability | 10/10 | STT+TTS local by default, LLM swappable to llama.cpp/vLLM |
| Documentation | 7/10 | README is thorough, but architecture docs are scattered across subdirectories |
| Production readiness | 8/10 | Running on robots in production, but 119 open issues suggest edge cases remain |
| Model selection | 8/10 | 3-6 backends per stage, but strongest open-source LLMs not included by default |
| License | 10/10 | Apache 2.0, unrestricted |
Overall: 8.5/10
6. Selection Guide
Choose speech-to-speech if you:
- Need a complete voice dialogue system, not just a single model
- Require swappable components — you have preferences for STT, LLM, and TTS
- Need fully local deployment with no data leaving the machine
- Need an OpenAI Realtime API-compatible interface
- Are building embedded voice interaction for robots, smart homes, etc.
Choose VibeVoice if you:
- Only need a high-quality ASR or TTS model
- Need ultra-long audio processing (60-min ASR / 90-min TTS)
- Need speaker diarization and long-form transcription
- Are in the Azure ecosystem and want Azure AI Foundry integration
Using Both Together
The most practical approach: VibeVoice-ASR for STT + VibeVoice-TTS for TTS + speech-to-speech as the pipeline framework + your LLM. speech-to-speech's modular design makes this entirely feasible — you just need to write a backend adapter for VibeVoice models.
Conclusion
speech-to-speech occupies a unique position in the open-source voice AI landscape: it doesn't train models, but it makes all models work together. This is classic "glue layer" value — harder to replace than any single-point model.
Its three core strengths: modularity (every stage swappable), protocol compatibility (OpenAI Realtime API standard), and fully local capability (one command to switch from cloud to local llama.cpp). No other open-source project currently delivers all three together.
Our recommendation: if you're building a voice agent, speech-to-speech is the best starting point today. It's not the flashiest project (VibeVoice has 6× the stars), but it's the one you're most likely to actually get running and integrate into a product.
References
- huggingface/speech-to-speech GitHub Repository — 8,101 stars, Apache 2.0, created 2024-08
- speech-to-speech README — Full documentation with all backends and configuration
- Realtime Engine README — Complete event reference for OpenAI Realtime protocol
- Issue #312: Chat Completions vs Responses streaming tool calls — vLLM compatibility discussion
- Reachy Mini Robot — Local Conversation Guide — speech-to-speech production case study
- Silero VAD v5 — Default VAD component
- Parakeet TDT 0.6B — NVIDIA default STT model
- Qwen3-TTS — Default TTS model
- llama.cpp — Fully local LLM inference
- Microsoft VibeVoice — Comparison project, 51,503 stars