项目简介
VibeVoice 是微软开源的前沿语音 AI 模型家族，包含文本转语音和自动语音识别模型。支持长音频、多说话人生成及多语言，采用连续语音分词和下一令牌扩散等创新技术。
README 预览
\n\n## 🎙️ VibeVoice: Open-Source Frontier Voice AI\n[](https://microsoft.github.io/VibeVoice)\n[](https://huggingface.co/collections/microsoft/vibevoice-68a2ef24a875c44be47b034f)\n[](https://openreview.net/pdf?id=FihSkzyxdv)\n[](https://arxiv.org/pdf/2601.18184)\n[](https://colab.research.google.com/github/microsoft/VibeVoice/blob/main/demo/VibeVoice_colab.ipynb)\n[](https://aka.ms/vibevoice-asr)\n\n[](https://trendshift.io/repositories/15465)\n\n\n\n\n\n\n \n \n\n\n\n\n\n📰 News\n\n\n\n\n2026-03-06: 🚀 VibeVoice ASR is now part of a Transformers release! You can now use our speech recognition model directly through the Hugging Face Transformers library for seamless integration into your projects.\n\n2026-01-21: 📣 We open-sourced VibeVoice-ASR, a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for User-Customized Context. Try it in [Playground](https://aka.ms/vibevoice-asr).\n- ⭐️ VibeVoice-ASR is natively multilingual, supporting over 50 languages — check the [supported languages](docs/vibevoice-asr.md#language-distribution) for details.\n- 🔥 The VibeVoice-ASR [finetuning code](finetuning-asr/README.md) is now available!\n- ⚡️ **vLLM inference** is now supported for faster inference; see [vllm-asr](docs/vibevoice-vllm-asr.md) for more details.\n- 📑 [VibeVoice-ASR Technique Report](https://arxiv.org/pdf/2601.18184) is available.\n\n2025-12-16: 📣 We added experimental speakers to VibeVoice‑Realtime‑0.5B for exploration, including multilingual voices in nine languages (DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 distinct English style voices. [Try it](docs/vibevoice-realtime-0.5b.md#optional-more-experimental-voices). More speaker types will be added over time.\n\n2025-12-03: 📣 We open-sourced VibeVoice‑Realtime‑0.5B, a real‑time text‑to‑speech model that supports streaming text input and robust long-form
常见问题 (3)
故障排除如何修复加载 VibeVoice 模型时出现的“TypeError: Object of type dtype is not JSON serializable”错误？
该错误是由于向 Transformers 的 from_pretrained() 方法传递了错误的参数名导致的：无法识别的 dtype 参数被注入到模型配置中，而 torch.dtype 类型不支持标准的 JSON 序列化。
修复步骤：
1. 在代码中找到 VibeVoiceASRForConditionalGeneration.from_pretrained() 调用（例如报错案例中的 demo/vibevoice_asr_inference_from_file.py）
2. 将传入的 dtype=xxx 参数重命名为官方有效的 torch_dtype=xxx
3. （macOS Apple Silicon/MPS 用户可选）设置 torch_dtype=torch.float16，并使用 hasattr(torch.backends, "xpu") 检查来包裹 XPU 相关逻辑，以避免在 macOS 上出现 AttributeError
4. 重新运行推理脚本即可解决该错误
故障排除如何修复 VibeVoice-ASR Gradio 演示链接无法打开的问题？
公开的 VibeVoice-ASR 演示托管在 Gradio 分享链接上，偶尔可能会出现无法访问的情况。首先，请重试或刷新 https://aka.ms/vibevoice-asr。如果你需要稳定的使用环境，可以在本地运行该演示：
python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR --share如果公开链接仍然无法访问，请报告或重新打开该 GitHub issue。
故障排除如何在 VibeVoice-Realtime 中创建或训练自定义音色的 .pt 文件？
VibeVoice-Realtime 的 .pt 文件是嵌入式/缓存的 voice-prompt 预设，而不是用户可训练的模型权重。目前没有公开的脚本或训练流程可用于生成自定义的 Realtime TTS 语音 .pt 文件。请使用内置语音，或通过以下命令下载实验性语音：
bash demo/download_experimental_voices.sh
如有自定义语音需求，请通过 Realtime 文档联系 VibeVoice 团队。该仓库中的 LoRA 微调代码仅适用于 VibeVoice-ASR，无法生成 Realtime TTS 语音预设 .pt 文件。