概要
VibeVoice は、Microsoft がオープンソースで公開した最先端の音声 AI モデルファミリーであり、テキスト音声合成（Text-to-Speech）モデルと自動音声認識（ASR）モデルを含みます。長時間音声、複数話者の生成、多言語に対応し、連続音声トークン化や次トークン拡散（next-token diffusion）などの革新的な技術を採用しています。
README プレビュー
\n\n## 🎙️ VibeVoice: Open-Source Frontier Voice AI\n[](https://microsoft.github.io/VibeVoice)\n[](https://huggingface.co/collections/microsoft/vibevoice-68a2ef24a875c44be47b034f)\n[](https://openreview.net/pdf?id=FihSkzyxdv)\n[](https://arxiv.org/pdf/2601.18184)\n[](https://colab.research.google.com/github/microsoft/VibeVoice/blob/main/demo/VibeVoice_colab.ipynb)\n[](https://aka.ms/vibevoice-asr)\n\n[](https://trendshift.io/repositories/15465)\n\n\n\n\n\n\n \n \n\n\n\n\n\n📰 News\n\n\n\n\n2026-03-06: 🚀 VibeVoice ASR is now part of a Transformers release! You can now use our speech recognition model directly through the Hugging Face Transformers library for seamless integration into your projects.\n\n2026-01-21: 📣 We open-sourced VibeVoice-ASR, a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for User-Customized Context. Try it in [Playground](https://aka.ms/vibevoice-asr).\n- ⭐️ VibeVoice-ASR is natively multilingual, supporting over 50 languages — check the [supported languages](docs/vibevoice-asr.md#language-distribution) for details.\n- 🔥 The VibeVoice-ASR [finetuning code](finetuning-asr/README.md) is now available!\n- ⚡️ **vLLM inference** is now supported for faster inference; see [vllm-asr](docs/vibevoice-vllm-asr.md) for more details.\n- 📑 [VibeVoice-ASR Technique Report](https://arxiv.org/pdf/2601.18184) is available.\n\n2025-12-16: 📣 We added experimental speakers to VibeVoice‑Realtime‑0.5B for exploration, including multilingual voices in nine languages (DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 distinct English style voices. [Try it](docs/vibevoice-realtime-0.5b.md#optional-more-experimental-voices). More speaker types will be added over time.\n\n2025-12-03: 📣 We open-sourced VibeVoice‑Realtime‑0.5B, a real‑time text‑to‑speech model that supports streaming text input and robust long-form
FAQ (3)
トラブル対応VibeVoiceモデルを読み込む際に 'TypeError: Object of type dtype is not JSON serializable' を修正する方法
トラブル対応
VibeVoiceモデルを読み込む際に 'TypeError: Object of type dtype is not JSON serializable' を修正する方法このエラーは、Transformers の from_pretrained() メソッドに誤ったパラメータ名が渡されたことが原因で発生します。認識されない dtype パラメータがモデル設定に注入され、torch.dtype 型は標準的な JSON シリアライズをサポートしていません。
修正手順:
1. コード内で VibeVoiceASRForConditionalGeneration.from_pretrained() の呼び出しを見つけます（エラーが発生したケースでは、例えば demo/vibevoice_asr_inference_from_file.py）
2. 渡している dtype=xxx パラメータを、公式に有効な torch_dtype=xxx に変更します
3. （macOS Apple Silicon/MPS ユーザーの場合は任意）torch_dtype=torch.float16 を設定し、macOS での AttributeError を回避するために、XPU 関連のロジックを hasattr(torch.backends, "xpu") チェックでラップします
4. 推論スクリプトを再実行してエラーを解決します
トラブル対応VibeVoice-ASR の Gradio デモリンクが機能しない問題を修正する方法
トラブル対応
VibeVoice-ASR の Gradio デモリンクが機能しない問題を修正する方法公開されている VibeVoice-ASR デモは Gradio の共有リンクでホストされており、時折利用できなくなることがあります。まず、https://aka.ms/vibevoice-asr を再試行するか、再読み込みしてください。安定した環境が必要な場合は、デモをローカルで実行してください:
python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR --share公開リンクが引き続きダウンしている場合は、GitHub issue を報告するか、再オープンしてください。
トラブル対応## Short answer
In VibeVoice-Realtime, a `.pt` file isn't a trained model — it's a **cached voice prompt**: a reference audio clip that has been run through the model's speech tokenizer and saved with `torch.save()`. So you don't "train" it; you **encode** a reference sample. However, note that Microsoft originally shipped only the preset voices and did not provide an official, fully documented voice-creation tool for the realtime model at release, so most approaches come from the community. Check the repo's issues/discussions for the latest status, as this has been evolving.
## Method 1: Create a custom .pt voice prompt (no training)
1. **Prepare a reference audio clip** of the target voice:
- WAV, mono, 24 kHz
- Roughly 5–15 seconds of clean, expressive speech
- No background noise or music
2. **Load the model and processor** in the same way the streaming demo does (`VibeVoiceStreamingForConditionalGenerationInference` and `VibeVoiceStreamingProcessor`).
3. **Encode the reference audio** through the model's acoustic/speech tokenizer to produce the same tensor structure used by the bundled voices. The easiest way to get the format right is to inspect an existing preset:
```python
import torch
preset = torch.load("demo/voices/streaming_model/en-Emma_woman.pt", map_location="cpu")
print(type(preset))
# inspect keys/shapes so your custom file matches exactly
for k, v in (preset.items() if isinstance(preset, dict) else []):
print(k, getattr(v, "shape", v))
```
4. **Save your encoded tensors** with `torch.save()` using the identical key structure, name it e.g. `en-MyVoice_woman.pt`, and drop it into the voices directory. The demo picks it up by filename when passed as `cached_prompt`.
Community members have posted working scripts for this encode step in the VibeVoice GitHub issues/discussions — searching there for "custom voice" will save you reverse-engineering time.
## Method 2: Fine-tune the model (better quality, more effort)
If prompt-based cloning sounds off or unstable, the alternative is fine-tuning:
- Prepare a dataset of the target speaker (audio + transcripts, typically 30 min+ of clean speech).
- Use a community LoRA/fine-tuning script for VibeVoice (several exist on GitHub/Hugging Face).
- After fine-tuning, either use the adapted weights directly or extract a voice prompt from the fine-tuned model.
## Caveats
- The realtime 0.5B model is optimized for preset voices and short-form streaming; zero-shot cloning quality from a custom prompt can be inconsistent.
- The full VibeVoice-1.5B model handles speaker conditioning more flexibly if cloning quality matters more than latency.
- Only clone voices you have rights/consent to use — Microsoft's license notes explicitly prohibit using it to impersonate real people without consent.
If you share which OS/environment you're on, I can help you adapt the encode script steps.
トラブル対応
## Short answer
In VibeVoice-Realtime, a `.pt` file isn't a trained model — it's a **cached voice prompt**: a reference audio clip that has been run through the model's speech tokenizer and saved with `torch.save()`. So you don't "train" it; you **encode** a reference sample. However, note that Microsoft originally shipped only the preset voices and did not provide an official, fully documented voice-creation tool for the realtime model at release, so most approaches come from the community. Check the repo's issues/discussions for the latest status, as this has been evolving.
## Method 1: Create a custom .pt voice prompt (no training)
1. **Prepare a reference audio clip** of the target voice:
- WAV, mono, 24 kHz
- Roughly 5–15 seconds of clean, expressive speech
- No background noise or music
2. **Load the model and processor** in the same way the streaming demo does (`VibeVoiceStreamingForConditionalGenerationInference` and `VibeVoiceStreamingProcessor`).
3. **Encode the reference audio** through the model's acoustic/speech tokenizer to produce the same tensor structure used by the bundled voices. The easiest way to get the format right is to inspect an existing preset:
```python
import torch
preset = torch.load("demo/voices/streaming_model/en-Emma_woman.pt", map_location="cpu")
print(type(preset))
# inspect keys/shapes so your custom file matches exactly
for k, v in (preset.items() if isinstance(preset, dict) else []):
print(k, getattr(v, "shape", v))
```
4. **Save your encoded tensors** with `torch.save()` using the identical key structure, name it e.g. `en-MyVoice_woman.pt`, and drop it into the voices directory. The demo picks it up by filename when passed as `cached_prompt`.
Community members have posted working scripts for this encode step in the VibeVoice GitHub issues/discussions — searching there for "custom voice" will save you reverse-engineering time.
## Method 2: Fine-tune the model (better quality, more effort)
If prompt-based cloning sounds off or unstable, the alternative is fine-tuning:
- Prepare a dataset of the target speaker (audio + transcripts, typically 30 min+ of clean speech).
- Use a community LoRA/fine-tuning script for VibeVoice (several exist on GitHub/Hugging Face).
- After fine-tuning, either use the adapted weights directly or extract a voice prompt from the fine-tuned model.
## Caveats
- The realtime 0.5B model is optimized for preset voices and short-form streaming; zero-shot cloning quality from a custom prompt can be inconsistent.
- The full VibeVoice-1.5B model handles speaker conditioning more flexibly if cloning quality matters more than latency.
- Only clone voices you have rights/consent to use — Microsoft's license notes explicitly prohibit using it to impersonate real people without consent.
If you share which OS/environment you're on, I can help you adapt the encode script steps.VibeVoice-Realtime の .pt ファイルは、埋め込み/キャッシュされたボイスプロンプトプリセットであり、ユーザーがトレーニング可能なモデルウェイトではありません。現在、カスタム Realtime TTS ボイス .pt ファイルを生成するための公開スクリプトやトレーニングパイプラインは存在しません。同梱されているボイスを使用するか、以下のコマンドで実験的なボイスをダウンロードしてください:
bash demo/download_experimental_voices.shカスタムボイスに関する要件については、Realtime ドキュメントを通じて VibeVoice チームにお問い合わせください。このリポジトリの LoRA ファインチューニングコードは VibeVoice-ASR にのみ適用され、Realtime TTS ボイスプリセット .pt ファイルを生成することはできません。