agents
Frameworklivekit/agents
Framework for building realtime voice AI agents with speech and vision capabilities.
Overview
Open-source Python framework by LiveKit for creating realtime conversational agents. Supports STT, LLM, TTS integrations, job scheduling, WebRTC clients, telephony, RPC data exchange, and semantic turn detection. Includes multi-agent handoff and built-in testing.
README Preview
\n\n\n \n \n \n\n\n\n\n\n\n[](https://pepy.tech/projects/livekit-agents)\n[](https://livekit.io/join-slack)\n[](https://twitter.com/livekit)\n[](https://deepwiki.com/livekit/agents)\n[](https://github.com/livekit/livekit/blob/master/LICENSE)\n\n\n\nLooking for the JS/TS library? Check out [AgentsJS](https://github.com/livekit/agents-js)\n\n## What is Agents?\n\n\n\nThe Agent Framework is designed for building realtime, programmable participants\nthat run on servers. Use it to create conversational, multi-modal voice\nagents that can see, hear, and understand.\n\n\n\n## Features\n\n- **Flexible integrations**: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.\n- **Integrated job scheduling**: Built-in task scheduling and distribution with [dispatch APIs](https://docs.livekit.io/agents/build/dispatch/) to connect end users to agents.\n- **Extensive WebRTC clients**: Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.\n- **Telephony integration**: Works seamlessly with LiveKit's [telephony stack](https://docs.livekit.io/sip/), allowing your agent to make calls to or receive calls from phones.\n- **Exchange data with clients**: Use [RPCs](https://docs.livekit.io/home/client/data/rpc/) and other [Data APIs](https://docs.livekit.io/home/client/data/) to seamlessly exchange data with clients.\n- **Semantic turn detection**: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.\n- **MCP support**: Native support for MCP. Integrate tools provided by MCP servers with one line of code.\n- **Builtin test framework**: Write tests and use judges to ensure your agent is performing as expected.\n- **Open-source**: Fully open-source, allowing you to run the entire stack on your own servers, including [LiveKit server](https://github.com/livekit/livekit), one of the most widely used WebRTC media servers.\n\n## Installation\n\nTo
FAQ (5)
TroubleshootingHow do I disable LiveKit Agents EOT weight preloading to reduce worker memory usage?
Upgrade to a LiveKit Agents release containing PR #7580, then start the worker with LIVEKIT_AGENTS_PRELOAD_EOT=0. Accepted disabling values are 0, false, no, and off.
Temporary shell start:
export LIVEKIT_AGENTS_PRELOAD_EOT=0
python your_worker.py
For systemd workers, add:
[Service]
Environment=LIVEKIT_AGENTS_PRELOAD_EOT=0
This skips the ~240 MB end-of-turn model preload while keeping the lightweight VAD preload. If a local TurnDetector is actually used later, its weights are loaded lazily on first use. Under Linux forkserver, EOT preloading remains enabled by default. If multiple workers run on one host, using one AgentServer with a larger num_idle_processes can also share a single copy instead of loading it once per worker.
TroubleshootingHow do I fix LiveKit Agents THREAD executor hanging on shutdown with 'job did not ack shutdown in time'?
Upgrade livekit-agents to **1.8.3 or newer**. This shutdown race was fixed by PR #7225; versions including 1.8.1 can hang when idle prewarmed THREAD runners exit before their shutdown acknowledgement is read.
pip install --upgrade 'livekit-agents>=1.8.3'
If using uv:
uv add 'livekit-agents>=1.8.3'
Then restart the worker and send SIGTERM/Ctrl+C after the idle runners are initialized. The worker should shut down promptly without repeated job did not ack shutdown in time errors.
If you cannot upgrade immediately, use JobExecutorType.PROCESS, which does not exhibit this hang. Sending a second signal only force-terminates the process and is not a graceful workaround.
TroubleshootingHow to fix "ResourceWarning: unclosed event loop" when using LiveKit Agents with JobExecutorType.THREAD?
This occurs in affected versions such as livekit-agents==1.8.4: _ProcClient.run() creates an asyncio event loop but does not cancel pending tasks, shut down async generators, reset the thread loop, or close the loop.
**Preferred fix:** upgrade livekit-agents to a release containing the PR fix(ipc): close the job loop when the runner exits (fixes issue #7629).
If you cannot upgrade immediately, patch the finally block in _ProcClient.run() to perform the same shutdown sequence as asyncio.run():
finally:
loop_monitor.stop_monitoring(loop)
try:
pending = [t for t in asyncio.all_tasks(loop) if not t.done()]
for task in pending:
task.cancel()
if pending:
loop.run_until_complete(
asyncio.gather(*pending, return_exceptions=True)
)
loop.run_until_complete(loop.shutdown_asyncgens())
loop.run_until_complete(loop.shutdown_default_executor())
finally:
asyncio.set_event_loop(None)
loop.close()
As a temporary workaround, use process isolation instead of threads:
server = AgentServer(job_executor_type=JobExecutorType.PROCESS)
After the fix, completed job loops should report is_closed() == True, and Python should no longer emit ResourceWarning: unclosed event loop.
TroubleshootingWhy are Muse STT transcripts and end-of-turn events delayed in LiveKit Agents, and how to fix the audio lag?
Root cause: In SpeechStream._send_audio (livekit-plugins-meta), the pacing clock (pacing_origin) starts when the first packet is *sent*, not when the stream starts. Since _run awaits _connect_ws() first, any audio frames queued while the Muse websocket is connecting create a permanent offset — every packet for the rest of the stream is sent late by the connect duration (e.g., 300 ms connect time ≈ 221 ms constant delay), because pacing never sends faster than real time and the backlog is never caught up. This delays transcripts and end-of-turn events by the same amount.
Fix (PR #7890): Start the pacing clock when the stream starts running instead of at the first send. The buffered backlog is flushed immediately once the socket is ready, and subsequent packets are paced against real time as before.
Workaround until the patched release: monkey-patch _send_audio to skip the pacing sleep while _input_ch still has queued frames (only sleep once the buffer is drained). Confirmed affected: livekit-agents==1.8.6, livekit-plugins-meta==1.8.6 with muse-voice-transcribe-1.0.
TroubleshootingWhy does LiveKit Agents invalidate preemptive generation when the final transcript only adds Arabic, CJK, Devanagari, or Urdu punctuation?
Affected versions: livekit-agents 1.8.3 and main @ 57b3227. _transcripts_equivalent() calls split_words(..., ignore_punctuation=True), but the punctuation removal list in livekit/agents/tokenize/_basic_word.py is ASCII-only. Therefore Arabic ، ؛ ؟, Urdu ۔, Devanagari ।, and fullwidth CJK ,。 remain attached to words, causing the final transcript to appear different and triggering preemptive generation invalidated ... because the transcript ... changed.
Fix the tokenizer to remove all Unicode punctuation categories (P*) in addition to the existing ASCII punctuation:
import unicodedata
def _is_punctuation(char: str) -> bool:
return (
char in tokenize.tokenizer.PUNCTUATIONS
or unicodedata.category(char).startswith("P")
)
Use this check in _basic_word.py wherever ignore_punctuation=True filters characters. This preserves symbols such as $ and +, which have category S, and Arabic diacritics, which have category Mn.
Verify with:
from livekit.agents.voice.agent_activity import _transcripts_equivalent
assert _transcripts_equivalent("مرحبا كيف حالك", "مرحبا، كيف حالك؟")
assert _transcripts_equivalent("你好 我很好", "你好,我很好。")
assert _transcripts_equivalent("आप कैसे हैं", "आप कैसे हैं।")
Rebuild or reinstall the patched package. Once the upstream fix is released, upgrade with pip install -U livekit-agents.