vllm

vLLM is an open-source, high-throughput and memory-efficient inference and serving engine for large language models, originally developed at UC Berkeley's Sky Computing Lab. Its signature innovation, PagedAttention, manages attention KV cache like virtual memory, dramatically reducing GPU memory waste, while continuous batching, chunked prefill, prefix caching, and speculative decoding push serving throughput to state-of-the-art levels. It supports FP8, INT8, INT4, GPTQ/AWQ and GGUF quantization, tensor/pipeline/expert parallelism for multi-GPU scaling, and 200+ model architectures including Llama, Qwen, DeepSeek, Mixtral, and multimodal models. Deployment is famously simple: one command launches an OpenAI-compatible API server, making it a drop-in backend for existing applications. In independent benchmarks, vLLM typically matches or beats Hugging Face TGI and llama.cpp on throughput while being far easier to set up than NVIDIA TensorRT-LLM, which requires manual engine compilation. Combined with broad hardware support (NVIDIA, AMD, Intel GPUs, TPUs, and CPUs) and a community of 2,000+ contributors, vLLM has become the de facto standard for self-hosted LLM serving in production.