Production Inference

vLLM

The industry standard for high-throughput LLM serving. Up to 24x faster than HuggingFace Transformers.

What is vLLM?

vLLM is an open-source library for fast LLM inference and serving. It's the backbone of many production AI systems, used by companies like Anyscale, Databricks, and together.ai.

The key innovation is PagedAttention—a memory management technique borrowed from operating systems that enables efficient batching and up to 24x higher throughput.

Key Features

PagedAttention

Near-zero KV cache waste. Enables efficient memory sharing across requests.

Continuous Batching

Dynamic batching of requests. No waiting for batch completion.

Quantization Support

GPTQ, AWQ, FP8 out of the box. Run 70B on consumer GPUs.

OpenAI-Compatible API

Drop-in replacement for OpenAI API. Use existing code.

Quick Start

Installation

pip install vllm

Start OpenAI-Compatible Server

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --port 8000

Use with OpenAI Client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

Production Configuration

python -m vllm.entrypoints.openai.api_server \
    --model TheBloke/Llama-2-70B-AWQ \
    --quantization awq \
    --tensor-parallel-size 2 \        # Split across 2 GPUs
    --max-model-len 4096 \            # Max context length
    --gpu-memory-utilization 0.9 \    # Use 90% of VRAM
    --port 8000

Flag	Description
--tensor-parallel-size	Number of GPUs to split model across
--quantization	awq, gptq, fp8
--max-model-len	Maximum context length
--gpu-memory-utilization	Fraction of VRAM to use (0.9 = 90%)

vLLM vs Alternatives

Tool	Best For	Throughput
vLLM	Production GPU servers, high throughput	Highest
TGI (HuggingFace)	Enterprise, Docker-first deployment	High
Ollama	Local development, macOS/CPU support	Medium
llama.cpp	CPU inference, edge devices	Low (CPU)