Model Serving for LLMs

Serving Frameworks Comparison

Framework	Best For	Throughput	Features
vLLM	High-throughput production	Highest	PagedAttention, continuous batching
TGI	HuggingFace ecosystem	High	Flash attention, quantization
Triton	Multi-model, multi-framework	High	TensorRT, dynamic batching
Ollama	Local development	Medium	Easy setup, macOS support
Ray Serve	Complex pipelines	High	Distributed, model composition
llama.cpp	CPU inference, edge	Low	Minimal deps, GGUF format

Deployment Patterns

Real-time

Low latency, always-on endpoints. Pay for uptime. Best for chat, interactive apps.

Serverless

Scale to zero, pay per request. Cold starts. Best for spiky traffic.

Async/Batch

Queue-based processing. Higher throughput, higher latency. Best for bulk jobs.

Multi-tenant

Shared infrastructure, isolated requests. LoRA adapters for customization.

vLLM Server

Start Server

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --port 8000 \
    --tensor-parallel-size 1

Use with OpenAI Client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Text Generation Inference (TGI)

# Docker deployment
docker run --gpus all -p 8080:80 \
    -e MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
    -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
    ghcr.io/huggingface/text-generation-inference:latest

# Python client
from huggingface_hub import InferenceClient

client = InferenceClient("http://localhost:8080")
response = client.text_generation(
    "What is machine learning?",
    max_new_tokens=200
)
print(response)

Inference Optimization

Technique	Speedup	Trade-off
Continuous Batching	2-5x throughput	Slightly higher latency
PagedAttention (vLLM)	2-4x throughput	None
Flash Attention	2-3x speed	Requires compatible GPU
Quantization (AWQ/GPTQ)	2x throughput, 50% VRAM	Minimal quality loss
Speculative Decoding	1.5-2x speed	Requires draft model
Tensor Parallelism	Near-linear scaling	Multi-GPU overhead

Key Metrics to Monitor

TTFT (Time to First Token)

Latency from request to first token. Target: <500ms for chat.

TPS (Tokens Per Second)

Generation speed. Target: 30-100 tokens/sec for good UX.

Throughput (req/sec)

Requests processed per second. Depends on batch size.

GPU Utilization

Target: 80-95%. Low = underutilized, 100% = bottleneck.

Scaling Strategies

Horizontal Scaling (Replicas)

Add more server instances. Each handles independent requests. Use load balancer to distribute traffic. Kubernetes HPA or cloud auto-scaling.

Vertical Scaling (Bigger GPUs)

Use larger GPU (A100 → H100). More VRAM for larger models or longer contexts. Tensor parallelism across multiple GPUs.

Model Sharding

Split model across multiple GPUs. Tensor parallelism (within node) or pipeline parallelism (across nodes). Required for 70B+ models.

Model Serving