Performance

Inference Optimization

Techniques to maximize LLM inference speed and reduce costs. From quantization to batching, speculative decoding to KV-cache optimization—deploy faster, cheaper, and at scale.

🎯 Why Optimize Inference?

60-80%

Cost Reduction

with proper optimization

3-10x

Faster Inference

tokens per second

4-8x

Less Memory

with quantization

💡 Key Insight: LLM inference is memory-bound, not compute-bound. Optimizations focus on reducing memory footprint and improving memory access patterns.

⚡ Optimization Techniques

Technique	Speedup	Memory Savings	Quality Impact	Complexity
Quantization (4-bit)	1.5-2x	4x	Minor loss	Low
Continuous Batching	2-5x	-	None	Medium
PagedAttention	2-4x	Up to 24x	None	Low (use vLLM)
FlashAttention	2-4x	5-20x	None	Low
Speculative Decoding	2-3x	-	None	Medium
KV-Cache Compression	1.2-1.5x	2-4x	Minimal	High

🔢 Quantization Methods

Reduce model precision from FP16/BF16 to INT8 or INT4 for faster inference and smaller memory footprint.

AWQ (Activation-aware Weight Quantization)

Best for production GPU inference. Preserves quality on important weights.

4-bit vLLM TensorRT-LLM

GPTQ (Post-Training Quantization)

Widely supported. Good balance of speed and quality.

4-bit ExLlama HuggingFace

GGUF (llama.cpp format)

Best for CPU/hybrid inference. Multiple quantization levels (Q4_K_M, Q5_K_M, etc.)

2-8 bit CPU Ollama

FP8 (8-bit Floating Point)

Minimal quality loss. Requires H100/Ada GPUs with native FP8 support.

8-bit H100 TensorRT-LLM

🚀 Inference Engines Comparison

Engine	Best For	Key Features	Performance
vLLM	Production serving	PagedAttention, continuous batching, OpenAI-compatible API	Excellent
TensorRT-LLM	Max NVIDIA perf	Kernel fusion, FP8, inflight batching	Best on NVIDIA
llama.cpp	Local/CPU inference	GGUF, Apple Metal, low resource	Good (CPU)
Ollama	Easy local LLMs	One-command setup, model library	Good
SGLang	Complex pipelines	RadixAttention, structured generation	Excellent
ExLlamaV2	GPTQ inference	Fast 4-bit, speculative decoding	Excellent

💻 Quick Start Examples

vLLM Production Serving with AWQ

# Install: pip install vllm

from vllm import LLM, SamplingParams

# Load AWQ quantized model
llm = LLM(
    model="TheBloke/Llama-2-13B-chat-AWQ",
    quantization="awq",
    dtype="half",
    max_model_len=4096
)

# Generate with batching
prompts = ["Hello, how are you?", "What is AI?"]
outputs = llm.generate(prompts, SamplingParams(
    temperature=0.7,
    max_tokens=256
))

Ollama Local LLM in 2 Commands

# Install Ollama, then:
ollama run llama3.1:8b-instruct-q4_K_M

# Or use the API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b-instruct-q4_K_M",
  "prompt": "What is inference optimization?",
  "stream": false
}'

✅ Optimization Best Practices

Do's

Start with vLLM + AWQ for production (easy wins)
Enable continuous batching for concurrent requests
Use streaming for better user experience
Profile before optimizing (measure tokens/sec)
Set appropriate max_tokens limits

Don'ts

Don't use FP32—always use FP16/BF16 minimum
Avoid very aggressive quantization (Q2) for critical tasks
Don't ignore context window size (costs memory)
Skip benchmarking quality after quantization
Forget to warm up models before benchmarking