Infrastructure

Hardware Selection

A comprehensive guide to selecting the right hardware for AI/ML workloads. Compare GPUs, TPUs, CPUs, and specialized accelerators based on your use case, budget, and performance requirements.

🎯 Hardware Decision Framework

Choose hardware based on your primary workload type. Different AI tasks have vastly different requirements.

LLM Inference

Running pre-trained models for text generation

High VRAM Fast Memory Bandwidth Low Latency

Model Training

Training or fine-tuning models from scratch

High TFLOPS Multi-GPU NVLink/Infiniband

Computer Vision

Image classification, detection, segmentation

Tensor Cores Moderate VRAM Batch Processing

Embeddings & RAG

Vector generation and similarity search

CPU OK 8-16GB VRAM High Throughput

⚡ Accelerator Types Comparison

Type	Best For	Pros	Cons	Example
NVIDIA GPU	General AI/ML	Best ecosystem, CUDA	Expensive, power hungry	H100, A100, RTX 4090
Google TPU	Large-scale training	Cost-effective, scalable	GCP only, JAX/TensorFlow	TPU v5e, TPU v4
AMD GPU	Budget training	Lower cost, ROCm growing	Limited ecosystem	MI300X, RX 7900 XTX
Apple Silicon	Local dev, inference	Unified memory, efficient	Mac only, limited training	M3 Max, M3 Ultra
Intel Gaudi	Training at scale	AWS integration, cost	Newer ecosystem	Gaudi 2, Gaudi 3
AWS Inferentia	High-volume inference	Low cost inference	AWS only, limited models	Inferentia2, Trainium

🟢 NVIDIA GPU Selection Guide

💻 Consumer GPUs (Best Value)

RTX 4090

24GB VRAM, 83 TFLOPS. Best bang for buck. Runs 7B-13B models locally.

~$1,600 | Best for: Local inference, fine-tuning small models

RTX 4080

16GB VRAM, 49 TFLOPS. Good for 7B quantized models.

~$1,000 | Best for: Development, smaller LLM inference

RTX 4060 Ti 16GB

16GB VRAM, budget option. Runs 7B Q4 models well.

~$450 | Best for: Entry-level AI development

🏢 Datacenter GPUs (Production)

H100 80GB

80GB HBM3, 990 TFLOPS. Fastest for training and inference. FP8 support.

~$30K or ~$4/hr cloud | Best for: Large model training, high-volume production

A100 80GB

80GB HBM2e, 312 TFLOPS. Industry standard. Widely available in cloud.

~$15K or ~$2-3/hr cloud | Best for: Training, 70B inference

L4 24GB

24GB, 121 TFLOPS. Efficient inference. Great for cost-conscious production.

~$0.70/hr GCP | Best for: Cost-efficient inference at scale

📋 Use Case → Hardware Mapping

Use Case	Min. VRAM	Recommended GPU	Monthly Budget
Local LLM (7B)	8GB	RTX 4060 Ti 16GB	$0 (own hardware)
Local LLM (13B-34B)	24GB	RTX 4090	$0 (own hardware)
Fine-tuning 7B (LoRA)	16GB	RTX 4080 / L4	~$500
Production Inference	24-80GB	L4 / A10G / A100	$500-$2,000
70B Model Inference	80GB+	A100 80GB / 2x L4	$1,500-$3,000
Full Training (7B+)	80GB+	H100 / 8x A100	$10,000+
Embeddings Generation	4-8GB	CPU / T4 / L4	$100-$300

📊 Key Hardware Metrics

Performance Metrics

VRAM (Video Memory)

Determines max model size. 24GB = 13B FP16, 80GB = 70B FP16

Memory Bandwidth

Critical for LLM inference. HBM3 (H100) >> GDDR6X (RTX)

TFLOPS (Compute)

Training speed. Look at FP16/BF16 TFLOPS, not FP32

Practical Considerations

Power Consumption

H100: 700W, RTX 4090: 450W. Factor in electricity costs

Multi-GPU Scaling

NVLink (400GB/s) >> PCIe (64GB/s). Critical for training

Availability

H100s scarce. Reserve capacity or use spot/preemptible instances

🌳 Quick Decision Tree

Q1:

Running locally or in cloud?

→ Local: RTX 4090 (best value) or Mac M3 (power efficient)

→ Cloud: Continue to Q2

Q2:

Training or inference?

→ Training: A100/H100 (multi-GPU), TPU v4 for large scale

→ Inference: L4 (cost), A10G (balanced), H100 (speed)

Q3:

Budget constraint?

→ Tight: Spot instances, Inferentia2, or quantized models on smaller GPUs

→ Flexible: H100 for best performance, reserved instances for discount