Production Monitoring

Logging, metrics, tracing, and alerts for GenAI applications in production

Why Monitor GenAI Applications?

GenAI applications have unique monitoring needs: LLM latency, token usage, cost tracking, and output quality. Without proper observability, you're flying blind—unable to detect degraded responses, runaway costs, or performance issues until users complain.

The Three Pillars of Observability: Logs (what happened), Metrics (how much/how fast), and Traces (the journey through your system).

📋

Logs

Discrete events. Debug info, errors, audit trails.

📊

Metrics

Numeric measurements. Latency, counts, costs.

🔗

Traces

Request flow across services.

Monitoring Architecture

Key Metrics for GenAI Applications

⚡ Performance

• Latency P50/P95/P99: Response time distribution
• Time to First Token: Streaming responsiveness
• Throughput: Requests per second
• Error rate: 4xx/5xx percentage

💰 Cost & Usage

• Tokens used: Input + output per request
• Cost per request: Track API spend
• Daily/monthly spend: Budget tracking
• Cache hit rate: Savings from caching

🤖 LLM-Specific

• Model used: gpt-4, claude-3, etc.
• Fallback rate: How often backup model is used
• Rate limit hits: 429 errors from providers
• Response quality: User feedback scores

🏗️ Infrastructure

• Instance count: Auto-scaling behavior
• Memory/CPU usage: Resource consumption
• Cold starts: Startup latency impact
• Container restarts: Stability indicator

Structured Logging

Best Practice: Use JSON structured logs. They're searchable, filterable, and integrate with Cloud Logging dashboards.

Python Structured Logging

import logging
import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_obj = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
            "function": record.funcName,
        }
        # Add extra fields if present
        if hasattr(record, 'request_id'):
            log_obj['request_id'] = record.request_id
        if hasattr(record, 'user_id'):
            log_obj['user_id'] = record.user_id
        if hasattr(record, 'tokens_used'):
            log_obj['tokens_used'] = record.tokens_used
        if hasattr(record, 'latency_ms'):
            log_obj['latency_ms'] = record.latency_ms
            
        return json.dumps(log_obj)

# Setup logger
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Usage
logger.info("LLM request completed", extra={
    "request_id": "req_123",
    "user_id": "user_456",
    "tokens_used": 1500,
    "latency_ms": 2340,
    "model": "gpt-4"
})

Custom Metrics with Prometheus

from prometheus_client import Counter, Histogram, Gauge
import time

# Define metrics
REQUEST_COUNT = Counter(
    'genai_requests_total', 
    'Total GenAI API requests',
    ['model', 'status']
)

REQUEST_LATENCY = Histogram(
    'genai_request_latency_seconds',
    'Request latency in seconds',
    ['model'],
    buckets=[0.1, 0.5, 1, 2, 5, 10, 30]
)

TOKENS_USED = Counter(
    'genai_tokens_total',
    'Total tokens used',
    ['model', 'type']  # type: input/output
)

ACTIVE_REQUESTS = Gauge(
    'genai_active_requests',
    'Currently processing requests'
)

# Usage in request handler
async def handle_chat(prompt: str, model: str):
    ACTIVE_REQUESTS.inc()
    start_time = time.time()
    
    try:
        response = await call_llm(prompt, model)
        
        # Record metrics
        REQUEST_COUNT.labels(model=model, status='success').inc()
        TOKENS_USED.labels(model=model, type='input').inc(response.input_tokens)
        TOKENS_USED.labels(model=model, type='output').inc(response.output_tokens)
        
        return response
    except Exception as e:
        REQUEST_COUNT.labels(model=model, status='error').inc()
        raise
    finally:
        latency = time.time() - start_time
        REQUEST_LATENCY.labels(model=model).observe(latency)
        ACTIVE_REQUESTS.dec()

Alerting Rules

Critical: Set up alerts BEFORE you go to production. The first major incident shouldn't be how you discover you need monitoring.

Alert	Condition	Severity	Action
High Error Rate	5xx errors > 5% for 5 min	Critical	Page on-call
High Latency	P95 > 30s for 10 min	Warning	Slack notification
Cost Spike	Daily spend > 2x avg	Warning	Email + Slack
Rate Limit Errors	429 errors > 10/min	Info	Slack notification
No Traffic	0 requests for 15 min	Critical	Page on-call

Cloud Logging Queries (GCP)

# Find all errors in the last hour
resource.type="cloud_run_revision"
resource.labels.service_name="genai-api"
severity>=ERROR
timestamp>="2024-01-01T00:00:00Z"

# Find slow requests (latency > 5000ms)
resource.type="cloud_run_revision"
jsonPayload.latency_ms > 5000

# Find rate limit errors
resource.type="cloud_run_revision"
jsonPayload.message:"rate limit" OR jsonPayload.message:"429"

# Track specific user's requests
resource.type="cloud_run_revision"
jsonPayload.user_id="user_12345"

# Calculate token usage by model
resource.type="cloud_run_revision"
jsonPayload.tokens_used > 0

Monitoring Tools

Tool	Type	Best For
Cloud Monitoring	Metrics, Dashboards	GCP-native apps
Cloud Logging	Log aggregation	GCP-native apps
Datadog	Full observability	Multi-cloud, enterprise
Grafana + Prometheus	Metrics, Dashboards	Open-source, self-hosted
LangSmith	LLM observability	LangChain apps
Helicone	LLM observability	OpenAI proxy with analytics

Best Practices Checklist

Structured logs: Use JSON format for searchability
Request IDs: Trace requests across services
Custom metrics: Track tokens, cost, latency by model
Dashboards: Create real-time visibility into system health
Alerts before prod: Set up alerts before launch
Don't log secrets: Redact API keys and PII
Retention policy: Define log retention (30-90 days typical)
Cost monitoring: Alert on unexpected API spend

Learn More

Essential Resources

Test Your Knowledge

Score 8/10 or higher to pass