GenAIHub
EN | PT
← Back to Technical Section

GenAI Deployment Best Practices

Production-ready strategies for deploying LLM applications at scale

Deploying GenAI Applications

Deploying GenAI applications to production requires careful consideration of latency, cost, reliability, and security. Unlike traditional web apps, LLM-powered applications must handle long response times, variable compute costs, and the unpredictable nature of AI outputs while maintaining excellent user experience.

Key Insight: Production GenAI systems are 80% engineering (caching, streaming, error handling, monitoring) and 20% AI. The model is just one component in a complex system.

⏱️

Latency

💰

Cost Control

🔒

Security

📊

Observability

Production Architecture

👤 User API Gateway Rate limit App Layer • Validation • Cache check • Streaming LLM Router • Model select • Fallback • Load balance OpenAI GPT-4 Anthropic Claude Redis Cache Vector DB Monitoring Logs • Metrics Traces • Alerts

Streaming Responses

Best Practice: Always use streaming for LLM responses. Users see tokens as they arrive, dramatically improving perceived latency even when total generation time is the same.

FastAPI Streaming Example

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import openai

app = FastAPI()

async def generate_stream(prompt: str):
    """Stream tokens from OpenAI"""
    response = await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        stream=True
    )
    
    async for chunk in response:
        if chunk.choices[0].delta.content:
            yield f"data: {chunk.choices[0].delta.content}\n\n"
    
    yield "data: [DONE]\n\n"

@app.post("/api/chat/stream")
async def chat_stream(prompt: str):
    return StreamingResponse(
        generate_stream(prompt),
        media_type="text/event-stream"
    )

Semantic Caching

Cache Similar Queries

import hashlib
import redis
from openai import OpenAI

client = OpenAI()
cache = redis.Redis()

def get_embedding(text: str) -> list:
    """Get embedding for semantic similarity"""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def get_cached_or_generate(prompt: str) -> str:
    # 1. Check exact match cache
    cache_key = hashlib.md5(prompt.encode()).hexdigest()
    cached = cache.get(cache_key)
    if cached:
        return cached.decode()
    
    # 2. Check semantic cache (similarity > 0.95)
    embedding = get_embedding(prompt)
    similar = find_similar_in_cache(embedding, threshold=0.95)
    if similar:
        return similar
    
    # 3. Generate new response
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    result = response.choices[0].message.content
    
    # 4. Cache result
    cache.setex(cache_key, 3600, result)  # 1 hour TTL
    store_embedding(cache_key, embedding)
    
    return result

Error Handling & Fallbacks

import tenacity
from openai import OpenAI, RateLimitError, APIError
from anthropic import Anthropic

openai_client = OpenAI()
anthropic_client = Anthropic()

@tenacity.retry(
    wait=tenacity.wait_exponential(min=1, max=60),
    stop=tenacity.stop_after_attempt(3),
    retry=tenacity.retry_if_exception_type((RateLimitError, APIError))
)
async def call_with_fallback(prompt: str) -> str:
    """Try OpenAI first, fallback to Anthropic"""
    try:
        response = openai_client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            timeout=30
        )
        return response.choices[0].message.content
    
    except (RateLimitError, APIError) as e:
        logger.warning(f"OpenAI failed: {e}, falling back to Anthropic")
        
        # Fallback to Anthropic
        response = anthropic_client.messages.create(
            model="claude-3-sonnet-20240229",
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.content[0].text
    
    except Exception as e:
        logger.error(f"All providers failed: {e}")
        raise

Rate Limiting & Cost Control

Per-User Limits

  • • Requests per minute/hour
  • • Tokens per day
  • • Concurrent requests
  • • Daily/monthly quotas

Cost Controls

  • • Max tokens per request
  • • Model selection by tier
  • • Budget alerts
  • • Automatic shutoff

Warning: An open LLM endpoint without rate limiting can generate thousands of dollars in API costs within hours. Always implement per-user and global limits.

Security Considerations

⚠️ Prompt Injection

Validate and sanitize user inputs. Never expose system prompts. Use structured outputs when possible.

🔑 API Key Security

Never expose keys to frontend. Use secret managers. Rotate keys regularly. Set spending limits.

🛡️ Output Filtering

Filter harmful content, PII, and sensitive data from LLM outputs before returning to users.

📝 Audit Logging

Log all requests/responses for compliance. Enable tracing. Retain for required periods.

Production Deployment Checklist

Infrastructure

  • ☐ Container with health checks
  • ☐ Auto-scaling configured
  • ☐ Secrets in Secret Manager
  • ☐ HTTPS/TLS enabled
  • ☐ Custom domain mapped

Reliability

  • ☐ Retry logic with backoff
  • ☐ Multi-provider fallback
  • ☐ Timeout handling
  • ☐ Circuit breaker pattern
  • ☐ Graceful degradation

Performance

  • ☐ Streaming responses
  • ☐ Response caching
  • ☐ Connection pooling
  • ☐ Async/await patterns
  • ☐ Load testing completed

Observability

  • ☐ Structured logging
  • ☐ Metrics (latency, tokens, cost)
  • ☐ Error tracking
  • ☐ Alerts configured
  • ☐ Request tracing

Best Practices Summary

  • Stream everything: Never wait for full response before showing output
  • Cache aggressively: Both exact match and semantic caching
  • Plan for failure: Retries, fallbacks, circuit breakers
  • Rate limit early: Protect both users and your budget
  • Monitor costs: Track tokens, set alerts, review daily
  • Log everything: Prompts, responses, latency, errors
  • Test with real traffic: Load test before launch
  • Iterate fast: Feature flags for A/B testing prompts

Learn More

Essential Resources

Related Topics

Test Your Knowledge

Score 8/10 or higher to pass