LLMOps

CI/CD for LLMs

Automated pipelines and safe deployment strategies for LLM applications. Learn how to test prompts, validate RAG indices, and deploy with confidence.

🎯 Why CI/CD for LLMs?

LLM applications require specialized CI/CD pipelines that test not just code, but also prompts, model configurations, and RAG indices. Automated testing and safe deployment reduce human error and accelerate iteration.

❌ Without CI/CD

• Manual prompt testing
• "It worked on my machine"
• Silent regressions in prod
• Slow, risky deployments

✅ With CI/CD

• Automated prompt regression tests
• Consistent evaluation metrics
• Fast, safe incremental rollouts
• Automatic rollback on failure

💡 Best Practice: Treat prompts like code. Include them in version control, test them in CI, and deploy them through the same pipeline as your application.

🔄 LLM-Specific Pipeline Stages

Prompt Testing

Validate templates with golden datasets

RAG Validation

Test retrieval quality & index freshness

Evaluation Suite

Groundedness, relevance, safety checks

Safe Deploy

Canary, blue-green, auto-rollback

🚀 Safe Deployment Strategies

Strategy	How It Works	Rollback	Best For
🔵🟢 Blue-Green	Two identical environments, instant switch	Instant	Zero-downtime deployments
🐤 Canary	1-10% traffic to new version, gradual increase	Fast	Risk mitigation, A/B testing
🌊 Rolling	Replace instances one by one	Medium	Resource-efficient updates
🏷️ Feature Flags	Toggle features without deploy	Instant	Prompt A/B tests, dark launches

🐤 Canary for LLMs

Start with 1-5% traffic. Monitor latency, error rates, and quality metrics (groundedness, relevance). Auto-rollback if metrics degrade.

🏷️ Prompt Feature Flags

Test new prompts on specific user cohorts. Compare business metrics before full rollout. Tools: LaunchDarkly, Flagsmith, ConfigCat.

📋 Example GitHub Actions Pipeline

# .github/workflows/llm-cicd.yaml
name: LLM CI/CD Pipeline

on:
  push:
    branches: [main]
    paths:
      - 'prompts/**'
      - 'src/**'
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run Prompt Unit Tests
        run: python -m pytest tests/prompts/ -v
        
      - name: Evaluate with Promptfoo
        run: |
          npx promptfoo eval --config promptfoo.yaml
          npx promptfoo view --output results.html
        
      - name: Check Safety (Guardrails)
        run: python scripts/safety_check.py --strict
        
      - name: RAG Retrieval Quality
        run: python scripts/rag_eval.py --threshold 0.8

  deploy-canary:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy Canary (5%)
        run: |
          gcloud run deploy $SERVICE --image $IMAGE \
            --tag canary --no-traffic
          gcloud run services update-traffic $SERVICE \
            --to-tags canary=5
      
      - name: Monitor for 10 minutes
        run: python scripts/monitor_canary.py --duration 600
        
      - name: Full Rollout (if healthy)
        if: success()
        run: |
          gcloud run services update-traffic $SERVICE \
            --to-latest

🧪 Testing Strategies for LLMs

Unit Tests

• Prompt template rendering
• Output format validation
• Edge cases (empty input, long text)
• Mock LLM responses

Integration Tests

• End-to-end RAG pipeline
• Real LLM API calls (staging)
• Chain/Agent execution
• Database interactions

Evaluation Tests

• Golden dataset comparisons
• LLM-as-Judge scoring
• Metric thresholds (RAGAS)
• Safety/toxicity checks

🛠️ CI/CD Tools for LLMs

Tool	Type	Best For	Integration
Promptfoo	Prompt Testing	CI prompt evaluation	GitHub Actions, CLI
RAGAS	RAG Evaluation	Retrieval quality metrics	Python, pytest
DeepEval	LLM Testing	pytest-style assertions	pytest native
LangSmith	Tracing + Eval	Debug, evaluate, monitor	LangChain native
Weights & Biases	Experiment Tracking	Compare prompt versions	Python SDK

✅ Best Practices

Do's

Version control prompts alongside code
Run evaluations on every PR
Set quality thresholds as gates
Use canary deployments with auto-rollback
Monitor production metrics continuously

Don'ts

Deploy prompts without testing
Skip evaluation for "small" changes
Deploy 100% traffic immediately
Ignore latency/cost metrics
Store API keys in pipeline logs

Test Your Knowledge

Score 8/10 or higher to pass