PromptOps

Prompt Management

Treat prompts as strategic assets. Learn versioning, testing, deployment workflows, and the tools that enable production-grade prompt operations.

🎯 Why Prompt Management?

❌ Without Management

• Prompts scattered across codebase
• No history of what worked before
• Production bugs hard to reproduce
• Inconsistent outputs across environments
• Security vulnerabilities undetected

✅ With Management

• Centralized prompt registry
• Full version history with rollback
• Automated testing before deploy
• A/B testing for optimization
• Injection attack prevention

💡 Key Insight: Prompts are the "source code" of LLM applications. A single-word change can dramatically alter output quality, cost, and safety.

🔄 Prompt Lifecycle

1. Create

Draft & iterate

2. Version

Track changes

3. Test

Validate quality

4. Deploy

Go to production

5. Monitor

Track & improve

📦 Prompt Templates

Store prompts as structured templates with variables, metadata, and version info.

YAML Template Format

# prompts/customer_support.yaml
name: customer_support
version: 2.1.0
description: Handle customer inquiries
model: gpt-4o-mini
temperature: 0.3

template: |
  You are a support agent for {company}.
  
  Guidelines:
  - Be helpful and professional
  - Escalate billing issues
  
  Context: {context}
  Question: {question}

variables:
  - name: company
    required: true
  - name: context
    default: ""
  - name: question
    required: true

tests:
  - input: {question: "refund policy"}
    assert_contains: ["refund", "days"]

Python Loading Templates

# Using a prompt registry
from prompt_registry import PromptRegistry

registry = PromptRegistry("./prompts")

# Load specific version
prompt = registry.get(
    "customer_support", 
    version="2.1.0"
)

# Render with variables
rendered = prompt.render(
    company="Acme Corp",
    context=retrieved_docs,
    question=user_query
)

# Call LLM
response = llm.generate(rendered)

🔀 Version Control Strategies

Strategy	When to Use	Pros	Cons
Git-based	Small teams, few prompts	Free, familiar workflow	No UI for non-engineers
Prompt Platform	Teams with PM/DS involvement	UI, collaboration, analytics	Additional cost
Database + API	Enterprise, dynamic updates	Hot reload, access control	Build/maintain yourself

📝 Semantic Versioning for Prompts

MAJOR (3.0.0): Breaking changes - different output format, behavior change

MINOR (2.1.0): New capabilities - added variables, improved instructions

PATCH (2.0.1): Bug fixes - typos, minor wording improvements

🧪 Prompt Testing

Unit Tests

• Output contains expected keywords
• JSON output is valid
• Response within token limit
• No harmful content patterns

Evaluation Suites

• Golden dataset comparisons
• LLM-as-judge scoring
• Human evaluation samples
• Regression tests vs baseline

Example: Promptfoo Test Config

# promptfoo.yaml
prompts:
  - prompts/customer_support.yaml

providers:
  - openai:gpt-4o-mini
  - anthropic:claude-3-haiku

tests:
  - vars:
      question: "How do I get a refund?"
    assert:
      - type: contains
        value: "refund"
      - type: llm-rubric
        value: "Response is helpful and professional"
      - type: cost
        threshold: 0.001

🛠️ Tools Comparison

Tool	Best For	Key Features	Pricing
LangSmith	LangChain users	Tracing, datasets, hub	Free tier + paid
Langfuse	Open-source preference	Self-host, observability	OSS + cloud
Promptfoo	Testing focus	CLI, CI/CD, evals	Free OSS
PromptLayer	Version control	History, analytics	Free + paid
Humanloop	Enterprise	Experiments, fine-tuning	Paid

✅ Best Practices

Do's

Store prompts separately from application code
Include metadata (author, date, model)
Test with diverse edge cases
Use feature flags for gradual rollout
Monitor quality metrics post-deploy

Don'ts

Hardcode prompts in source files
Deploy without testing
Forget to test injection attacks
Ignore cost implications of changes
Delete old versions (keep for rollback)