What is Promptfoo?
Promptfoo is an open-source, config-first toolkit for testing and comparing prompts, models and RAG/agent pipelines. You declare test cases and assertions in a simple YAML file and run them from the CLI or CI — no SDK or cloud account required. It also ships a powerful red-teaming engine that auto-generates adversarial inputs to probe for jailbreaks, prompt injection, PII leaks and toxic output. It's used by teams at OpenAI and Anthropic.
"Write your eval as a config, not as code. Run the same prompt across GPT, Claude and Gemini side by side, see a comparison matrix, and fail the build when a model regresses."
Two Superpowers
Evaluation Matrix
Run N prompts × M models × K test cases and get a side-by-side pass/fail grid with latency and cost. Perfect for choosing a model or iterating on a prompt.
Red Teaming
Auto-generates attacks (jailbreaks, injection, harmful content, PII) mapped to OWASP LLM Top 10 and produces a vulnerability report for your app.
Assertion Types
| Type | Example | Use For |
|---|---|---|
| Deterministic | contains, equals, regex, is-json | Format & keyword checks |
| Similarity | similar (embedding distance) | Semantic equivalence |
| LLM-rubric | llm-rubric, g-eval | Subjective quality, tone |
| RAG | context-faithfulness, context-recall | RAG grounding |
| Custom | javascript / python function | Any bespoke logic |
Quick Start
npx promptfoo@latest init # scaffold
npx promptfoo@latest eval # run
npx promptfoo@latest view # open the web report# promptfooconfig.yaml
prompts:
- "Summarize this in one sentence: {{text}}"
providers:
- openai:gpt-4o-mini
- anthropic:claude-3-5-sonnet
tests:
- vars:
text: "The Eiffel Tower is located in Paris, France."
assert:
- type: contains
value: "Paris"
- type: llm-rubric
value: "is a single concise sentence"