Quality Metrics - Technical

Why Quality Metrics Matter

Unlike traditional software with binary pass/fail tests, LLM outputs exist on a quality spectrum. Metrics provide objective, reproducible ways to measure and compare AI performance across versions, prompts, and use cases.

Objective Comparison

Compare models and prompts fairly

Track Progress

Monitor quality over time

Detect Regressions

Alert when quality drops

Core Quality Metrics

Faithfulness

Measures whether the response is grounded in the provided context. A faithful response only contains information that can be derived from the source documents (no hallucinations).

Formula RAGAS, DeepEval


                                Faithfulness = (Claims supported by context) / (Total claims in response)

✓ High (>0.9): Reliable ⚠ Medium (0.7-0.9): Caution ✗ Low (<0.7): Hallucinating

Answer Relevance

Evaluates how well the response addresses the user's actual question. A relevant answer directly answers what was asked without going off-topic or providing unnecessary information.

Measurement RAGAS, LLM-as-Judge


                                Relevance = cosine_similarity(question_embedding, answer_embedding)

Context Precision

For RAG systems: measures how much of the retrieved context is actually relevant to the question. High precision means the retriever is finding the right documents.


                                Context Precision = (Relevant chunks retrieved) / (Total chunks retrieved)

Context Recall

For RAG systems: measures whether all needed information was retrieved. High recall means no important documents were missed by the retriever.


                                Context Recall = (Ground truth info in context) / (Total ground truth info)

Coherence

Measures how well-structured and logically organized the response is. A coherent answer flows naturally and presents information in a clear, understandable way.

Typically measured via LLM-as-Judge on a 1-5 scale evaluating logical flow, structure, and readability.

Metrics Reference Table

Metric	What it Measures	Use Case	Tools
Faithfulness	Grounding in context	RAG, Q&A	RAGAS, DeepEval
Answer Relevance	Query-response alignment	All LLM apps	RAGAS, LangSmith
Context Precision	Retrieval accuracy	RAG pipelines	RAGAS
Context Recall	Retrieval completeness	RAG pipelines	RAGAS
Coherence	Logical structure	Long-form content	LLM-as-Judge
Fluency	Language naturalness	All LLM apps	LLM-as-Judge
Toxicity	Harmful content	Safety monitoring	Perspective API, Guardrails
BLEU / ROUGE	N-gram overlap	Translation, summarization	evaluate, sacrebleu
BERTScore	Semantic similarity	Any text comparison	bert-score

Implementation Examples

RAGAS - RAG Quality Metrics

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from datasets import Dataset

# Prepare evaluation data
eval_data = {
    "question": ["What is machine learning?"],
    "answer": ["Machine learning is a subset of AI that enables systems to learn from data."],
    "contexts": [["Machine learning is a branch of artificial intelligence (AI) that enables computers to learn from data and improve their performance without being explicitly programmed."]],
    "ground_truth": ["Machine learning is a type of AI where computers learn from data."]
}

dataset = Dataset.from_dict(eval_data)

# Run evaluation
result = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall
    ]
)

print(result)
# {'faithfulness': 0.95, 'answer_relevancy': 0.88, ...}

DeepEval - Unit Testing for LLMs

from deepeval import evaluate
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    HallucinationMetric
)
from deepeval.test_case import LLMTestCase

# Create test case
test_case = LLMTestCase(
    input="What are the benefits of exercise?",
    actual_output="Exercise improves cardiovascular health, boosts mood, and helps maintain healthy weight.",
    retrieval_context=["Regular exercise strengthens the heart, releases endorphins for better mood, and burns calories to maintain weight."]
)

# Define metrics with thresholds
faithfulness = FaithfulnessMetric(threshold=0.7)
relevancy = AnswerRelevancyMetric(threshold=0.7)
hallucination = HallucinationMetric(threshold=0.3)  # Lower is better

# Run evaluation
evaluate([test_case], [faithfulness, relevancy, hallucination])

LLM-as-Judge for Custom Metrics

from pydantic import BaseModel, Field
from openai import OpenAI

class QualityScore(BaseModel):
    coherence: int = Field(ge=1, le=5, description="Logical flow and structure")
    completeness: int = Field(ge=1, le=5, description="Coverage of the topic")
    accuracy: int = Field(ge=1, le=5, description="Factual correctness")
    reasoning: str = Field(description="Brief explanation")

def evaluate_quality(question: str, answer: str) -> QualityScore:
    client = OpenAI()
    
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are an expert evaluator. 
            Rate the answer quality on coherence, completeness, and accuracy (1-5 scale)."""},
            {"role": "user", "content": f"Question: {question}\n\nAnswer: {answer}"}
        ],
        response_format=QualityScore
    )
    
    return response.choices[0].message.parsed

# Example usage
score = evaluate_quality(
    question="Explain quantum computing",
    answer="Quantum computing uses quantum bits..."
)
print(f"Coherence: {score.coherence}/5")
print(f"Reasoning: {score.reasoning}")

Choosing the Right Metrics

RAG Applications

Faithfulness - Critical for grounding
Context Precision/Recall - Retrieval quality
Answer Relevance - Response quality

Chatbots & Assistants

Answer Relevance - On-topic responses
Coherence - Natural conversation
Toxicity - Safety monitoring

Content Generation

Coherence - Well-structured output
Fluency - Natural language
Creativity - Originality (domain-specific)

Translation & Summarization

BLEU/ROUGE - Standard benchmarks
BERTScore - Semantic similarity
Faithfulness - Source preservation

Best Practices

Use Multiple Metrics

No single metric captures all quality aspects. Combine 3-5 relevant metrics for comprehensive evaluation.

Set Clear Thresholds

Define minimum acceptable scores before deploying. E.g., "Faithfulness must be ≥ 0.85 for production."

Validate with Humans

Periodically validate automated metrics correlate with human judgment on sample sets.

Track Over Time

Monitor metrics trends in production. Dashboard alerts when quality dips below thresholds.

Tools & Libraries

RAGAS

RAG-specific metrics: faithfulness, context precision/recall, answer relevancy.

DeepEval

14+ metrics with pytest integration and threshold-based assertions.

LangSmith

Built-in evaluators with custom metric support and production monitoring.

HF Evaluate

BLEU, ROUGE, BERTScore, and 50+ standard NLP metrics.

PromptFoo

Assertion-based testing with llm-rubric and custom metric support.

Braintrust

End-to-end evaluation with experiments, scoring, and CI integration.

Quality Metrics for LLMs