Unit Tests for GenAI Applications

Write reliable tests for ML code, APIs, and AI-powered features

Why Unit Test GenAI Code?

Unit testing for GenAI applications ensures that individual components—prompt templates, data processors, API handlers—work correctly before integration. While LLM outputs can be non-deterministic, surrounding code must be tested rigorously to catch bugs early and enable confident refactoring.

Key Insight: Even for non-deterministic LLM outputs, you can test input validation, response parsing, error handling, and mock the AI responses for predictable test scenarios.

🐛

Catch bugs early

🔄

Safe refactoring

📖

Documentation

⚡

Fast feedback

The Testing Pyramid

Unit Tests (70%)

Test individual functions in isolation. Fast, reliable, run locally.

Integration (20%)

Test component interactions. Database, API calls, external services.

E2E Tests (10%)

Full user journeys. Slowest but catch real-world issues.

Pytest Basics

Simple Unit Test

# tests/test_utils.py
import pytest
from src.utils import format_prompt, parse_response

class TestFormatPrompt:
    def test_basic_template(self):
        """Test prompt template formatting"""
        result = format_prompt(
            template="Hello {name}, your query: {query}",
            name="User",
            query="What is AI?"
        )
        assert result == "Hello User, your query: What is AI?"
    
    def test_missing_variable_raises_error(self):
        """Test that missing variables raise KeyError"""
        with pytest.raises(KeyError):
            format_prompt(template="Hello {name}", query="test")
    
    def test_empty_template(self):
        """Test empty template returns empty string"""
        result = format_prompt(template="")
        assert result == ""


class TestParseResponse:
    def test_parse_json_response(self):
        """Test parsing JSON from LLM response"""
        raw = '{"answer": "42", "confidence": 0.95}'
        result = parse_response(raw)
        assert result["answer"] == "42"
        assert result["confidence"] == 0.95
    
    def test_invalid_json_returns_none(self):
        """Test graceful handling of invalid JSON"""
        result = parse_response("not valid json")
        assert result is None

Fixtures for Reusable Setup

# tests/conftest.py
import pytest
from src.client import GenAIClient

@pytest.fixture
def sample_prompt():
    """Provide a sample prompt for tests"""
    return "Explain machine learning in simple terms"

@pytest.fixture
def mock_client(mocker):
    """Create a mocked GenAI client"""
    client = mocker.Mock(spec=GenAIClient)
    client.generate.return_value = {
        "text": "ML is teaching computers to learn from data",
        "tokens_used": 50
    }
    return client

@pytest.fixture
def test_data():
    """Provide sample test data"""
    return [
        {"input": "What is 2+2?", "expected": "4"},
        {"input": "Capital of France?", "expected": "Paris"},
    ]

Testing GenAI Components

Testing API Endpoints (FastAPI)

# tests/test_api.py
import pytest
from fastapi.testclient import TestClient
from src.main import app

client = TestClient(app)

class TestChatEndpoint:
    def test_chat_returns_200(self):
        """Test successful chat request"""
        response = client.post(
            "/api/chat",
            json={"message": "Hello", "user_id": "test123"}
        )
        assert response.status_code == 200
        assert "response" in response.json()
    
    def test_chat_validates_input(self):
        """Test input validation"""
        response = client.post(
            "/api/chat",
            json={"message": ""}  # Empty message
        )
        assert response.status_code == 422  # Validation error
    
    def test_chat_handles_rate_limit(self, mocker):
        """Test rate limiting behavior"""
        mocker.patch("src.main.check_rate_limit", return_value=False)
        response = client.post(
            "/api/chat",
            json={"message": "Hello", "user_id": "test123"}
        )
        assert response.status_code == 429

Mocking LLM Responses

# tests/test_llm_service.py
import pytest
from unittest.mock import patch, MagicMock

class TestLLMService:
    @patch("src.llm_service.openai.ChatCompletion.create")
    def test_generate_response(self, mock_openai):
        """Test LLM response generation with mocked API"""
        # Mock the OpenAI response
        mock_openai.return_value = MagicMock(
            choices=[MagicMock(message=MagicMock(content="Mocked response"))]
        )
        
        from src.llm_service import generate_response
        result = generate_response("Test prompt")
        
        assert result == "Mocked response"
        mock_openai.assert_called_once()
    
    @patch("src.llm_service.openai.ChatCompletion.create")
    def test_handles_api_error(self, mock_openai):
        """Test graceful handling of API errors"""
        mock_openai.side_effect = Exception("API Error")
        
        from src.llm_service import generate_response
        result = generate_response("Test prompt")
        
        assert result is None  # Or your error handling behavior

Running Tests

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/test_api.py

# Run specific test class
pytest tests/test_api.py::TestChatEndpoint

# Run specific test
pytest tests/test_api.py::TestChatEndpoint::test_chat_returns_200

# Run with coverage report
pytest --cov=src --cov-report=html

# Run only fast tests (marked)
pytest -m "not slow"

# Run tests in parallel
pytest -n auto  # requires pytest-xdist

# Stop on first failure
pytest -x

# Show print statements
pytest -s

What to Test in GenAI Apps

✅ DO Test

• Prompt template formatting
• Input validation & sanitization
• Response parsing & error handling
• API endpoint behavior
• Rate limiting & authentication
• Data processing pipelines
• Embedding generation logic
• Database operations

⚠️ Mock or Skip

• Actual LLM API calls (slow, costly)
• External API dependencies
• Non-deterministic outputs
• Third-party services
• Network-dependent code
• Real database in unit tests

Best Practices

Tip: Use pytest-cov to track test coverage. Aim for 80%+ coverage on business logic, but focus on meaningful tests over hitting coverage numbers.

AAA Pattern: Arrange (setup), Act (call code), Assert (verify)
One assertion per test: Each test should verify one behavior
Descriptive names: test_parse_response_returns_none_for_invalid_json()
Use fixtures: Share setup code via conftest.py fixtures
Mock external dependencies: Don't call real APIs in unit tests
Test edge cases: Empty inputs, None values, large inputs
Keep tests fast: Unit tests should run in milliseconds
CI integration: Run tests on every push via GitHub Actions

Test Project Structure

my-genai-project/
├── src/
│   ├── main.py
│   ├── llm_service.py
│   └── utils.py
├── tests/
│   ├── conftest.py          # Shared fixtures
│   ├── unit/
│   │   ├── test_utils.py
│   │   ├── test_llm_service.py
│   │   └── test_validators.py
│   ├── integration/
│   │   ├── test_api.py
│   │   └── test_database.py
│   └── e2e/
│       └── test_user_flows.py
├── pytest.ini               # Pytest configuration
├── requirements-dev.txt     # Test dependencies
└── .github/workflows/
    └── test.yml             # CI pipeline

Learn More

Essential Resources

Test Your Knowledge

Score 8/10 or higher to pass