Twilio Voice & Media Streams
Build Real-Time AI Voice Agents with Telephony Integration
What is Twilio Media Streams?
Twilio Media Streams enables real-time audio streaming from phone calls to your server via WebSockets. Instead of waiting for a call to end, you receive audio chunk-by-chunk, process it through STT → LLM → TTS pipelines, and stream responses back—creating ultra-low latency AI voice assistants.
"Media Streams fork the audio from live phone calls, enabling you to build voice AI agents that can understand, reason, and respond in real-time with sub-second latency."
Real-Time
Sub-second latency
Bidirectional
Send & receive audio
PSTN/SIP
Any phone number
WebSocket
Standard protocol
Bidirectional Streaming Architecture
Unlike standard SIP trunks, Media Streams fork the audio. Your server receives the caller's audio chunk-by-chunk and sends back AI-generated audio for playback.
Caller
Phone
Twilio
PSTN
Server
WebSocket
AI
STT→LLM→TTS
AI Voice Pipeline Flow
How It Works
1 TwiML Initiates the Stream
When a call arrives, respond with TwiML containing a <Stream>
element pointing to your WebSocket server.
<Response>
<Connect>
<Stream url="wss://your-server.com/media-stream" />
</Connect>
</Response>2 WebSocket Receives Audio Chunks
Twilio sends JSON messages with base64-encoded audio chunks. Each chunk is ~20ms of audio
in audio/x-mulaw format at 8kHz.
{
"event": "media",
"media": {
"payload": "base64-encoded-audio...",
"timestamp": "1234567890"
}
}3 Process Through AI Pipeline
Forward audio to streaming STT (Deepgram), send transcribed text to LLM, generate response, convert to speech with TTS, and stream back through the WebSocket.
Python Implementation (FastAPI)
# FastAPI WebSocket handler for Twilio Media Streams
from fastapi import FastAPI, WebSocket
import base64
import audioop
import json
app = FastAPI()
@app.websocket("/media-stream")
async def handle_media_stream(websocket: WebSocket):
await websocket.accept()
# Initialize your AI services
stt_client = DeepgramClient()
llm_client = OpenAI()
tts_client = ElevenLabsClient()
while True:
try:
message = await websocket.receive_json()
if message['event'] == 'media':
# Decode audio (mulaw 8kHz -> PCM 16kHz)
audio_chunk = base64.b64decode(message['media']['payload'])
pcm_audio = audioop.ulaw2lin(audio_chunk, 2)
pcm_16k = audioop.ratecv(pcm_audio, 2, 1, 8000, 16000, None)[0]
# Forward to STT for transcription
transcript = await stt_client.transcribe(pcm_16k)
if transcript:
# Get LLM response
response = await llm_client.chat(transcript)
# Convert to speech and stream back
audio_response = await tts_client.synthesize(response)
await send_audio_to_twilio(websocket, audio_response)
elif message['event'] == 'stop':
break
except Exception as e:
print(f"Error: {e}")
breakKey Challenges & Solutions
Latency Optimization
Every millisecond counts. Users expect instant responses like a human conversation.
- Use streaming STT (Deepgram Nova-2)
- Stream LLM responses token-by-token
- Use optimized TTS (OpenAI TTS-1, ElevenLabs Turbo)
Barge-In / Interruption
Users will interrupt the bot. You must detect speech and stop playback immediately.
- Implement VAD (Voice Activity Detection)
- Send "clear" message to stop Twilio playback
- Cancel pending TTS generation
Audio Format Conversion
Twilio uses mulaw 8kHz, but AI models expect linear PCM at higher sample rates.
- Convert
mulaw → PCM with
audioop - Resample 8kHz → 16kHz for STT
- Convert TTS output back to mulaw for Twilio
Concurrent Calls
Each call needs its own state, context, and AI session.
- Use async Python (FastAPI/Starlette)
- Maintain per-call conversation state
- Scale with Redis for session storage
Recommended AI Services
| Component | Service | Latency | Notes |
|---|---|---|---|
| STT | Deepgram Nova-2 | <100ms< /td> | Streaming, high accuracy |
| STT | OpenAI Whisper API | ~500ms | Batch only, multi-language |
| LLM | GPT-4o / GPT-4o-mini | <200ms TTFT | Streaming enabled |
| TTS | ElevenLabs Turbo | <100ms< /td> | Natural voices, streaming |
| TTS | OpenAI TTS-1 | <150ms< /td> | Cost-effective, good quality |
Use Cases
Customer Support
24/7 AI agents for tier-1 support
Appointment Booking
Schedule meetings via phone
Surveys & Screening
Automated phone interviews
Outbound Reminders
Appointment confirmations
Translation
Real-time multilingual
Healthcare Triage
Symptom collection