GenAIHub
EN | PT
Back to Technical
Audio AI

Advanced TTS & Speech Synthesis

State-of-the-art TTS has shifted from Mel-spectrogram generation to Neural Codec Language Modeling and Flow Matching. Models like VALL-E 2 and Voicebox now achieve human parity in zero-shot scenarios.

VALL-E 2 (Neural Codec LM)

The Concept

Instead of predicting continuous waveforms, VALL-E 2 treats TTS as a language modeling task. It quantizes audio into discrete tokens (using EnCodec) and uses a Transformer to predict the next audio token based on text and acoustic prompts.

Why it Matters

It was the first to claim Human Parity on zero-shot benchmarks (LibriSpeech). It features "Repetition-Aware Sampling" to fix the infinite loop issues common in autoregressive audio generation.

VALL-E 2 Inference Pipeline

graph LR T[Text Input] --> P[Phonemes] A[3s Audio Prompt] --> C[Audio Codec Codes] P --> LLM(Auto-Regressive Transformer) C --> LLM LLM --> |Predicts| NC[Next Acoustic Tokens] NC --> |Non-AR Refinement| RC[Refined Codes] RC --> |Neural Vocoder| W[Waveform Output]

Voicebox (Flow Matching)

Non-Autoregressive Speed

Voicebox (Meta) moves away from the "left-to-right" token generation. It uses FloMat (Flow Matching), a cousin of Diffusion, to generate speech in parallel. This makes it up to 20x faster than VALL-E.

Speech Infilling

Unlike standard TTS, Voicebox is trained on "Text-Guided Speech Infilling". This makes it incredibly powerful at noise removal, editing misspoken words in recordings, and cross-lingual style transfer.

Open Source Leaders (2024/25)

Parler TTS (Stability AI)

Most TTS models depend on "Reference Audio" for style. Parler is trained to follow Natural Language Descriptions.

"A female speaker with a slightly low-pitched voice delivering the news in a professional tone."

XTTS v2 (Coqui)

The current king of open-source voice cloning. Can clone a voice into 17 different languages from just a 6-second sample. Uses a VQ-VAE + GPT architecture similar to Tortoise but optimized for streaming capability.

Using Parler TTS

import torch
from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer, AutoFeatureExtractor

# Load Model
model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler_tts_mini_v0.1").to("cuda:0")
tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler_tts_mini_v0.1")

# Description controls the style!
prompt = "Hey, how are you doing today?"
description = "A male speaker with a deep voice speaking slowly and sadly."

input_ids = tokenizer(description, return_tensors="pt").input_ids.to("cuda:0")
prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda:0")

# Generate
generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)