Text-to-Speech: Voice Synthesis Goes Mainstream

If L5-04 (Whisper) is “audio in”, TTS is “audio out”.

2020 TTS: robotic, monotone, obvious AI. 2025 TTS: indistinguishable from human in many cases.

A quiet revolution.

What TTS Does

Input: text Output: human-like speech audio

"Hello, world!" → [waveform.wav]

Modern TTS controls:

• Voice (which speaker)
• Tone (happy, sad, neutral)
• Pace (fast, slow)
• Pitch
• Emphasis on specific words
• Language + accent

Architecture Evolution

Era 1: Concatenative (~2010)

Record a real speaker reading thousands of sentences. To synthesize, stitch together recorded pieces.

Result: Siri-circa-2011. Worked but stilted.

Era 2: Parametric / RNN (2014-2018)

Use statistical models or RNNs to predict acoustic features (pitch, duration), then a vocoder generates audio.

Less data needed. Quality improved but still robotic.

Era 3: Neural End-to-End (2017+)

Tacotron (2017, Google) — single neural network from text to mel-spectrogram. With WaveNet vocoder for audio.

Quality jumped significantly.

Era 4: Modern Neural TTS (2020+)

Transformer-based, diffusion-based, or hybrid. Examples:

• Tacotron 2 (Google 2018)
• FastSpeech (Microsoft 2019)
• VITS (2021) — end-to-end, no separate vocoder
• NaturalSpeech (Microsoft 2022)
• Voicebox (Meta 2023) — flexible voice cloning
• ElevenLabs (2023-) — best-in-class commercial

Era 5: LLM-Based (2024+)

Treat audio tokens like text tokens, use an LLM to generate audio.

• OpenAI TTS (with voices like “alloy”, “echo”)
• GPT-4o native audio generation
• Gemini Live native audio

These can be prompted in natural language to adjust style.

Quality Today

ElevenLabs / OpenAI TTS / Suno on a normal sentence:

Indistinguishable from human in:

• Casual reading
• Audiobooks
• Voice acting (for some characters)
• Marketing voiceovers

Still imperfect in:

• Long-form emotion (sustained anger, grief)
• Specific accents (regional dialects)
• Singing (mostly — Suno is changing this)
• Whispering / breathing details

Voice Cloning

Modern TTS can clone a voice from 30 seconds of audio:

# ElevenLabs-style API
clone_voice(sample_audio="my_voice.mp3")
synthesize("Hello from my cloned voice", voice="my_cloned_voice")

Implications:

• Personalized AI assistants
• Audiobook narration in author’s voice
• Restoring voices (deceased family members, illness)

And concerns:

• Voice phishing scams (fake calls from “your boss”)
• Misinformation (deepfake speeches)
• Consent issues

Popular Tools

Tool	Strength	Pricing
ElevenLabs	Best overall + cloning	Free tier + paid
OpenAI TTS	Integrated with ChatGPT, simple	$15/M chars
Google Cloud TTS	Many languages, enterprise	Per-char
Microsoft Azure	Voices for many languages	Per-char
Coqui TTS	Open source	Free
XTTS-v2 (Coqui)	Voice cloning open source	Free
Bark (Suno)	Music + voice + sound effects	Free
Suno	Full songs with lyrics	Free + paid

For most use cases: ElevenLabs. For self-hosting: XTTS-v2 or Coqui TTS.

SSML: Adding Control

Speech Synthesis Markup Language — give the TTS more direction:

<speak>
  Welcome to <emphasis level="strong">HelloAI</emphasis>.
  <break time="500ms"/>
  Let me tell you about <prosody rate="slow">deep learning</prosody>.
  My voice can be <prosody pitch="+20%">excited</prosody>
  or <prosody pitch="-20%">calm</prosody>.
</speak>

Most cloud TTS supports SSML. ElevenLabs uses a simpler text-based system.

A Concrete Example

Generate an audiobook from a text file:

from elevenlabs import generate, save

with open("chapter1.txt") as f:
    text = f.read()

# Split into manageable chunks
chunks = [text[i:i+5000] for i in range(0, len(text), 5000)]

# Generate audio for each
audio_parts = []
for chunk in chunks:
    audio = generate(
        text=chunk,
        voice="Adam",   # or your custom cloned voice
        model="eleven_multilingual_v2"
    )
    audio_parts.append(audio)

# Stitch and save
final = b"".join(audio_parts)
save(final, "chapter1.mp3")

A 50,000-word novel → ~10 hours of audio → cost ~$15 with ElevenLabs.

Compare: hiring a voice actor for the same → $1,000-10,000+.

Music Generation: A Cousin

Closely related: Suno and Udio generate full songs with lyrics + vocals + instrumentation.

# Suno API style
song = generate_song(
    lyrics="...",
    genre="jazz",
    duration_seconds=120
)

Quality is shockingly good. Used for:

• Custom jingles
• Indie music exploration
• Personalized songs (birthdays, weddings)
• Memes

Same controversies as voice cloning: copyright, AI vs human artistry.

Latency Considerations

For real-time apps (voice assistants):

Approach	Latency
Pre-generate full audio	200ms-2s
Streaming (audio chunks come out as generated)	100-300ms first byte
LLM with native audio output (GPT-4o, Gemini Live)	<200ms continuous

Latency matters for conversation — anything >500ms feels awkward.

Voice + LLM = Voice Assistant

The full pipeline:

[User speaks]
    ↓
Whisper (or streaming ASR)
    ↓
Text
    ↓
LLM (GPT-4 / Claude / etc.)
    ↓
Response text
    ↓
TTS (ElevenLabs / OpenAI)
    ↓
[Audio output]

Total latency: ~1-2 seconds for ChatGPT Voice Mode. GPT-4o native audio: ~300ms (no transcription/synthesis step).

The biggest UX improvement in voice assistants in 2024 was eliminating the ASR → LLM → TTS chain. Native audio LLMs change the game.

Ethical Issues

Consent

Cloning someone’s voice without permission is identity theft. Most reputable services require consent verification.

Misuse

• Fake phone scams (impersonating relatives in trouble)
• Political misinformation
• Workplace harassment

Mitigations:

• Watermarking AI-generated audio (Anthropic, Adobe working on this)
• Detection models
• Legal frameworks (UK / EU criminalizing deepfake fraud)

Job Displacement

Voice actors, dubbing artists, audiobook narrators — directly impacted.

Same conversation as image gen, video gen: AI augments / displaces creative roles. Society hasn’t fully adjusted.

What’s Coming

Multilingual + Cross-lingual

“Read this English text in my voice, with a Japanese accent” — works now.

Real-time emotion

Adjust tone mid-sentence based on context. Live conversation feel.

Singing + creative voice

Suno already does this. Coming to all TTS.

On-Device

Phone-sized models that generate high-quality voice locally — privacy + offline.

Next recommended: L5-06 Video Generation or L5-04 Whisper.