Whisper: Speech Recognition Solved (Mostly)

In 2022, OpenAI released Whisper — an automatic speech recognition (ASR) model.

It was open-source (Apache 2.0), multilingual (99 languages), and best-in-class in quality.

Whisper effectively solved speech-to-text for most use cases. This is its story.

The Pre-Whisper World

Before 2022, speech recognition was hard:

• Required specialized companies (Google, Amazon, Nuance)
• Closed APIs
• Quality varied by language (English fine, others uneven)
• Punctuation / casing often wrong
• Background noise / accents = errors

Each company built their own dataset and pipeline. Open-source efforts (Mozilla DeepSpeech) were limited.

What Whisper Did

OpenAI trained Whisper on 680,000 hours of multilingual audio scraped from the web — diverse, noisy, real-world.

Architecture: standard Transformer encoder-decoder.

• Encoder: takes audio mel-spectrograms, outputs representation
• Decoder: generates text autoregressively

Trained on multiple tasks simultaneously:

• Transcription (audio → text in same language)
• Translation (audio → English text from any language)
• Language identification
• Voice activity detection

Just by training on a lot of varied data, the model became robust to:

• Background noise
• Accents
• Speed (fast/slow speakers)
• Music in background
• Multiple speakers (somewhat)

Model Sizes

Model	Params	Languages	Speed
tiny	39M	99	Realtime+
base	74M	99	Realtime
small	244M	99	~realtime
medium	769M	99	< realtime
large	1.55B	99	Slowest, best quality
large-v3 (2023)	1.55B	99	Improved version

For most use cases: medium or large. Tiny/base for embedded.

Performance

On standard benchmarks (WER = word error rate):

Language	Whisper Large WER	Pre-Whisper SOTA
English (clean)	3-5%	4-6%
English (noisy)	8-12%	15-25%
Mandarin	8-12%	10-15%
Japanese	10-15%	15-25%
Low-resource languages	30-50%	50-80%

Cross-language, multi-condition robustness was the breakthrough.

Using Whisper

Locally with openai-whisper

pip install openai-whisper

# Command line
whisper audio.mp3 --model large --language en

# Python
import whisper
model = whisper.load_model("large")
result = model.transcribe("audio.mp3")
print(result["text"])

Faster Implementations

The original Python implementation is slow. Better:

Implementation	Speed	Notes
openai-whisper	1×	Reference, slow
faster-whisper (CTranslate2)	4×	Best CPU/GPU
whisper.cpp	5× CPU	C++ port, runs on Mac/embedded
WhisperX	3-10× + timestamps	Aligned with words
Distil-Whisper (HF)	6×	Distilled smaller variant

For production: faster-whisper or WhisperX.

Via API

OpenAI offers a hosted API:

from openai import OpenAI
client = OpenAI()
audio = open("audio.mp3", "rb")
result = client.audio.transcriptions.create(model="whisper-1", file=audio)
print(result.text)

$0.006 per minute → for occasional use, simpler than self-hosting.

Strengths

1. Multilingual

Single model handles 99 languages. No per-language tuning needed.

2. Robustness

Handles real-world audio: phone calls, podcasts, lectures, voice memos.

3. Punctuation + Capitalization

Outputs include proper punctuation, casing, paragraphs. Most prior ASR was lowercase + no punctuation.

4. Open Source

MIT-licensed weights. Companies can self-host without API costs.

5. Translation Built-in

Whisper can directly translate from any language to English text:

result = model.transcribe("french.mp3", task="translate")
# → English text of the French content

Limits

1. Hallucination

Whisper sometimes invents text during silence:

• “Thank you for watching!” (because trained on YouTube)
• Random filler when audio is unclear

Solutions:

• Use voice activity detection (VAD) to skip silence
• WhisperX includes this

2. Speaker Diarization

Whisper transcribes but doesn’t identify who is speaking. For “Speaker A said… Speaker B said…”:

• Use pyannote-audio or similar for diarization
• Combine with Whisper transcription

3. Real-Time Latency

Whisper processes 30-second chunks. Real-time captioning requires chunking + overlap handling.

Solutions: whisper-live projects, streaming variants.

4. Long Files

Long audio (1h+) can drift in time alignment. Solution: chunk + align.

Common Use Cases

• Podcast transcription (1 hour = ~2 minutes processing)
• Meeting notes + summarization (Whisper + GPT-4)
• Subtitle generation (with timestamps via WhisperX)
• Voice commands (combined with LLM for understanding)
• Audio search (transcribe + index → search by text)
• Accessibility (live captions)
• Compliance (call recording transcription)

The Ecosystem It Enabled

Post-Whisper, an entire ecosystem of audio AI emerged:

• Otter.ai — meeting transcription (premium)
• Descript — audio/video editing via text
• Wispr Flow — speech-to-text typing
• Rev — transcription service (still uses humans + AI hybrid)

Whisper made “audio understanding” accessible. Companies built on top instead of from scratch.

Beyond Whisper

What’s next:

Whisper Successors

• OpenAI continues improvement (whisper-large-v3 in 2023)
• Open-source models match / exceed in specific languages

SeamlessM4T (Meta 2023)

Translation across 100 languages + speech-to-speech. “Universal translator” idea — Star Trek tech for real.

Real-time Streaming ASR

Models designed for low-latency:

• NVIDIA Canary
• Conformer-based architectures
• Trade some accuracy for streaming

Speaker-Aware Models

Models that natively handle diarization + transcription.

A Concrete Example

Transcribe a podcast + summarize:

import whisper
from openai import OpenAI

# 1. Transcribe with Whisper
model = whisper.load_model("medium")
result = model.transcribe("podcast.mp3", language="en")
transcript = result["text"]

# 2. Summarize with GPT
client = OpenAI()
summary = client.chat.completions.create(
    model="gpt-4",
    messages=[{
        "role": "user",
        "content": f"Summarize this podcast transcript:\n\n{transcript}"
    }]
).choices[0].message.content

print(summary)

This is how modern podcast / video summarization apps are built.

Technical Insight: Mel-Spectrogram

Whisper doesn’t see raw audio. Audio is converted to log-mel spectrogram:

Audio (1D waveform)
   ↓ STFT (sliding FFT)
Power spectrogram (frequency × time)
   ↓ Mel filterbank (perceptual scale)
Mel spectrogram (80 channels × time)
   ↓ log
Log-mel spectrogram → input to Whisper

This is the standard audio representation for deep learning.

Why? Mel scale matches human hearing — close frequencies near 100Hz, less precision at 10kHz.

Cost Comparison

• OpenAI API: $0.006/minute →$0.36/hour
• Self-host on GPU: ~$0.05/hour (compute cost)
• Self-host on CPU with whisper.cpp: ~$0 (slow but free)

For high volume: self-host. For low volume / convenience: API.

Next recommended: L5-05 TTS or L5-01 Multimodal Overview.