Video Generation: From Sora to Modern Video AI

L5-02 covered Diffusion math. This piece: extending diffusion from single frames to time — video generation.

2024-2026 saw an explosion in AI video: Sora / Runway Gen-3 / Veo / Kling / Pika made “text → 1-minute high-quality video” a reality. This is the next chapter of the AIGC revolution.

Why Video Is Hard

Why is video so much harder than images?

1. Dimensional Explosion

• Single image: (H, W, 3) → ~500k pixels
• 1-min 30fps 1080p video: (1800, 1080, 1920, 3) → ~11 billion pixels
• 20,000× more data

2. Temporal Consistency

Frame 1 = a white cat Frame 100 should still be the same white cat — not suddenly change color or disappear.

This “object permanence” is physical common sense — AI has to learn it.

3. Motion / Physics

• Objects thrown must fall (gravity)
• Water must flow correctly
• Clothing must move with material properties
• …

Model has to implicitly learn physics — not just “copy-paste frames”.

4. Long Dependencies

10-second video = 300 frames — Frame 300 must “align” with Frame 1:

• Same character
• Consistent lighting
• Coherent story

Transformer attention over 300 frames for “global consistency” is very expensive.

Sora’s Core Innovations

1. Spatial-Temporal Patches

L5-01 introduced this. Sora cuts video into 4D space-time patches:

Video (T × H × W × 3)
   ↓ split into space-time patches (1 frame + 16×16 pixel block)
"Space-time tokens" sequence
   ↓
Transformer processes them like text

Each patch has both spatial and temporal info — Transformer unified attention in spacetime.

2. DiT (Diffusion Transformer)

Not U-Net — Transformer doing the diffusion denoising:

Noisy video + timestep t + text prompt
   ↓
DiT (Transformer)
   ↓
Predict noise
   ↓
Denoise one step

DiT advantages:

• Scales up well (bigger models benefit more)
• Flexibly handles any resolution + length
• Borrows from LLM training experience

3. Multi-Scale Training

Sora’s training data mixed:

• Short / long videos
• Landscape / portrait
• 4K / 480p
• Different frame rates

No fixed input format — model learns “flexibility”.

”World Model” Hypothesis

OpenAI’s report repeatedly emphasized:

“Sora is an early version of a world simulator (emergent world simulator)”

Evidence:

• Physical consistency emerges
• Multi-view understanding
• Long-term consistency
• Simple causal relationships

Academic debate — Some think Sora really “understands” physics; others say it’s high-level “interpolation”.

Either way — Sora drove the “video generation → AGI” discussion forward.

Mainstream Video Models

2026 landscape:

Model	Company	Max Length	Quality	Note
Sora	OpenAI	60s	Top	Best physical consistency
Veo 2	Google	2 min	High	YouTube integrated
Runway Gen-3	Runway	30s	High	Commercial-ready, film industry favorite
Kling	Kuaishou	2 min	High	China leader
Pika 2.0	Pika	16s	Mid-high	Easy, social-oriented
Hunyuan Video	Tencent	5s	Mid	Open-source (13B)
CogVideoX	Zhipu	6s	Mid	Open-source
Open-Sora	HPC AI	4s	Mid	Fully open reproduction

Closed source vs open source — gap ~6-12 months.

Engineering Challenges

The practical issues:

1. Compute Cost

Generating 1 minute of video:

• Memory: A100 80GB minimum, often multi-GPU
• Time: Sora ≈ 1-5 min to generate 1 min of video
• Cost: roughly $0.5-2 per generation

Video APIs are 100× more expensive than image APIs.

2. Prompt Engineering

Video prompts are much more complex than image prompts:

Image prompt: "a cat on a windowsill, watercolor style"

Video prompt:
"A serene scene of an orange tabby cat sitting on a wooden windowsill,
gently grooming its fur. Soft golden hour light streams in through
the slightly fogged glass. The camera slowly dollies in over 5 seconds,
ending in a close-up of the cat's content expression. The cat blinks
slowly, peaceful and unhurried. Watercolor aesthetic, gentle palette,
subtle paper texture visible."

Need to describe:

• Subject + action
• Camera movement
• Lighting changes
• Time pacing
• Style

Video director / prompt engineer is becoming a new role.

3. Controllability

Want Sora to generate “a virtual version of my son” — near-impossible.

Video models lack:

• Image-to-Video (partial, hard to precisely control)
• Character consistency (multiple videos of same person)
• Physical manipulation (fine adjustment)

This is why video AI is still far from “actual filmmaking”.

4. Copyright + Ethics

Video AI’s unique risks:

• Training on videos (YouTube, etc.) — copyright?
• Celebrity deepfakes (more realistic)
• Political misinformation
• Sexual / violent content

All vendors add watermarks + moderation — but deepfake defense remains unsolved.

Use Cases

1. Content Creation

• Video blog / social shorts
• Advertising / marketing
• Personal vlog post-production
• AI-generated stock footage

Cuts video production cost 100× — average people can do what TV stations used to.

2. Film / TV

• Storyboarding
• VFX / post
• Stock footage
• Early test shots

2023 Hollywood strikes — actors, writers, editors all worried about replacement.

3. Education

• AI-generated lecture videos
• Historical scene recreations
• Science visualization
• Multilingual versions (generated simultaneously in multiple languages)

4. Gaming

• Cutscenes
• NPC behavior
• Procedurally generated scenes
• Personalized stories

5. Training / Simulation

• Medical simulation (surgery practice)
• Flight / driving sims
• Industrial safety training

Using It

Via API (simplest)

# OpenAI Sora API (hypothetical)
from openai import OpenAI
client = OpenAI()

response = client.videos.generate(
    model="sora",
    prompt="A cat sitting on a windowsill, watching rain. Cinematic, slow camera dolly.",
    duration=10,  # seconds
    resolution="1080p",
)

video_url = response.url

Via Open Source

# Hunyuan Video / CogVideoX
from diffusers import HunyuanVideoPipeline

pipe = HunyuanVideoPipeline.from_pretrained(
    "tencent/HunyuanVideo",
    torch_dtype=torch.bfloat16,
).to("cuda")

video = pipe(
    prompt="A peaceful waterfall in a forest",
    num_frames=64,
    height=720,
    width=1280,
).videos[0]

export_to_video(video, "output.mp4")

Open-source video generation needs 8+ H100 GPUs — not consumer hardware.

The Next 2-3 Years

Predictions:

1. Quality + Length Continue Up

• 2026: 60s high-quality
• 2027-2028: 5-10 min coherent stories
• 2029-2030: full short films (30 min+)

2. Control / Consistency Breakthroughs

Character consistency + physical control + editing capability — the keys to “practical” use.

3. Real-Time Generation

Now: 60s takes 5 min. Future: real-time (for games / VR):

• 60fps video in 1/60 sec
• Needs 100× current compute
• Possibly 2027-2030

4. Multimodal Fusion

Text + image + audio + 3D + video — unified generation. GPT-5 / Claude 5 class models might be this.

5. Mass Tooling

Like ChatGPT for text — video gen will get its “ChatGPT for normal users”. Pika and others are exploring.

Next recommended: L5-04 Whisper or L5-02 Diffusion Math.