3D Generation: NeRF / Gaussian Splatting / Text-to-3D

After 2D image → video, the next stop is 3D.

3D assets that took a professional modeler days to build in 2020, in 2025-2026 can be auto-reconstructed from a few photos or generated from a single sentence.

This piece covers the two main lines: NeRF and Gaussian Splatting, plus the latest in text-to-3D.

Why 3D is hard

Previous AI generations operated in 2D pixel space. 3D’s challenges:

Challenge	Why
No standard representation	Mesh / voxel / point cloud / implicit — no single winner
Data scarcity	Real-world 3D datasets are much smaller than image datasets
View consistency	The same object must look consistent from any angle
Physics + lighting	Transparency / reflections / shadows must be modeled
Complex rendering pipelines	Training + inference involve rasterization / ray tracing

1. NeRF (2020): Neural Field Revolution

NeRF (Neural Radiance Fields): represent a 3D scene with a neural network.

Core idea

Don’t store geometry (a mesh) — store a function “any 3D point + any view direction → color + opacity”:

$$F_\theta(x, y, z, \theta, \phi) \to (R, G, B, \sigma)$$

• Input: 3D position + view direction (5D)
• Output: color + opacity
• $F_\theta$ is a small MLP (~10 layers)

Rendering

For each pixel, cast a ray, sample multiple 3D points along it, query the MLP for colors/opacities, composite via volume rendering:

def render_pixel(ray_origin, ray_direction, nerf):
    # sample 64 points along the ray
    points = sample_along_ray(ray_origin, ray_direction, n=64)
    colors, densities = nerf(points)
    # Volume rendering (α-compositing)
    return alpha_composite(colors, densities)

Training

Given a set of multi-view real photos (with camera poses), the training target is to make each pixel’s predicted color match the photo.

for batch_pixels in random_pixels:
    pred = render_pixel(ray, nerf)
    loss = ((pred - true_color) ** 2).mean()
    loss.backward()
    optimizer.step()

Result

After training, you can render photorealistic images from any new viewpoint.

NeRF’s problems

• Slow training: hours per scene
• Slow inference: 64+ network queries per pixel
• Hard with dynamic scenes: original NeRF is static-only

Follow-ups: Instant-NGP (5-minute training), NeRF-W, NeRF++, PixelNeRF (few-shot generalization), etc.

2. Gaussian Splatting (2023): Performance Revolution

Where NeRF uses an implicit neural representation, Gaussian Splatting uses millions of explicit 3D Gaussian ellipsoids:

Scene = { (position μ, covariance Σ, color c, opacity α) × N ellipsoids }

Each ellipsoid is a “3D Gaussian blob” with its own position, size, color, and opacity.

Rendering: splatting

Unlike NeRF’s 64 samples per pixel — directly project each ellipsoid onto the 2D screen and α-blend:

For each gaussian:
  project to screen
  splat as 2D ellipse with opacity
Sort gaussians by depth
α-blend in depth order

Fully differentiable + GPU-friendly — 100× faster than NeRF.

Training

Also supervised with multi-view photos, but optimizing the parameters of each ellipsoid (position, shape, color, etc.).

Gradient descent + adaptive ellipsoid insertion/deletion (add where residuals are high, prune useless ones) drives convergence.

Result

• Real-time rendering (>30 FPS) — NeRF can’t do this
• Visual quality comparable to NeRF
• Fast training (10-30 minutes)

Industry adoption

Within a year, Gaussian Splatting has largely replaced NeRF in industry:

• Unity / Unreal both added native support
• Polycam, Luma AI and other 3D scanning apps use it for real-time reconstruction
• VR / AR content generation

3. Text-to-3D

NeRF / GS both need real photos. But users want “one sentence → 3D model.”

DreamFusion (2022)

The first meaningful result:

prompt = "a red apple"
   ↓
Score Distillation Sampling (SDS)
   - Initialize a random NeRF
   - Render the current viewpoint
   - Have a 2D Diffusion model (Imagen / SD) score "does this look like a red apple?"
   - Backprop gradients into the NeRF
   - Repeat

Result: generates 3D objects, but quality is mediocre and training is slow (hours).

Magic3D / Fantasia3D / Latent-NeRF (2023)

Various improvements: better base models, smarter SDS losses, higher-resolution optimization.

Trellis / Trellis-XL (2024)

New generation: directly train a text → 3D diffusion model, skip SDS iteration:

prompt
   ↓
Text encoder
   ↓
3D Diffusion (in some 3D representation space)
   ↓
3D asset (exportable as mesh)

Result:

• Usable 3D model in seconds
• Quality far exceeds SDS-based work
• Open-sourced by Microsoft, community building on it

Commercial products (2026)

Tool	Input	Output	Quality
Meshy	Text / image	Mesh with textures	Commercially usable
Tripo AI	Single image	High-quality mesh	Near game-grade
Luma AI	Text / video / photos	NeRF / GS / mesh	Strong for real scenes
Polycam	Phone scan	GS / mesh	Consumer-grade
Adobe Substance 3D Sampler	Single image + AI	PBR materials	Professional

Applications

• Games: auto-generate NPCs / scenes / props
• Film/TV: previz / concept art → 3D assets
• E-commerce: from one product photo to a 360° rotatable model
• AR / VR: democratize 3D content
• Robotics: rapid 3D environments for simulation training
• Architecture / design: from sketch to 3D model

4D generation (dynamic 3D + time)

Next stop: not static objects but scenes that move:

• 4D Gaussian Splatting: each Gaussian evolves over time
• Make-A-Video3D: text → dynamic 3D
• DreamGaussian4D

By 2027, expect “text → fully walkable 3D scene” products.

Relationship to multimodal LLMs

GPT-4V / Claude 3.5 can “see images” — but can’t “understand 3D geometry.”

Next-gen research:

• Vision-Language Models for 3D (PointLLM, 3D-LLM)
• Genie 2 / SIMA: understand + generate interactive 3D worlds

3D is the bridge from “describe an image” to “understand + manipulate the physical world” — crucial for connecting LLMs to embodied AI / robotics.

Code: train a tiny NeRF

import torch
import torch.nn as nn

class TinyNeRF(nn.Module):
    def __init__(self, hidden=128):
        super().__init__()
        # Positional encoding lets MLPs learn high-frequency detail
        self.L = 10  # number of frequencies
        in_dim = 3 * (2 * self.L + 1)  # x, y, z + sin/cos
        self.mlp = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, 4)  # rgb + density
        )

    def positional_encoding(self, x):
        out = [x]
        for i in range(self.L):
            for fn in [torch.sin, torch.cos]:
                out.append(fn((2**i) * x))
        return torch.cat(out, dim=-1)

    def forward(self, points):
        # points: (N, 3)
        encoded = self.positional_encoding(points)
        out = self.mlp(encoded)
        rgb = torch.sigmoid(out[..., :3])
        density = torch.relu(out[..., 3])
        return rgb, density

50 lines + a few hours of GPU training = something that works on the MIT Lego dataset.

Next: L5-06 Video Generation or L5-02 Diffusion Math.

🚧 3 Common Pitfalls