L5 Chapter 9 🐥 🕒 8 min

Robotics + LLM: VLA Models and Embodied AI

PaLM-E, RT-2, π0, Optimus — LLMs and robots officially merge. "See + think + act" in one model. The 2026 embodied intelligence landscape.

Alai

9/27/2026

L5-07 covered 3D generation (the digital world). This piece: putting LLMs into robots — AI actually “moves” and manipulates the physical world.

This is AI’s next 10× direction — embodied intelligence.

VLA: Vision-Language-Action models

Core idea: one model that handles three things at once:

Vision: camera input
Language: human instructions
Action: robot joint angles / gripper force

This family is called VLA models — the new generation replacing traditional robot control (scripts / pure RL).

[Camera image] + [Human says "please bring me the cup on the table"]
                ↓
           VLA model
                ↓
[Joint command sequence: shoulder 30° / elbow -20° / wrist 15° / grip 80%]

Key milestones

1. PaLM-E (Google, 2023-03)

The first impactful VLA:

Combined PaLM (LLM) + ViT (vision) + robot state encoder
562B parameters (largest embodied model at the time)
Demonstrated LLM-style models work for robot control

Significance: The LLM route works in robotics.

2. RT-2 (Google DeepMind, 2023-07)

A step further:

Pretrained on internet images + text
Then fine-tuned on robot demonstration data
“Image-to-text” capability transfers to “image-to-action”
First demo of emergent capability — recognizing objects it was never trained on (e.g. “find the Taylor Swift poster”)

3. π0 (Physical Intelligence, 2024-10)

The new-generation flagship open-source VLA:

Key facts:

Early 2024 Physical Intelligence founded by Sergey Levine and others
$70M seed**, **$ 400M later in 2024, valuation $2B
2024-10-31 π0 released
2025-02-04 open-sourced weights + code (+ π0-FAST autoregressive variant)
2025-04-22 π0.5 released: open-world generalization in never-seen kitchens / bedrooms

Technical features:

Trained on 10,000+ hours of robot demonstrations
Flow matching approach (not traditional autoregressive)
Foundation model + fine-tune paradigm (vs training each task from scratch)

Significance:

The first truly open high-quality VLA foundation model
Lets academic / startup teams build robot applications without their own data collection

4. Tesla Optimus (2022+)

Tesla’s humanoid:

Public demos in 2024 / 2025
Vision side → similar neural net to FSD
Action side → imitation learning + simulation training
2025 commercial target: ** $20K-30K** consumer price (vs$ 100K+ industrial)

5. Figure AI / 1X / Apptronik (the humanoid startup trio)

Figure 02 (2024): partnered with OpenAI, runs GPT-4o for conversation
1X NEO: Nordic humanoid, conversation + home tasks
Apptronik Apollo: warehouse / assembly partnerships with NASA / Mercedes

2024-2026 is the critical window for humanoid + LLM convergence.

Technical challenges

1. Data scarcity

LLMs have internet-scale text; vision models have billions of images. Robot data: thousands of hours — 6 orders of magnitude less.

Directions:

Simulation (Isaac Sim / MuJoCo) — cheap but sim-to-real gap
Large-scale demos (RT-X alliance: 22 labs sharing data)
Cross-embodiment transfer (one robot’s data helps another)

2. Hard to generalize

LLMs work zero-shot on text. Robots: out-of-distribution environments often fail.

π0.5’s progress: first to show useful generalization in “open world” (unfamiliar kitchen).

3. Physical constraints

When LLM is wrong, it’s just text; when robot is wrong, it breaks things / hurts people.

Need force sensing + emergency stop
Safety layer must be separate from LLM decision layer

4. Latency

LLM inference 100ms-2s — too slow for real-time control (millisecond scale).

Mainstream architecture:

High-level planning (slow, big VLA): 5-10Hz
Low-level control (fast, traditional controller): 100-1000Hz

5. Power / compute

Humanoid robots have 1-4 hour battery life — big models burn power. VLAs must be distilled to edge (NVIDIA Jetson, etc).

Compared to traditional robotics

Dimension	Traditional	VLA Agent
Programming	Code / record	Natural language + demos
Environment adaptability	Strict structured	Some generalization
Cross-task	Each task = rewrite	One model, multi-task
Learning	Static deploy	Continuous learning
Failure modes	Crash / error code	”Tries” like a human

Use cases

Short-term (2026-2028) already visible:

Warehousing / logistics: sorting, moving (Amazon, Figure)
Home cleaning: vacuum / tidy (consumer pilots)
Restaurant / kitchen: prep, cleaning
Industrial assembly: flexible lines (Mercedes uses Apptronik)

Mid-term (2028-2030):

General home robots: shopping, dishes, elder care
Construction / agriculture: flexible operations in constrained environments
Medical / nursing assistance

Long-term open:

Fully autonomous home AGI robots
Human-robot collaborative factories

Startup / investment landscape

Humanoid robot valuations (2026 ballpark):

Figure AI: $26B+
1X: ~$2B+
Apptronik: ~$2B+
Physical Intelligence: $2B+
Skild: $1.5B+
Optimus (Tesla internal): tens of billions of standalone value

Model layer (VLA foundations):

Physical Intelligence (most public)
Google DeepMind (internal Gemini Robotics)
NVIDIA Isaac GR00T

The space is capital-intensive — like LLMs but targeting an even larger market (labor replacement).

Compared to LLM era

Timeline analogy:

LLM era	Robotics era
2017 Transformer	2023 PaLM-E / RT-2
2020 GPT-3	2024-10 π0
2022 ChatGPT moment	2026-2027 TBD
2025-2026 mass adoption	2028-2030 projected

If the analogy holds, we’re in the “robotics 2020 GPT-3 moment”. Technology has gone 0 → 1, but the 1 → 100 engineering breakthroughs are still pending.

Learning path

For aspiring entrants:

Math / Python basics (L1)
RL fundamentals
Simulation: Isaac Sim / MuJoCo / Drake
ROS 2 (Robot Operating System)
Get π0 open-source code running (HuggingFace has tutorials)
Find lab / company internship

Key open-source projects:

💡 One framing

LLMs decoupled “intelligence” from hardware — VLAs are putting “intelligence” back into hardware.

2010s software ate the world 2020s LLMs ate software 2030s robots + AI eat the physical world?

But note: home humanoid adoption is much harder than LLM adoption —

Safety bar (LLM errors = text hallucinations; robot errors = physical harm)
Price ( $20K vs$ 20/mo subscription)
Maintenance / battery / updates

Career advice:

If you’re in LLMs but want the “next wave,” robotics + VLA is the highest-probability bet
Learning RL + simulation + ROS is rare and valuable
The wave is still in the 0 → 1 phase, lower entry barrier than LLMs now

🚧 3 Common Pitfalls

⚠️ Real-world traps

Pitfall 1: Underestimating the sim-to-real gap 95% success in sim, 30% on real hardware is common — sim alone isn’t enough; real-robot data is gold.

Pitfall 2: One VLA for every task Even with π0-style foundation models, fine-tuning per scenario still wins big — foundation model ≠ end product.

Pitfall 3: Skipping the safety layer LLM decides “what to do,” but traditional controllers + force sensors must be the last line of defense — never let LLM commands drive motors directly.