Vision-Language Agents: Letting AI Operate Your Screen

L4-04 covered Agent concepts. L4-09 covered “calling API tools.” This piece: let the Agent operate your computer directly — see the screen, click, type.

This is the most impactful LLM product form factor of 2024-2026 — no API needed; use the GUI when there’s no GUI.

One-line definition

Computer-Use Agent (CUA / VLA Agent / GUI Agent):

A vision-language model (VLM) observes screenshots → decides next step → outputs mouse/keyboard actions → executed in a real operating system / browser.

Difference from traditional RPA (UiPath / Selenium):

• RPA: record scripts, hard-code coordinates and selectors → breaks when UI changes
• CUA: looks at the screenshot each step → adapts like a human → keeps working even when UI changes

Main players

1. Anthropic Computer Use (2024-10)

The first to ship “Computer Use” as an API product:

# Claude takes screenshots + task, returns mouse/keyboard action sequences
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    tools=[{"type": "computer_20241022", "name": "computer",
            "display_width_px": 1024, "display_height_px": 768}],
    messages=[{"role": "user", "content": "Build me a budget table in Numbers"}]
)
# Model returns: {action: "screenshot"} / {action: "mouse_move", x, y} / {action: "key", text: "..."}

Architecture:

• Model doesn’t directly control your computer — it returns action descriptions
• Your code (or SDK reference impl) executes them
• Usually runs in a sandboxed VM / Docker container

2. OpenAI Operator (2025-01)

OpenAI’s counterpart, focused on browser automation:

• Runs in cloud virtual browser (user doesn’t expose their local screen)
• Suited for booking / shopping / form-filling
• Accessed via ChatGPT web; no public low-level API

3. Google Gemini 2.5 Computer Use (2025)

Gemini’s version — deepest integration with Android / Workspace.

Benchmark reality

The two key benchmarks:

OSWorld (general OS + multi-app):

Model	Score
OpenAI CUA	38.1%
Claude Computer Use	22%
Human baseline	~72%

WebVoyager (browser only):

Model	Score
OpenAI CUA	87%
Claude Computer Use	56%

Note: scores change quickly with version updates — always check current model cards.

Key observations:

• Browser tasks (WebVoyager) succeed far more often than general OS tasks (OSWorld)
• Human 72% vs current best 38% — still a 2× gap on general OS
• Simple tasks (click button) work; complex tasks (multi-app coordination) often fail

Technical challenges

1. GUI Grounding: identifying elements on screen

The model must figure out from a screenshot “where exactly is the Submit button” — not a text task, but a pixel-position task.

Main approaches:

• VLM directly outputs (x, y) coordinates (Claude / OpenAI)
• OCR-label all clickable elements → model picks by number (some open-source Agents)

2. Operational Knowledge: knowing what to do next

“Open Excel → New file → Pick template → Fill data” — that’s task-level procedural knowledge, harder to train than GUI grounding.

3. Error recovery

Slow loads / popups / CAPTCHAs / expired logins — production has more edge cases than an Agent will ever see in training.

4. Safety / isolation

An Agent running on your machine = can execute any action:

• Delete files
• Transfer money
• Send messages
• Irreversible operations

Must be isolated in a VM / container / browser sandbox.

Use cases

Already works:

• Data migration: copy from system A to system B
• Simple form filling: signup, shopping, food ordering
• Repetitive GUI testing (Selenium replacement)
• Accessibility helper for users with disabilities

Not yet:

• Complex multi-app coordination (“find email attachment → process in Excel → reply”)
• Enterprise long workflows (full sales funnel, HR onboarding)
• Creative work (designing posters in Photoshop)

Compared to traditional RPA

Dimension	Traditional RPA	Computer Use Agent
Setup	Record scripts / write selectors	Natural language task
UI tolerance	Breaks when UI changes	Adapts
Learning curve	Engineer / business training	Anyone can write tasks
Speed	Fast (seconds)	Slow (screenshot each step)
Cost	One-time dev, reused	LLM tokens per run
Best for	High-frequency stable flows	Long-tail / exploratory tasks

Not a replacement relationship — RPA is for stable high-volume; CUA is for one-off / long-tail / adaptive tasks. Future RPA tools will likely embed CUA for “self-healing.”

Example: have Claude clean your desktop

import anthropic, subprocess, base64

def run_action(action):
    """Execute the action Claude returned"""
    if action["action"] == "screenshot":
        subprocess.run(["screencapture", "-x", "/tmp/s.png"])
        return open("/tmp/s.png", "rb").read()
    elif action["action"] == "mouse_move":
        subprocess.run(["cliclick", f"m:{action['x']},{action['y']}"])
    elif action["action"] == "left_click":
        subprocess.run(["cliclick", f"c:{action['x']},{action['y']}"])
    elif action["action"] == "key":
        subprocess.run(["cliclick", f"kp:{action['text']}"])

client = anthropic.Anthropic()
messages = [{"role": "user", "content": "Move all screenshots from Desktop to ~/Screenshots/"}]

while True:
    resp = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        tools=[{"type": "computer_20241022", "name": "computer",
                "display_width_px": 1440, "display_height_px": 900}],
        messages=messages
    )
    if resp.stop_reason == "end_turn":
        break
    for block in resp.content:
        if block.type == "tool_use":
            result = run_action(block.input)
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result", "tool_use_id": block.id,
                "content": [{"type": "image", "source": {
                    "type": "base64", "media_type": "image/png",
                    "data": base64.b64encode(result).decode()
                }}]
            }]})

Warning: this is illustrative — don’t run it raw on your main machine. Use a VM.

2026 maturity assessment

CUA today:

• ✅ Browser tasks (WebVoyager 87%) are close to “daily usable”
• ⚠️ Desktop tasks (OSWorld 38%) still far from production
• ⚠️ Per-step latency (screenshot + inference) makes tasks 10× slower
• ⚠️ Token costs high (each screenshot ≈ a few thousand tokens)

Use / don’t-use cheat sheet:

Scenario	Usable today?
Browser info gathering / form filling	✅ Yes
Data migration (A → B)	✅ Yes
Replacing repetitive customer-support clicks	✅ Yes in scoped contexts
Full project-level work	❌ Too early
Operating complex pro software (Photoshop / CAD)	❌ Too early

Relationship to MCP / Tool Use

Not replacement — complementary:

• Function Calling / Tool Use: call services with APIs
• MCP: standardized tool protocol
• Computer Use: handle software / websites without APIs

The strongest Agents use all three — prefer API when available (fast, stable, cheap), use CUA as fallback.

Recommended further reading

• HelloAI L4-04 Agent construction
• HelloAI L4-09 Tool Use engineering
• HelloAI L6-03 Red teaming and jailbreaks (CUA introduces new attack surface)
• Papers: WebArena / WebVoyager / OSWorld
• Anthropic Computer Use announcement
• OpenAI Operator announcement

🚧 3 Common Pitfalls