Building AI Agents: From Tool Use to Autonomous Workflows

L4-01 / L4-02 / L4-03 covered “what LLMs are” and “how to make them more useful”. This piece is about: how do you let an LLM actually DO things?

The answer is the Agent.

From Chat to Agent

A chat LLM just talks. An Agent acts:

User:   "What's the weather in Tokyo this weekend?"

Chat:   "I don't know — I can't look up real-time info."

Agent:  → Calls weather API → Gets data → "Saturday cloudy 22°,
         Sunday clear 25° — nice weather for sightseeing."

The difference: calling tools + acting on results.

Agent’s Three Cornerstones

A working Agent needs three things:

1. Tools (Tools)

External capabilities the LLM can invoke:

• Web search / weather / stock API
• File read/write / command execution
• Send email / database queries
• Other LLM calls (sub-agents)

2. The Loop (ReAct loop)

The Agent runs in a loop:

Loop:
  1. Think — analyze the current state
  2. Decide — pick a tool to call (or finish)
  3. Act — call the tool
  4. Observe — read the result
  Until done

This pattern is called ReAct (Reasoning + Acting), 2022 paper.

3. Memory

Without memory, the Agent forgets previous steps in the same task.

Categories:

• Short-term memory: context window history
• Long-term memory: vector DB / database
• Working memory: scratch space for the current task

A Minimal Agent

Let’s build one (Python):

import openai
import json

# Tools the Agent can use
TOOLS = {
    "get_weather": lambda city: f"{city}: clear, 22°C",
    "calculator": lambda expr: str(eval(expr)),
}

def agent_loop(user_input, max_steps=10):
    messages = [
        {"role": "system", "content": "You are an Agent. Call tools to solve the user's problem."},
        {"role": "user", "content": user_input}
    ]

    for step in range(max_steps):
        # 1. Ask LLM to think + decide
        response = openai.chat.completions.create(
            model="gpt-4",
            messages=messages,
            tools=[
                {"type": "function", "function": {"name": "get_weather", "parameters": {...}}},
                {"type": "function", "function": {"name": "calculator", "parameters": {...}}},
            ]
        )

        msg = response.choices[0].message

        # 2. If LLM wants to call a tool
        if msg.tool_calls:
            for call in msg.tool_calls:
                tool_name = call.function.name
                args = json.loads(call.function.arguments)
                result = TOOLS[tool_name](**args)

                # 3. Tool result becomes context
                messages.append({"role": "tool", "content": result, "tool_call_id": call.id})

        # 4. If LLM thinks it's done, return the answer
        else:
            return msg.content

    return "Max steps exceeded"

# Usage
print(agent_loop("What's the weather in Tokyo, then compute (22+25)/2"))
# → "Tokyo clear 22°C, average temp is 23.5°"

About 30 lines for a working Agent. Real-world ones are more complex but the skeleton is this.

Agent Patterns

1. ReAct

Thought: I need to find the weather first.
Action: get_weather(Tokyo)
Observation: Clear, 22°C.
Thought: Now I need to compute.
Action: calculator((22+25)/2)
Observation: 23.5
Thought: I have both pieces. Done.
Final Answer: 23.5°C.

The classic, most-used pattern.

2. Plan-and-Execute

Two phases:

1. Planner: break the task into sub-tasks
2. Executor: run sub-tasks in order

Suited for complex multi-step tasks.

3. Multi-Agent

Multiple specialized Agents collaborating:

• Researcher Agent: search + gather info
• Writer Agent: draft
• Critic Agent: review + suggest

See L4-10.

4. Tree of Thoughts (ToT)

The Agent explores multiple solution paths, picks best:

• Suited for puzzles, math
• More expensive but better quality

Real-World Cases

1. Cursor / Claude Code

AI coding Agent:

• Tools: file read/write, bash command, git, npm
• Pattern: ReAct + sandbox
• Capability: read project structure, modify multiple files, run tests

2. AutoGPT (2023, history)

First popular fully-autonomous Agent.

• Goal: a long-term objective like “Build a startup”
• Behavior: self-decomposes, runs continuously
• Problem: too easy to get stuck in loops / hallucinate ridiculous plans

3. Devin (2024)

“AI software engineer”:

• Codes in a long-running sandbox
• Can run for hours
• Reportedly does end-to-end tasks

4. Claude Computer Use (2024)

Anthropic let Claude:

• See screenshots (screen)
• Control mouse + keyboard
• Operate any GUI

Like a remote desktop AI assistant.

Common Issues with Agents

1. Hallucinated tool calls

The LLM calls a non-existent tool, or passes wrong args:

LLM: get_weather(city="Mars")
Tool error: city not supported

Solutions:

• Strict tool schema (type validation)
• Catch errors and feed back to the LLM
• Multi-round retry

2. Infinite loops

The Agent keeps calling the same tool getting nowhere:

get_weather(Tokyo) → 22°C
get_weather(Tokyo) → 22°C
get_weather(Tokyo) → 22°C
...

Solutions:

• max_steps limit
• Detect repetitive behavior + force-stop
• Better prompting (tell the LLM “if you’ve tried something already don’t try it again”)

3. Token explosion

Many tool calls / long history → context fills up:

• 100 steps → potentially 100k tokens
• Cost soars
• Eventually exceeds context limit

Solutions:

• Periodic summarization (compress history)
• Strip old tool results
• Use cheaper model for analysis (e.g., Haiku) + flagship for final answer

4. Security

An Agent that can call tools = an Agent that can do harm:

• Send wrong emails
• Delete files
• Spend money
• Prompt injection (see L4-07)

Solutions:

• Sandbox all execution
• Approval flow for dangerous ops (need user confirm)
• Whitelist allowed tools
• Audit logs for every action

Agent Frameworks

Don’t write from scratch. Mainstream choices:

Framework	Best for	Maturity
LangChain	Generic, broad ecosystem	Mature, sometimes overengineered
LangGraph	Stateful complex workflows	Mature
AutoGen	Multi-Agent	Microsoft, active
CrewAI	Role-based Multi-Agent	Easy to use
OpenAI Assistants API	Hosted Agent	OpenAI managed
MCP	Tool protocol standard	Anthropic standard, growing

For a beginner: start with LangGraph or CrewAI; for production, write your own thin layer.

A Real Architecture Template

A production-grade Agent typically has these layers:

┌─────────────────────────────────┐
│       User Interface             │
│   (chat / CLI / API)             │
└────────────────┬─────────────────┘
                 ↓
┌─────────────────────────────────┐
│       Agent Loop                 │
│   (ReAct / plan-execute)         │
└────────────────┬─────────────────┘
                 ↓
┌──────────┐ ┌──────────┐ ┌──────┐
│  LLM     │ │  Memory  │ │Tools │
│ (Claude/ │ │ (vector  │ │ MCP  │
│  GPT)    │ │  DB)     │ │ APIs │
└──────────┘ └──────────┘ └──────┘
                 ↓
┌─────────────────────────────────┐
│       Audit + Telemetry          │
│   (every step logged)            │
└─────────────────────────────────┘

Each layer:

• LLM: the “brain”
• Memory: long-term storage
• Tools: external action
• Audit: traceability for debugging + safety

When Do You Need an Agent

Don’t reach for an Agent without thinking:

Need	Use Agent?
Single-turn Q&A	❌ Plain chat
Search + summarize	❌ RAG
Math, code generation	❌ Plain chat with code tool
Multi-step task, tool use	✅ Agent
Long-running autonomous	✅ Agent
Multi-source aggregation	✅ Agent

Many problems labeled “we need an Agent” actually just need RAG. Agents are useful, but overuse adds complexity + cost + risk.

Next recommended: L4-05 LoRA Fine-tuning or L4-07 MCP Protocol.