From Perceptron to Multi-Layer Neural Network

Modern deep learning started with one math model in 1958.

This piece traces the journey from “Perceptron” to “multi-layer neural network” — to see how the field actually evolved.

The Perceptron (1958)

Frank Rosenblatt at Cornell built the Perceptron — designed to mimic biological neurons.

A perceptron:

• Takes inputs $x_1, x_2, ..., x_n$
• Each weighted by $w_1, w_2, ..., w_n$
• Sums + bias: $z = \sum w_i x_i + b$
• Outputs: $1$ if $z > 0$, else $0$

   x_1 ──┐
   x_2 ──┤
   x_3 ──┼──Σ──→ activation → output
   ...   │
   x_n ──┘

Looks familiar? It’s basically logistic regression but with a step function instead of sigmoid.

Marvel of the time: Rosenblatt’s perceptron could learn — automatically adjust weights based on examples.

The First AI Hype (1958-1969)

Rosenblatt promised perceptrons would soon:

• Recognize people from photos
• Translate languages
• Reproduce themselves

Funding poured in. Newsroom claimed: “the embryo of an electronic computer that the Navy expects will be able to walk, talk, see, write, reproduce itself, and be conscious of its existence.”

The First AI Winter (1969-1980s)

Minsky & Papert (1969) wrote the book “Perceptrons”, proving:

A single perceptron can only learn linearly separable functions.

The XOR problem:

Inputs: (0,0) → 0
        (0,1) → 1
        (1,0) → 1
        (1,1) → 0

You can’t draw a single line that correctly separates these. Perceptron can’t learn XOR.

This was devastating. Research funding dried up. “AI Winter”.

The Solution: Multi-Layer (Decades Later)

The fix was obvious: stack multiple layers.

Input → [Layer 1] → [Layer 2] → [Layer 3] → Output
        (hidden)    (hidden)    (hidden)

Each layer:

• Linear transformation: $z = Wx + b$
• Non-linear activation: $a = f(z)$

Multi-layer perceptrons (MLPs) can learn ANY function — proven by Universal Approximation Theorem (1989).

But there was a problem: how to train them?

Backprop Solves Training (1986)

Rumelhart, Hinton & Williams published “Learning representations by back-propagating errors”. Used the chain rule to compute gradients for every layer.

Now we could train deep networks. In principle.

In practice, deep networks were:

• Hard to train (vanishing gradients)
• Computationally expensive
• Outperformed by SVMs in 1990s-2000s for many tasks

The neural network field stalled again.

The Modern Era (2012+)

What changed in 2012:

1. ImageNet — a million labeled images for training
2. GPUs — 100× compute boost
3. ReLU activation — solved vanishing gradients
4. Dropout — prevented overfitting
5. Better initialization — Xavier, He

In 2012, AlexNet (a deep CNN) won ImageNet by a huge margin. The neural network era began.

Anatomy of a Modern MLP

A typical MLP:

import torch.nn as nn

class MLP(nn.Module):
    def __init__(self, input_dim=784, hidden_dim=256, num_classes=10):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),    # Layer 1
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),   # Layer 2
            nn.ReLU(),
            nn.Linear(hidden_dim, num_classes),  # Output layer
        )

    def forward(self, x):
        return self.net(x)

That’s a fully working multi-layer neural network.

Components Explained

Linear Layer

$y = Wx + b$

Just matrix multiplication + bias. The “learnable” part — W and b update during training.

Activation Function

Non-linearity. Without it, stacking layers is useless (still just linear).

Common choices:

• ReLU: $\max(0, x)$ — fast, no vanishing gradient
• GELU: smoother ReLU, used in Transformers
• Sigmoid / Tanh: legacy, mostly avoided in hidden layers

Output Layer

Depends on task:

• Regression: linear (no activation)
• Binary classification: sigmoid
• Multi-class: softmax

Training Loop

for epoch in range(num_epochs):
    for x_batch, y_batch in dataloader:
        # 1. Forward
        y_pred = model(x_batch)
        loss = loss_fn(y_pred, y_batch)

        # 2. Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

That’s it. Every deep learning training loop, in 6 lines.

Why MLPs Aren’t Enough

MLPs work for tabular data and simple tasks. But:

• Don’t scale to images: 1000x1000 image = 1M input dim → 256M params just for input layer
• No structure exploitation: doesn’t know “nearby pixels relate”
• Don’t handle sequences: no concept of order

Solutions came in later chapters:

• CNNs for images (L3-06)
• RNNs / LSTMs for sequences (L3-04)
• Transformers for everything (L3-08)

But these are all specialized neural networks — built on the same foundation: linear layers + nonlinear activations + backprop.

A Concrete Example: MNIST

Classify hand-written digits:

import torch
import torch.nn as nn
from torchvision import datasets, transforms

# Data
train_data = datasets.MNIST('.', train=True, download=True,
                            transform=transforms.ToTensor())
loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)

# Model
class MNIST_MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return self.fc3(x)

# Train
model = MNIST_MLP()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

for epoch in range(5):
    for X, y in loader:
        logits = model(X)
        loss = criterion(logits, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

After 5 epochs: ~97% accuracy on MNIST. Not bad for a few minutes of training.

What Modern MLPs Look Like

Today’s MLPs in production:

• In Transformers: every Transformer block has an MLP component
• In tabular data: still the workhorse (alongside gradient boosting)
• In embedding models: simple MLPs for projecting / refining
• In recommendation: deep & wide MLPs power large recommender systems

The “humble” MLP didn’t disappear — it became a building block of much larger systems.

Next recommended: L3-02 Backpropagation or L3-04 RNN / LSTM.