Logistic Regression and Classification

L2-02 covered linear regression — predicting continuous numbers. What if you want yes/no answers? Like “is this email spam?” or “will this user click?”

That’s classification — and the cleanest place to start is logistic regression.

The Problem with Linear Regression

You could try to use linear regression for classification:

y = w·x + b
if y > 0.5: class A
else:       class B

Problems:

• Output isn’t bounded (could be -100 or 100)
• Sensitive to outliers
• No probability interpretation

The Sigmoid Solution

Wrap the linear output in a sigmoid:

$$P(y=1 | x) = \sigma(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}}$$

The sigmoid squashes any real number to (0, 1):

σ(-∞) → 0
σ(0)  = 0.5
σ(+∞) → 1

Now you have a probability.

Why Sigmoid

The sigmoid function has nice properties:

• Smooth and differentiable (gradient descent works)
• S-shaped — strongly classifies confident points, soft on borderline
• Probability interpretation — output is the probability of class 1

Training: Cross-Entropy Loss

We learned this in L1-05. For each example:

$$L = -[y \log \hat{y} + (1-y) \log (1-\hat{y})]$$

Where $\hat{y} = \sigma(w \cdot x + b)$.

Why not squared error? Because it’s non-convex when combined with sigmoid — has many local minima. Cross-entropy with sigmoid is convex — guaranteed to find the global optimum.

Visualization

For a 2D problem (two features):

Class 1 (●)                   Class 0 (○)
              ●  ●
           ●        ●  ●
        ●     ●   ●    ●
              .─.
            /     \
○  ○     ○        \    ○
    ○  ○            \  ○
         ○         ○ ○   ○

       ← decision boundary →

Logistic regression finds a linear boundary between classes. Far from the boundary → confident. Near it → low confidence.

Multi-Class: Softmax

For 3+ classes, generalize sigmoid to softmax:

$$P(y=k | x) = \frac{e^{w_k \cdot x + b_k}}{\sum_j e^{w_j \cdot x + b_j}}$$

Each class gets its own weight vector. Output is a probability distribution over all classes.

# 3-class example
logits = [2.0, 1.0, 0.1]
probs = softmax(logits) ≈ [0.66, 0.24, 0.10]
# sum = 1.0

In PyTorch

import torch
import torch.nn as nn

# Binary logistic regression
class LogisticRegression(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.linear = nn.Linear(input_dim, 1)

    def forward(self, x):
        return torch.sigmoid(self.linear(x))

# Multi-class
class MultiClassLR(nn.Module):
    def __init__(self, input_dim, num_classes):
        super().__init__()
        self.linear = nn.Linear(input_dim, num_classes)

    def forward(self, x):
        return self.linear(x)  # raw logits — softmax in loss

# Training (multi-class)
model = MultiClassLR(input_dim=10, num_classes=3)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(100):
    logits = model(X)
    loss = nn.CrossEntropyLoss()(logits, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

CrossEntropyLoss combines softmax + cross-entropy for numerical stability.

With Scikit-learn

For tabular data, scikit-learn is the standard:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

# Binary classification
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

# Multi-class
model = LogisticRegression(max_iter=1000, multi_class='multinomial')

10× less code than PyTorch. Use scikit-learn when you can.

Regularization

Like linear regression, add L1 or L2 penalty:

LogisticRegression(penalty='l2', C=1.0)  # default
LogisticRegression(penalty='l1', solver='liblinear')
LogisticRegression(penalty='elasticnet', l1_ratio=0.5, solver='saga')

L1 → sparse weights (feature selection) L2 → small weights (smooth)

Interpreting Weights

Logistic regression’s superpower: interpretable.

model.coef_  # weights for each feature
# Positive weight → feature predicts class 1
# Negative weight → feature predicts class 0
# Large magnitude → strong predictor

Better than a “black box” deep network for explainable AI / regulated industries.

Limits

• Linear boundary only — can’t learn complex patterns
• Needs feature engineering — raw features may not be linearly separable
• Sensitive to outliers (somewhat)

For “linearly separable” or “nearly so” — perfect. For complex (image, text) — use deep nets.

Real Applications

Despite being old (1958), logistic regression is everywhere:

• Email spam detection (along with naive Bayes)
• Credit scoring (banks love its interpretability)
• Medical diagnosis (disease yes/no)
• Click-through rate prediction (still used in ad systems)
• A/B test analysis (conversion = yes/no)

Industry insight: if your problem is “binary classification” and you don’t have a giant deep net for it — try logistic regression first. It’s often surprisingly good.

Connection to Neural Networks

A single neuron with sigmoid is exactly logistic regression:

input → linear → sigmoid → output

A “deep neural network” is just a stack of these:

input → linear → sigmoid → linear → sigmoid → ... → output

This is the bridge from classical ML to deep learning.

A Worked Example

Predict if a student passes an exam from hours studied:

import numpy as np
from sklearn.linear_model import LogisticRegression

# Data
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
passed = np.array([0, 0, 0, 0, 1, 1, 1, 1])

# Train
model = LogisticRegression()
model.fit(hours, passed)

# Predict
print(model.predict_proba([[3.5]]))  # ~50% chance
print(model.predict_proba([[7.0]]))  # ~95% chance

# Coefficients
print(f"Weight: {model.coef_[0][0]:.2f}")
print(f"Bias:   {model.intercept_[0]:.2f}")
# Each hour studied increases log-odds of passing

Next recommended: L2-04 Decision Trees or L2-07 Evaluation Metrics.