Evaluation, Overfitting, and Regularization

L2-02 through L2-06 covered specific algorithms. This piece: how do you actually know if a model is “good”?

The wrong answer: “It’s 99% accurate on my training data!” The right answer involves: train/val/test splits, multiple metrics, and resisting overfitting.

The Train / Val / Test Split

Never evaluate on training data. Always split:

Set	Purpose	Typical size
Training	Model learns from this	70-80%
Validation	Tune hyperparameters	10-15%
Test	Final, unbiased evaluation	10-15%

Critical: never look at the test set until you’re done. If you do, you’ve leaked it.

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)

Metrics for Classification

Accuracy

$$\text{Accuracy} = \frac{\text{correct predictions}}{\text{total}}$$

Trap: misleading when classes are imbalanced.

If 99% of emails are not spam, predicting “not spam” gets 99% accuracy. But it’s a useless model.

Precision, Recall, F1

For binary classification:

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

• Precision = TP / (TP + FP) — when you say positive, how often right?
• Recall = TP / (TP + FN) — of actual positives, how many did you catch?
• F1 = harmonic mean of precision and recall

Use case decides which matters:

• Cancer detection → high recall (don’t miss any)
• Spam filter → high precision (don’t block real mail)

ROC AUC

For probabilistic classifiers, plot the Receiver Operating Characteristic curve. AUC (area under) → how well the model ranks examples.

AUC = 1.0  → perfect
AUC = 0.9  → very good
AUC = 0.5  → no better than random
AUC < 0.5  → backwards (worse than random)

Confusion Matrix

Visualize all four cells:

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_true, y_pred))

For multi-class, useful for spotting class-specific errors.

Metrics for Regression

• MSE: $\frac{1}{n}\sum (y - \hat{y})^2$ — penalizes large errors
• MAE: $\frac{1}{n}\sum |y - \hat{y}|$ — robust to outliers
• R²: $1 - \frac{\text{MSE}}{\text{var}(y)}$ — proportion of variance explained
• RMSE: $\sqrt{\text{MSE}}$ — same units as target

Overfitting

The most important concept in ML:

Overfitting = model learns the noise + specifics of training data, rather than general patterns.

Symptoms:

• High training accuracy
• Low validation accuracy
• Big gap between them

Train accuracy: 99%
Val accuracy:   65%
→ overfit

What Causes Overfitting

1. Too many parameters for the dataset
2. Too few training examples
3. Too many training epochs without regularization
4. Features that leak the target

Cross-Validation

Standard practice for small datasets:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5)  # 5-fold CV
print(f"Accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

K-fold CV:

• Split data into K parts
• Train on K-1, test on 1
• Rotate which one is held out
• Average the K results

More robust than a single train/val split. Standard cv=5 or cv=10.

Bias-Variance Trade-off

A model’s error has two main sources:

Source	Cause	Cure
Bias	Model too simple	Use more complex model
Variance	Model too complex	Use simpler / regularize

       Total Error
            ↑
            │     ╱
            │    ╱  ← variance
            │   ╱
            │   \
            │    \  ← bias
            │     \
            └──────→
            simple — complex

The “sweet spot” minimizes total. Finding it = the work of evaluation.

Regularization

The fight against overfitting. Three main techniques:

1. L1 / L2 Penalty

Add weight magnitude to the loss:

Loss = data_loss + λ × penalty

• L2 (Ridge): $\sum w_i^2$ — small weights, smooth
• L1 (Lasso): $\sum |w_i|$ — many weights = 0, sparse (feature selection)
• Elastic Net: combination

from sklearn.linear_model import Ridge, Lasso, ElasticNet
model = Ridge(alpha=1.0)         # L2
model = Lasso(alpha=0.1)         # L1
model = ElasticNet(alpha=0.1, l1_ratio=0.5)  # both

2. Dropout (Neural Networks)

Randomly drop neurons during training:

nn.Dropout(p=0.5)

Forces network to not rely on single neurons. Strong regularization.

3. Early Stopping

Watch validation loss. Stop when it stops improving:

best_val_loss = float('inf')
patience = 5
counter = 0

for epoch in range(100):
    train_one_epoch()
    val_loss = evaluate()

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        counter = 0
        save_model()
    else:
        counter += 1
        if counter >= patience:
            break  # stop training

Other Anti-Overfitting Techniques

Data augmentation

For images:

- Random crops
- Flips
- Rotations
- Color jitter

Effectively increases training data → less overfitting.

For text:

• Synonym replacement
• Back-translation
• Random word dropout

More data

The most reliable cure for overfitting. If you can collect more — do.

Simpler model

Sometimes you don’t need 100M params for the task. Try a smaller one.

Feature selection

Drop irrelevant features. Use L1 regularization to do it automatically.

A Real Workflow

# 1. Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# 2. Cross-validate baseline
from sklearn.linear_model import LogisticRegression
base = LogisticRegression()
scores = cross_val_score(base, X_train, y_train, cv=5, scoring='f1')
print(f"Baseline: {scores.mean():.3f}")

# 3. Try with regularization
from sklearn.model_selection import GridSearchCV
params = {'C': [0.01, 0.1, 1, 10, 100]}
grid = GridSearchCV(LogisticRegression(), params, cv=5, scoring='f1')
grid.fit(X_train, y_train)
print(f"Best: {grid.best_params_}, score: {grid.best_score_:.3f}")

# 4. Final test (only ONCE)
final_model = grid.best_estimator_
print(f"Test F1: {f1_score(y_test, final_model.predict(X_test)):.3f}")

Note: I don’t touch the test set until the very last step.

Common Mistakes

1. Tuning on the test set

# WRONG
for hyperparam in [0.01, 0.1, 1, 10]:
    model.fit(X_train, y_train, alpha=hyperparam)
    print(score(X_test, y_test))  # ← peeking at test

This invalidates your “test” score.

2. Data leakage

Features that contain the target:

# WRONG: "average_purchase_amount" might be the answer to "did they make a purchase?"

3. Imbalanced split

# WRONG: stratified split needed for imbalanced data
train_test_split(X, y, stratify=y, ...)

4. Overfitting to validation

If you try 100 things on the validation set, you’ve kind of overfit to it. Hold out a final test set you only use once.

Next recommended: L2-08 SVM or L2-09 Optimizers.