Support Vector Machines (SVM)

Before deep learning, SVM (1995) was the king of classification for about 15 years.

It’s still useful, beautiful mathematically, and worth understanding.

The Core Idea

For binary classification, many lines can separate the data. Which is “best”?

*  *
   *      |   /
*    *    |  /     line A
       *  | /
    *    *|/       line B
        */         (which is better?)
       /|*
      / | *  *
   B / A|     *  *

SVM says: the line with the largest margin (distance to nearest points).

            |        ←  Margin
*  *        |        ←
   *        |
*    *      |    *
       *   <|>       ← support vectors
    *    *  |
            |
       *    |
            |          *  *

Why? Largest margin = best generalization. Intuition: a wider buffer is more robust to noise.

Support Vectors

The points on or inside the margin are the support vectors — they define the boundary.

All other points can be removed and the answer doesn’t change.

This is why SVM is memory-efficient at test time — only need to remember support vectors, not all training data.

Hard-Margin SVM

If data is linearly separable, find $w$ and $b$ that:

• Correctly classify all points: $y_i (w \cdot x_i + b) \geq 1$
• Maximize margin: $\frac{2}{\|w\|}$

Equivalent to:

$$\min_{w, b} \frac{1}{2} \|w\|^2 \text{ subject to } y_i(w \cdot x_i + b) \geq 1$$

This is a convex optimization problem — has a unique global solution.

Soft-Margin SVM

Real data isn’t perfectly separable. Add slack variables $\xi_i \geq 0$:

$$\min_{w, b, \xi} \frac{1}{2} \|w\|^2 + C \sum \xi_i$$

Subject to: $y_i(w \cdot x_i + b) \geq 1 - \xi_i$

C is the regularization parameter:

• Large C → strict (close to hard-margin), risks overfitting
• Small C → lenient (more violations allowed), risks underfitting

C is the SVM hyperparameter you tune most.

The Kernel Trick

SVM’s killer feature. What if data isn’t linearly separable?

Original (2D):
  ○ ○ ○
 ○ ● ● ○
○ ● ● ● ○
 ○ ● ● ○
  ○ ○ ○

Cannot separate with a line.

Trick: project to higher dimension where it IS linearly separable:

Transform: (x, y) → (x, y, x² + y²)

In 3D:
   ○ ○ ○ ○ ○ ○ ○ ○      (outer ring at high z)

      ● ● ●               (inner cluster at low z)
       ● ●

Now a horizontal plane separates them.

The “kernel trick” lets SVM operate in this high-dim space without explicitly computing the projection — using kernel functions.

Kernel Functions

A kernel $K(x_i, x_j)$ computes the dot product in the transformed space:

Kernel	Form	When
Linear	$x_i \cdot x_j$	Already linearly separable
Polynomial	$(x_i \cdot x_j + c)^d$	Polynomial boundaries
RBF (Gaussian)	$\exp(-\gamma \|x_i - x_j\|^2)$	Most general, default choice
Sigmoid	$\tanh(\alpha x_i \cdot x_j + c)$	Like neural net

RBF (Radial Basis Function) is the default — works well in most cases.

from sklearn.svm import SVC

# Linear SVM
svm = SVC(kernel='linear', C=1.0)

# RBF SVM (default)
svm = SVC(kernel='rbf', C=1.0, gamma='scale')

# Polynomial
svm = SVC(kernel='poly', degree=3, C=1.0)

In Scikit-Learn

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Pipeline: normalize + SVM (always normalize for SVM!)
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf', C=1.0))
])

pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

Always normalize before SVM — kernel functions are scale-sensitive.

Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

params = {
    'svm__C': [0.1, 1, 10, 100],
    'svm__gamma': ['scale', 'auto', 0.01, 0.1, 1],
    'svm__kernel': ['rbf', 'linear']
}

grid = GridSearchCV(pipe, params, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)

print(grid.best_params_)

SVM for Regression: SVR

Same idea, applied to regression:

from sklearn.svm import SVR

svr = SVR(kernel='rbf', C=1.0, epsilon=0.1)
svr.fit(X_train, y_train)

Less popular than classification but works.

Multi-Class

SVM is binary by default. For multi-class, scikit-learn uses:

• One-vs-Rest: train k classifiers (each class vs all others)
• One-vs-One: train k(k-1)/2 classifiers (every pair)

Both handled internally:

svm = SVC(decision_function_shape='ovr')  # default
svm = SVC(decision_function_shape='ovo')

Strengths

• High accuracy with small datasets
• Good generalization (theoretically backed by VC theory)
• Works in high dimensions (kernel trick — even when N < D)
• Convex optimization → unique solution
• Sparse model — only support vectors matter

Weaknesses

• Slow on large datasets — training is O(n² to n³)
• Choosing kernel + gamma + C is non-trivial
• Not probabilistic by default (Platt scaling needed for probabilities)
• Doesn’t scale to millions of examples like neural nets do
• No good way to inspect the decision (vs interpretable linear model)

SVM vs Other Methods

For tabular data with N ~ 1k-100k:

Method	Speed	Accuracy	Interpretability
Logistic Regression	Fast	Good (if linear)	High
Decision Tree	Fast	Medium	High
Random Forest	Medium	Very good	Medium
SVM (RBF)	Slow on large N	Very good	Low
Gradient Boosting (XGBoost, LightGBM)	Medium	Best for tabular	Low
Neural Net	Slow	Variable	None

In 2024-2025: Gradient Boosting is usually the go-to for tabular. SVM is still competitive on small datasets.

When to Use SVM

• Small dataset (< 10k samples)
• High dimensions (e.g., text classification with TF-IDF)
• Need good generalization without much tuning
• Non-linear boundary suspected (use RBF kernel)

When to avoid:

• 1M samples (too slow)

• Need probability scores (logistic regression simpler)
• Need interpretability
• Image / audio data (use deep nets)

Historical Importance

SVM dominated 1995-2012 because:

• Solid theoretical foundation
• Worked well with available data sizes
• Kernel trick was elegant + practical
• Compute was limited (deep nets needed more)

In 2012 AlexNet beat all SVM-based image classifiers on ImageNet. The era ended.

But SVM isn’t dead — it’s still:

• Best choice for small specialized datasets
• Standard baseline in many papers
• Taught in every ML course

Concrete Example: Text Classification

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline

texts = [...]  # 1000 documents
labels = [...] # categories

pipe = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=5000)),
    ('svm', LinearSVC(C=1.0))
])

pipe.fit(texts, labels)
pipe.predict(["A new test document"])

LinearSVC is the optimized linear SVM — much faster than RBF on text data.

For 2024: still competitive with simple BERT setups for some specialized text classification.

Next recommended: L2-09 Optimizers or L3-01 Perceptron to MLP.