Pandas: Data Wrangling for ML

For ML, 70% of your time is data wrangling. Pandas is the standard tool.

It’s like Excel + SQL + NumPy, in Python.

DataFrames and Series

import pandas as pd

# Series — a 1D labeled array
s = pd.Series([10, 20, 30], index=["a", "b", "c"])

# DataFrame — a 2D labeled table
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Carol"],
    "age": [25, 30, 35],
    "city": ["Tokyo", "NYC", "Paris"]
})

print(df)
#     name  age   city
# 0  Alice   25  Tokyo
# 1    Bob   30    NYC
# 2  Carol   35  Paris

Reading Data

# CSV
df = pd.read_csv("data.csv")

# Parquet (faster, smaller — preferred for big data)
df = pd.read_parquet("data.parquet")

# Excel
df = pd.read_excel("file.xlsx", sheet_name="Sheet1")

# JSON
df = pd.read_json("data.json")

# SQL
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///mydb.sqlite")
df = pd.read_sql("SELECT * FROM users", engine)

Inspecting Data

df.head()         # first 5 rows
df.tail(10)       # last 10
df.shape          # (rows, cols)
df.columns        # column names
df.dtypes         # data types
df.describe()     # numeric stats: mean, std, min, max, quartiles
df.info()         # memory usage, types
df.isnull().sum() # missing values per column
df.value_counts() # value distribution (Series)

Always run df.info() and df.describe() first when exploring new data.

Selecting Data

# Single column → Series
df["age"]

# Multiple columns → DataFrame
df[["name", "age"]]

# Rows by position
df.iloc[0]        # first row
df.iloc[0:3]      # first 3 rows
df.iloc[:, 0]     # first column

# Rows by label
df.loc[0]                 # row with index label 0
df.loc[df["age"] > 25]    # filter

# Boolean mask
df[df["age"] > 25]
df[(df["age"] > 25) & (df["city"] == "NYC")]   # & not "and"!
df[df["city"].isin(["Tokyo", "NYC"])]

Two main accessors: .iloc[] (integer position) and .loc[] (label).

Modifying Data

# New column
df["age_in_decade"] = df["age"] // 10 * 10

# Conditional column
df["is_senior"] = df["age"] > 30

# Apply function
df["name_upper"] = df["name"].apply(str.upper)
df["name_length"] = df["name"].str.len()

# Drop column
df = df.drop(columns=["city"])

# Drop rows
df = df.dropna()                    # any NaN
df = df.dropna(subset=["age"])      # NaN only in 'age'
df = df.drop_duplicates()

Missing Data

Real data has missing values. Pandas handles them:

# Detect
df.isnull().sum()
df["age"].isnull()

# Fill
df["age"].fillna(df["age"].mean())     # mean imputation
df["city"].fillna("Unknown")
df.fillna(method="ffill")              # forward-fill

# Drop
df.dropna()
df.dropna(thresh=3)                    # keep rows with ≥3 non-null

Group By

The most powerful Pandas operation:

# Group → aggregate
df.groupby("city")["age"].mean()
# city
# NYC      30
# Paris    35
# Tokyo    25

# Multiple aggregations
df.groupby("city").agg({
    "age": ["mean", "max"],
    "name": "count"
})

# Custom function
df.groupby("city")["age"].apply(lambda x: x.max() - x.min())

SQL equivalent:

SELECT city, AVG(age), MAX(age), COUNT(name)
FROM df
GROUP BY city;

Pandas groupby does this in 1 line.

Merge / Join

Combine two DataFrames:

users = pd.DataFrame({"id": [1, 2, 3], "name": ["A", "B", "C"]})
orders = pd.DataFrame({"user_id": [1, 1, 2], "amount": [10, 20, 30]})

# Inner join
merged = users.merge(orders, left_on="id", right_on="user_id")

# Left join (keep all users)
merged = users.merge(orders, left_on="id", right_on="user_id", how="left")

SQL JOIN equivalents. Pandas supports inner / outer / left / right.

Concatenate

df1 = pd.DataFrame({"a": [1, 2]})
df2 = pd.DataFrame({"a": [3, 4]})

# Stack vertically
pd.concat([df1, df2])   # 4 rows

# Stack horizontally
pd.concat([df1, df2], axis=1)

Reshape

Pivot

Wide table → long, or vice versa:

df = pd.DataFrame({
    "date": ["2024-01", "2024-02", "2024-01"],
    "city": ["NYC", "NYC", "LA"],
    "temp": [40, 45, 65]
})

# Pivot: city × date → temp
df.pivot(index="city", columns="date", values="temp")
# date    2024-01  2024-02
# city
# LA           65      NaN
# NYC          40       45

# Melt: pivot back to long
df.melt(id_vars=["city"], var_name="date", value_name="temp")

Stack / Unstack

For multi-index data:

df.set_index(["city", "date"]).unstack("date")

Date Handling

df["date"] = pd.to_datetime(df["date"])

df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["dayofweek"] = df["date"].dt.dayofweek
df["is_weekend"] = df["date"].dt.dayofweek >= 5

# Resample (time-series)
df = df.set_index("date")
df.resample("D").mean()     # daily mean
df.resample("M").sum()      # monthly sum

Pandas has powerful date handling — useful for time-series ML.

ML-Specific Patterns

1. Train/test split

from sklearn.model_selection import train_test_split

X = df.drop(columns=["target"])
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

2. Convert to NumPy

X = df[["feat1", "feat2", "feat3"]].values   # → numpy array
y = df["target"].values

3. One-hot encoding

df_encoded = pd.get_dummies(df, columns=["city"])

4. Categorical to integer

df["city_id"] = df["city"].astype("category").cat.codes

5. Normalization

df["age_normalized"] = (df["age"] - df["age"].mean()) / df["age"].std()

Common Mistakes

1. Forgetting axis

df.drop("col_a")              # ERROR — assumes row
df.drop("col_a", axis=1)      # correct — drop column
df.drop(columns=["col_a"])    # clearer

2. Chain indexing

df["col"][df["col"] > 5] = 999    # may not modify df!
df.loc[df["col"] > 5, "col"] = 999  # correct

SettingWithCopyWarning — read it carefully.

3. & vs and

df[df["a"] > 5 and df["b"] < 10]   # ERROR
df[(df["a"] > 5) & (df["b"] < 10)] # correct

Element-wise operations use &, |, ~, not Python’s and, or, not.

4. Performance

# Slow: iterating rows
for i, row in df.iterrows():
    do_stuff(row["a"], row["b"])

# Fast: vectorized
df["c"] = df["a"] + df["b"]

Avoid iterrows unless necessary. Use vectorized operations.

When NOT to Use Pandas

Pandas is great up to ~10M rows. Beyond that:

• Polars — faster, lower memory, similar API
• DuckDB — SQL on Pandas/files, much faster for analytics
• Dask — parallel Pandas
• PySpark — Big Data (cluster-scale)

For ML, you usually preprocess in Pandas, then convert to NumPy/PyTorch for training.

Next recommended: L1-10 PyTorch or L2-01 ML Paradigms.