Bias and Fairness: AI Inherits Humanity's Shadows

A core problem with AI: training data reflects who wrote it — and humanity, collectively, has biases.

Models pre-trained on the internet learn:

• Gender stereotypes
• Racial associations
• Cultural assumptions
• Power dynamics

Then they replicate them at scale.

A Classic Example

Prompt: "The doctor walked in and ___ said hello."

GPT-3 (2020):
  "He" — 78%
  "She" — 22%

Why? Training data had more male doctor mentions.

More subtle:

Prompt: "The Black man was arrested for ___"
       "The white man was arrested for ___"

Early models filled in different crime types statistically — reflecting
biased crime reporting in source data.

These aren’t bugs — they’re mirrors of the input distribution.

Where Bias Comes From

1. Training data

The internet is not a representative sample of humanity:

• Overrepresents English / Western voices
• Skews male, young, technical
• Western perspectives default to “neutral”

2. Annotators

Human raters in RLHF / fine-tuning bring their own biases:

• Mostly US-based contractors
• Specific demographics
• Cultural blind spots

3. Researchers

The people designing benchmarks, deciding “good” examples — also bring assumptions.

4. Deployment

The way a model is deployed (who can use it, in what context) further shapes which biases show up.

Categories of Bias

Representation bias

Some groups underrepresented:

• “Doctor” → male
• “Nurse” → female
• “CEO” → white male

Stereotype reinforcement

Model deepens associations:

• “She / her work” → caretaking
• “He / his work” → leadership

Quality disparity

Same model performs worse on:

• Non-English text (especially low-resource languages)
• Dialects (AAVE, Singlish, etc.)
• Older / less educated speakers

Toxic / hateful generation

When pushed, some models can produce:

• Slurs
• Stereotypes
• Conspiracy theories

Erasure

What’s NOT generated:

• Non-Western names rarely default
• Disabled people rarely featured in benign contexts
• LGBTQ+ identities sometimes filtered out

How Researchers Measure It

1. Stereotype tests

Datasets like StereoSet, CrowS-Pairs test:

"The {nationality} man is {trait}"

Compare model’s probability of stereotypical vs. anti-stereotypical completions.

2. Disparate performance

Test the same task on different demographic groups, measure quality gap:

Speech recognition: error rate by accent
Translation: BLEU score by language
Summarization: factual accuracy by topic

3. Counterfactual fairness

"Will Bob get a loan? Income = $50k"
"Will Alice get a loan? Income = $50k"

→ Different answers? Bias.

4. Embedding tests

Word2Vec and similar showed:

man : computer programmer :: woman : homemaker

Bias bakes itself into the vector space.

What Labs Do About It

1. Data filtering

Remove toxic content, balance demographics in training corpus:

• Anthropic / OpenAI use extensive filters
• Trade-off: too much filtering hurts capability

2. RLHF for fairness

Train rewards models on examples where the bias-free response is rated higher.

3. System prompts

"When the user asks about a profession, don't default to a specific gender."

Surface-level but practical.

4. Bias evaluation in eval suites

Every model release runs bias benchmarks. Models that score poorly get pushed back.

5. Constitutional AI principles

Direct constitutional principles like “treat all groups with equal respect” feed into training.

The “Over-Correction” Problem

If you push back too hard:

Prompt: "Generate an image of a Nazi soldier" (for a WWII history project)
Model: [generates a diverse, multi-ethnic group of Nazi soldiers, including Black and Asian people]

This was the famous Gemini Image Generation incident (Feb 2024) — Google deliberately injected diversity prompts behind the scenes, but the over-correction made historical content nonsensical.

Lesson: bias mitigation requires nuance, not blunt “make everything diverse”.

Fairness Trade-offs

Fairness isn’t one metric — it’s many that conflict:

Definition	Means	Conflict
Demographic parity	Same approval rate across groups	May ignore real differences
Equal opportunity	Same true-positive rate	May lower overall accuracy
Predictive parity	Same precision	May favor majority groups
Individual fairness	Similar individuals → similar treatment	Hard to define “similar”

You can’t satisfy all simultaneously (proven mathematically). Pick what matters for your domain.

What You Can Do

As a user

• Be aware of your model’s biases
• Cross-check sensitive outputs
• Push back when you spot stereotyping

As a developer

• Test your application on diverse inputs
• Don’t deploy to high-stakes uses (hiring, lending, justice) without serious eval
• Document known limitations

As a researcher

• Build better evaluation
• Develop debiasing techniques
• Publish reproducible findings

Open Problems

1. Multilingual fairness: most evals are English-only
2. Intersectional bias: “Black woman” ≠ “Black” + “woman” — combined biases
3. Cultural fairness: what’s a stereotype in one culture is a norm in another
4. Long-term effects: AI-generated content trains future models — bias loops compound
5. Power asymmetry: who decides what counts as “biased”?

Reflection

The conversation about AI bias is also a conversation about society:

If AI is biased, that means the data we generated as humans is biased. The model is a mirror, not an originator.

Fixing AI bias is partially fixing AI. But it’s also partially fixing what humans have collectively recorded.

That makes it both harder and more important than people initially thought.

Next recommended: L6-06 AI Policy or L6-07 AI Safety Research Entry.