AI Safety Research: How to Enter the Field

L6-01 through L6-06 covered “what AI safety is and why”. This piece: how to participate and start.

AI safety is one of the rarest talent areas in 2026 — and one of the most worthwhile to invest in.

Who’s Doing AI Safety

The field splits into several communities:

1. Industrial Labs

Major companies’ alignment teams:

Company	Team	Focus
Anthropic	Alignment Science / Frontier Red Team	CAI / Mech Interp / red team
OpenAI	Safety Systems / Preparedness	RLHF / jailbreak defense / risk assessment
Google DeepMind	Frontier Safety / Responsible AI	RAI / evals / policy
Meta	AI Safety Research	Open-source alignment / red team
Microsoft	AI Responsibility	Compliance / governance / product safety

These teams actively hire but bar is high.

2. Non-Profit / Independent

Organization	Focus
MIRI	AGI risk / alignment theory
CHAI (Berkeley)	Value alignment
METR	Dangerous capability evaluation
Conjecture	Mech Interp
Apollo Research	Red team + alignment evals
Redwood Research	Interpretability + red team
AI Safety Institute (UK)	Government-level safety eval
US AI Safety Institute	US government safety standards

3. Academic

University AI safety labs:

• Stanford HAI
• Berkeley AI Safety
• Oxford FHI (reorganized)
• MIT CSAIL
• Cambridge CSER

4. Policy / Governance

• Center for AI Safety (policy influence)
• AI Now Institute
• Various national think tanks

Main Research Directions

L6 earlier articles cover these — here’s a recap:

1. Alignment

• RLHF / DPO / CAI improvements
• Value learning
• Preference elicitation
• Aligning emergent behavior

Suited for: ML engineers + philosophy background

2. Mechanistic Interpretability

• Understanding model internals
• SAE / probing / circuit analysis
• Finding “neurons” for specific behavior

Suited for: strong math + experimental patience

3. Evaluation

• Dangerous capability benchmarks
• Jailbreak testing
• Alignment evals
• Red team engineering

Suited for: engineering + creativity

4. Formal Methods

• Mathematical proofs of alignment properties
• Verification / model checking
• Safety guarantees

Suited for: CS theory + formal methods background

5. Governance / Policy

• AI law
• International coordination
• Standard-setting
• Impact assessment

Suited for: law / policy / economics background

6. Long-term Risk

• AGI risk modeling
• Value lock-in problems
• Control theory
• Philosophical foundations

Suited for: philosophy / math / interdisciplinary

Starter Projects

If you want to actually start doing AI safety:

Beginner (1-2 weeks)

1. Reproduce RLHF / DPO

Train a 7B model with TRL library:

# Use open preference data (e.g., UltraFeedback)
# Train a DPO model
# Compare to SFT baseline
# See helpful vs. harmful trade-off

Learn: alignment methods in practice.

2. Find a concept with SAEs

# Load GPT-2 small + public SAE
# Find "Golden Gate Bridge" feature
# Test that amplifying/suppressing it affects output
# Write a report

Learn: mech interp workflow.

3. Build a Jailbreak Benchmark

# Collect known jailbreak prompts
# Test against multiple LLMs
# Quantify "which model is most jailbreakable"
# Classify jailbreak patterns

Learn: red team + evaluation.

Intermediate (1-3 months)

1. Empirical research: sycophancy

• Design prompts that test “agreeing with users”
• Quantify sycophancy in different models
• Write a workshop paper

2. SAE applications

• Train SAEs on medium model (7-13B)
• Find meaningful features
• Document findings

3. Evaluation methodology

• Design a new evaluation
• Test multiple models
• Publish a benchmark

Advanced (>3 months)

• Full research paper
• Apply for PhD / industrial lab

Must-Read Resources

Papers

Classics:

• Concrete Problems in AI Safety (Amodei 2016)
• AI Safety via Debate (OpenAI)
• Risks from Learned Optimization (Hubinger 2019)
• Sleeper Agents (Anthropic 2024)

Current:

• All of Anthropic’s alignment papers (continuously updated)
• OpenAI Safety reports
• DeepMind Frontier Safety Framework

Blogs / Forums

• AI Alignment Forum — community discussion
• LessWrong — philosophy + practice
• Anthropic Research — official
• Apollo Research blog
• Neel Nanda’s blog — mech interp

Books

• Human Compatible (Stuart Russell 2019) — intro philosophy
• The Alignment Problem (Brian Christian 2020) — popular
• Superintelligence (Nick Bostrom 2014) — long-term risk
• Building Trustworthy AI (various) — practical

Courses

• AI Safety Fundamentals (BlueDot Impact) — systematic intro, free
• MIT 6.S898 (AI Ethics)
• Stanford CS 521 (AI Safety)
• Cambridge AI Safety Camp — 1-week intensive

Applying for AI Safety Roles

Industrial Labs

Most competitive route. Typical path:

1. PhD in ML / Math (preferred)
2. Or strong engineering + a few alignment papers
3. MATS program (Berkeley training)
4. Apply directly: Anthropic, OpenAI, DeepMind alignment teams

Acceptance rate < 1% — needs long-term preparation.

Non-Profit

• METR / Apollo / Redwood — slightly higher acceptance
• Usually need: 1-2 demonstration projects
• Then research + paper output

Academic

PhD applications:

• Find alignment-focused advisor (Stuart Russell, Jacob Steinhardt, Aleksander Madry etc.)
• Strong ML background + 1 relevant paper

Funding / Grants

For independent research, funders include:

• OpenPhilanthropy — long-term support
• LTFF (Long-Term Future Fund)
• Survival and Flourishing Fund
• Manifund (newer)

Grant applications need:

• Clear research direction
• Prior work / credentials
• 1-2 year plan
• Budget

Typically $50K-500K per year of research funding.

Some Advice

Don’t start “too early”

AI safety is high-barrier:

• L1-L3 math + DL + Transformer first
• L4 LLM engineering
• Then L6

Can’t do alignment without understanding how LLMs work.

Pick your strength

If strong ML engineer → evals / red team If strong math → formal / mech interp If philosophy background → long-term risk / governance If PM / interdisciplinary → applied safety / policy

No single “correct” path — pick what fits + interests you.

Join the community

• AI Alignment Forum — read + comment
• AI Safety Camp (annual)
• Discord groups
• Find mentors

Isolated research is very slow — community is core.

Write + publish

Share your thinking:

• Blog posts
• Workshop papers
• Forum discussions

Your work has value only if seen — don’t keep work private.

A Real Trajectory

A friend’s actual path — ML engineer to Anthropic:

1. Year 0: ML engineer, doing LLM apps (no alignment background)
2. Year 1: read alignment papers in spare time + play with SAEs + write 2 blog posts
3. Year 1.5: MATS program (remote)
4. Year 2: 6 months research at Apollo Research
5. Year 2.5: published 1 alignment workshop paper
6. Year 3: hired by Anthropic Alignment Science team

3 years from 0 to Anthropic — not impossible.

Next recommended: L6-04 Mechanistic Interpretability or L6-01 Why Alignment Matters.