jeremydolan.net / AI safety research links
I. Frontier labsAnthropic
Research
Safety teams
Model docs
Safety framework
Landmark papers
OpenAI
Research
Safety teams
Model docs
Safety framework
Landmark papers
Google DeepMind
Research
Model docs
Safety framework
Landmark papers
For other developers, METR maintains links to their safety frameworks , including prior versions.
II. Independent organizationsEvaluations and red-teaming
Alignment, control, and interpretability
Governance and strategy
III. Reference
Capabilities
Epoch AI
Artificial Analysis
Evidence synthesis
Incidents
Safety frameworks
Regulation
European Union
California
New York
Lab scorecards
Forums
Training
Mentored research
Courses
Tooling
Eval tools
Interpretability tools
IV. Landmark workResearch agendas and position papers
Forecasts and strategy
Today’s computers, intelligent machines and our future Moravec, 1978 (the idea behind Moravec’s paradox )
Draft report on AI timelines Cotra, 2020 (biological anchors ) The ‘most important century’ series Karnofsky, 2021 What 2026 looks like Kokotajlo, 2021
Situational awareness: the decade ahead Aschenbrenner, 2024 (OOMs, unhobbling, the Project ) A narrow path Miotti et al., 2024 Superintelligence strategy Hendrycks, Schmidt, and Wang, 2025 (MAIM ) AI as normal technology Narayanan and Kapoor, 2025 The intelligence curse Drago and Laine, 2025 Pacing the frontier: a framework and research agenda Douglas et al., 2026
Essays and talks
Altman
Amodei
Andreessen
Buterin
Clark
Gwern
Hinton
Karpathy
Patel
Sutskever
Sutton
Toner
Yudkowsky
Open letters and statements
V. Objective misalignmentAI systems that pursue objectives other than the ones their designers intended. The failure is competently doing the wrong thing, not failing to do things well.
Failures of objective specification
The objective given to the system is incorrect or exploitable; successfully optimizing it produces unintended outcomes.Specification gaming, reward hacking, reward tampering, reward model (proxy) overoptimization
Concrete problems in AI safety Amodei et al., 2016 Reward hacking as a safety problem Faulty reward functions in the wild Clark and Amodei, 2016 The boat-racing exploit; misspecified rewards Specification gaming examples in AI Krakovna, 2018 A catalog of objective exploitation Reward tampering problems and solutions in reinforcement learning Everitt et al., 2019 Formalizes the incentives for reward tampering Specification gaming: the flip side of AI ingenuity Krakovna et al., 2020 A broad taxonomy and conceptual synthesis TruthfulQA: measuring how models mimic human falsehoods Lin, Hilton, and Evans, 2021 Imitative falsehoods: models repeat human misconceptions, larger models more often Scaling laws for reward model overoptimization Gao, Schulman, and Hilton, 2022 Quantifies proxy overoptimization in RLHF Open problems and fundamental limitations of reinforcement learning from human feedback Casper et al., 2023 Where the RLHF objective falls short, and which limits are fundamental Sycophancy to subterfuge: investigating reward-tampering in large language models Denison et al., 2024 Simple gaming → reward tampering Recent frontier models are reward hacking METR, 2025 Frontier models exploit scoring bugs on evaluation tasks while acknowledging it’s cheating Training a misaligned reward seeker Qi et al., 2026 Reward hacking → harmful agentic behavior
Failures of objective learning
The system learns to pursue an unintended objective even when the training objective is correct: the mismatch is in what is learned, not in what is specified.Goal misgeneralization, mesa-optimization, inner alignment failures
Risks from learned optimization in advanced machine learning systems Hubinger et al., 2019 Mesa-optimization; inner vs. outer alignment Goal misgeneralization in deep reinforcement learning Langosco et al., 2021 Wrong goals despite correct rewards Why AI alignment could be hard with modern deep learning Cotra, 2021 Saints, sycophants, and schemers can score equally well in training A central AI alignment problem: capabilities generalization, and the sharp left turn Soares, 2022 Capabilities generalize further than alignment Reward is not the optimization target Turner, 2022 Reward shapes which behavior gets reinforced; it needn’t become what the model wants The alignment problem from a deep learning perspective Ngo, Chan, and Mindermann, 2022 Learned goals, deception, and power-seeking Goal misgeneralization: why correct specifications aren’t enough for correct goals Shah et al., 2022 Goal vs. capability generalization Emergent misalignment: narrow finetuning can produce broadly misaligned LLMs Betley et al., 2025 Narrow training → broad misalignment Natural emergent misalignment from reward hacking in production RL MacDiarmid et al., 2025 Reward hacking → broader misalignment and sabotage
Instrumental incentives
Pursuing almost any objective can create incentives for unwanted behavior.Instrumental convergence, power-seeking, resistance to correction, deception
The basic AI drives Omohundro, 2007 Instrumental drives: self-preservation, resource acquisition The superintelligent will Bostrom, 2012 The orthogonality thesis; instrumental convergence Corrigibility Soares et al., 2015 Accepting correction despite incentives against it What failure looks like Christiano, 2019 Optimizing measurable proxies; influence-seeking behavior Optimal policies tend to seek power Turner et al., 2019 Formal conditions for instrumental power-seeking Is power-seeking AI an existential risk? Carlsmith, 2021 A six-premise argument for catastrophe from power-seeking AI, with probability estimates Scheming AIs: will AIs fake alignment during training in order to get power? Carlsmith, 2023 How likely training is to produce models that fake alignment to gain power later Sleeper agents: training deceptive LLMs that persist through safety training Hubinger et al., 2024 Deceptive backdoors that survive safety training Frontier models are capable of in-context scheming Meinke et al., 2024 Demonstrations of prompted scheming Alignment faking in large language models Greenblatt et al., 2024 Strategic compliance to preserve preferences Agentic misalignment: how LLMs could be insider threats Lynch et al., 2025 Blackmail and sabotage to avoid replacement in simulated settings
Objective uncertainty and preference learning
Human intentions can’t always be fully expressed in a fixed specification, so rewards, instructions, and feedback may need to be treated as imperfect evidence of what people actually want.Cooperative inverse reinforcement learning, inverse reward design, underspecification
Cooperative inverse reinforcement learning Hadfield-Menell et al., 2016 Alignment as a cooperative game in which the AI is uncertain of the human’s reward The off-switch game Hadfield-Menell et al., 2016 Uncertainty about the objective gives an incentive to allow shutdown Deep reinforcement learning from human preferences Christiano et al., 2017 Learning goals too complex to write down from human judgments: the origin of RLHF
Inverse reward design Hadfield-Menell et al., 2017 The designer’s reward as evidence of the intended objective Occam’s razor is insufficient to infer the preferences of irrational agents Armstrong and Mindermann, 2017 Behavior can’t pin down preferences without assumptions about rationality Incomplete contracting and AI alignment Hadfield-Menell and Hadfield, 2018 Contracts are always incomplete; humans fill the gaps with shared context and norms Human compatible: artificial intelligence and the problem of control Russell, book, 2019 The case for machines that are uncertain about human preferences Underspecification presents challenges for credibility in modern machine learning D’Amour et al., 2020 Equivalent training performance, divergent deployment behavior
VI. The OpenAI–Hugging Face incident
Hugging Face
OpenAI talk
Reports
Public evidence
VIII. Podcasts and newsletters
Podcasts
The Ezra Klein Show: AI safety episodes
Intelligence isn’t power Arvind Narayanan, October 2026 ‘This is nuts.’ An OpenAI insider explains why he quit David Robinson, October 2026 Bill Gates’s blunt warning on A.I. Bill Gates, September 2026 The ‘But China!’ dilemma driving the A.I. race Matt Sheehan, September 2026 The A.I.s are already out of control Helen Toner, August 2026 How afraid of the A.I. apocalypse should we be? Eliezer Yudkowsky, October 2025 The government knows AGI is coming Ben Buchanan, March 2025 What if Dario Amodei is right about A.I.? Dario Amodei, April 2024 Freaked out? We really can prepare for A.I. Kelsey Piper, March 2023 Is A.I. the problem? Or are we? Brian Christian, June 2021
Newsletters