Jan Leike

Alignment team lead at OpenAI

Research Associate at the Future of Humanity Institute

jan@openai.com

@janleike

My Research

My research aims to make progress on the alignment problem. The central question I’m studying is:

How can we design competitive, scalable, and general machine learning algorithms that act in accordance with their user’s intentions?

I lead the Alignment Team at OpenAI. Our approach to alignment research has three pillars:

1. Training AI systems using human feedback
2. Training AI systems to assist human evaluation
3. Training AI systems to do alignment research

Sometimes I write down thoughts about alignment research on my blog.

Publications

My publication list on Google Scholar

Alignment of large language models

Since joining OpenAI in 2021, my focus has been on training large language models to do what humans intend.

• Self-critiquing models for assisting human evaluators
  William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike. 2022.
• Training language models to follow instructions with human feedback
  Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe. Neural Information Processing Systems, 2022.
• Recursively summarizing books with human feedback
  Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano. 2021.

Reinforcement learning from human feedback

During my time at DeepMind 2016-2020, I worked on empirical research related to learning reward functions for deep reinforcement learning.

• Safe Deep RL in 3D Environments using Human Feedback
  Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg, Jan Leike, 2022.
• Quantifying Differences in Reward Functions
  Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. International Conference on Learning Representations (Spotlight), 2021.
• Pitfalls of Learning a Reward Function Online
  Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. International Joint Conference on Artificial Intelligence, 2021.
• Learning Human Objectives by Evaluating Hypothetical Behavior
  Siddharth Reddy, Anca D Dragan, Sergey Levine, Shane Legg, and Jan Leike. International Conference on Machine Learning, 2020. Blog post.
• Scalable agent alignment via reward modeling: a research direction
  Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Blog post. Video.
• Reward learning from human preferences and demonstrations in Atari
  Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Neural Information Processing Systems, 2018.
• Learning to understand goal specifications by modelling reward
  Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. International Conference on Learning Representations, 2019.
• AI Safety Gridworlds
  Jan Leike, Miljan Martic, Victoria Krakovna, Pedro Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017. Blog post. Video.
• Deep Reinforcement Learning from Human Preferences
  Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Neural Information Processing Systems, 2017. Blog post. Video.

Theoretical reinforcement learning

I wrote my PhD thesis on general reinforcement learning, reinforcement learning in non-ergodic, partially observable environments. My most interesting results are that Bayesian reinforcement learning agents can misbehave drastically if given a bad prior, that Thompson sampling learns to act optimally in any environment, and a formal solution to an open problem in game theory. If this interests you, take a look at my short introduction to general reinforcement learning.

• Nonparametric General Reinforcement Learning.
  Jan Leike. PhD Thesis, 2016.
• Exploration Potential.
  Jan Leike. European Workshop on Reinforcement Learning, 2016.
• A Formal Solution to the Grain of Truth Problem.
  Jan Leike, Jessica Taylor, and Benya Fallenstein. Uncertainty in Artificial Intelligence, 2016.
• Thompson Sampling is Asymptotically Optimal in General Environments.
  Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Uncertainty in Artificial Intelligence, 2016. Best student paper award.
• Loss Bounds and Time Complexity for Speed Priors.
  Daniel Filan, Jan Leike, and Marcus Hutter. AI & Statistics, 2016.
• On the Computability of Solomonoff Induction and Knowledge-Seeking.
  Jan Leike and Marcus Hutter. Algorithmic Learning Theory, 2015.
• Solomonoff Induction Violates Nicod’s Criterion.
  Jan Leike and Marcus Hutter. Algorithmic Learning Theory, 2015.
• Sequential Extensions of Causal and Evidential Decision Theory.
  Tom Everitt, Jan Leike, and Marcus Hutter. Algorithmic Decision Theory, 2015. Source code to the examples.
• On the Computability of AIXI.
  Jan Leike and Marcus Hutter. Uncertainty in Artificial Intelligence, 2015.
• Bad Universal Priors and Notions of Optimality.
  Jan Leike and Marcus Hutter. Conference on Learning Theory, 2015.
• A Definition of Happiness for Reinforcement Learning Agents.
  Mayank Daswani and Jan Leike. Artificial General Intelligence, 2015.
• Indefinitely Oscillating Martingales.
  Jan Leike and Marcus Hutter. Algorithmic Learning Theory, 2014.

Software Verification

During my Master’s degree at the University of Freiburg I developed the termination analysis tool Ultimate LassoRanker together with Matthias Heizmann. This tool can automatically prove termination and nontermination properties of C programs. It won two second places and two first places in the termination category of the SV-COMP from 2015 to 2018. The following papers are mostly related to that work.

• Geometric Nontermination Arguments.
  Jan Leike and Matthias Heizmann. Tools and Algorithms for the Construction and Analysis of Systems, 2018.
• Ranking Templates for Linear Loops.
  Jan Leike and Matthias Heizmann. Logical Methods in Computer Science, 2015.
• Geometric Series as Nontermination Arguments for Linear Lasso Programs.
  Jan Leike and Matthias Heizmann. International Workshop on Termination, 2014.
• Ranking Templates for Linear Loops.
  Jan Leike and Matthias Heizmann. Tools and Algorithms for the Construction and Analysis of Systems, 2014.
• Synthesis for Polynomial Lasso Programs.
  Jan Leike and Ashish Tiwari. Verification, Model Checking, and Abstract Interpretation, 2014. Source code to the experiments.
• Linear Ranking for Linear Lasso Programs.
  Matthias Heizmann, Jochen Hoenicke, Jan Leike, and Andreas Podelski. Automated Technology for Verification and Analysis, 2013.
• Ranking Function Synthesis for Linear Lasso Programs.
  Jan Leike. Master’s Thesis. University of Freiburg, 2013.

Sometimes my papers contain technical errors; you can see my list of errata. If you find a mistake not listed there, please let me know!