Jérémy Scheurer is a research scientist in the evaluations team at Apollo Research, an AI evaluations organization that does interpretability research. He recently published work on how “Large Language Models can Strategically Deceive their Users when Under Pressure.” Just before that, Jérémy contracted with OpenAI and worked in its dangerous capabilities evaluations team. Previously, Jérémy was a Research Scientist at FAR AI (and NYU), collaborating with Ethan Perez, where he published work on learning from language feedback. His research interests include understanding and predicting (dangerous) capabilities in large language models (LLMs), creating LM evaluations, developing the science of evals, evaluating and preventing deceptive alignment, and combining evals with interpretability methods.

My research is focused on measuring and reducing risks from LLMs:

• Developing realistic demonstrations of strategic deception and deceptive alignment.

• Developing techniques for process-based supervision, such as learning from language feedback.

• Understanding how data influences model capabilities.

• Understanding the limitations of RLHF, interpretability methods, and black-box audits.

I’m generally interested in projects that demonstrate dangerous capabilities, especially deceptive alignment (model organisms), and projects that try to understand, measure, or predict such capabilities. Separately, I want to push forward the science of evals, i.e., I want to develop the understanding that makes the field of evals a more rigorous science instead of an art that largely depends on specific prompts, etc. Towards developing a science of evals, I want to understand better how capabilities emerge, how supervised fine-tuning and RLHF change capabilities, how we can predict them, etc. Finally, I’m very interested in developing novel methods that combine interpretability and evaluations in order to go beyond black-box evaluations.

To better understand what kind of projects I’m generally interested in, I provide a non-exhaustive list of relevant work (in no particular order and leaving out many areas I’m also interested in):

• Large Language Models can Strategically Deceive their Users when Put Under Pressure

• Measuring and understanding Sycophancy

• Measuring situational awareness in LLMs

• How to Catch an AI Liar

• Large Language Models Represent Space and Time

• Scaling Laws for Neural Language Models

As an indicative guide (this is not a score sheet), I evaluate candidates according to the following criteria:

• Engineering skills:

  • Background: Do you have a solid engineering foundation? Are you proficient in Python and familiar with key libraries like LLM APIs and Pytorch?

  • Machine Learning Experience: Have you fine-tuned open-source models, written high-quality, reproducible code, and engaged in empirical work?

  • Large Language Models (LLMs): Have you experimented with LLMs like

  • Engineering is the skill I probably value most overall. Empirically I have observed that being able to quickly implement clean experiments and run them, is an extremely valuable skill.

• Prompt engineering skills:

  • Have you experimented with LLMs like GPT-4 or Claude 3?

  • Do you understand their capabilities and limitations? Do you have good intuitions about how they solve problems?

  • A typical project with me will most likely involve “playing” around with LLMs for a while in order to get good intuitions on a certain question. Then, we will more quantitatively investigate/evaluate them and write code for large-scale evaluations etc. But a lot of the work will most likely involve interacting with LLMs.

  • While expertise is not required, a willingness to learn and experiment with LLMs is crucial.

• Science Background:

  • I seek candidates who demonstrate scientific thinking, can manage research projects, find solutions independently, ask relevant questions, seek necessary help, and understand related literature.

• Self-Drivenness:

  • Are you motivated and capable of advancing the project independently? Can you explore new ideas and avenues driven by curiosity?

  • If you have bugs or other technical issues, can you find solutions to them yourself? Can you ask for advice and help from other experts in the field?

  • You should not get the sense that you will be on your own. I will be fairly hands-on throughout the project and support you whenever you need help. However, I still think that being self-driven is an important trait to have and can determine a lot about the success of a project.

• Working in an Organized manner:

  • Structured Work: Can you work systematically, seek help when needed, and communicate your ideas clearly and efficiently?

  • Code and Documentation: Is your code well-organized for reproducibility? Can you consistently record and present your work?

  • Are you find with presenting slides every week with your updates?

  • I’m mostly highlighting this point since I like being organized and structured. If you prefer a more “random” approach of exploring various things at the same time (which can work really well for some people), I might not be a good fit.

Mentorship approach:

Expect a 1-hour weekly meeting with additional Slack communication. I'm available for extra meetings and quick Slack responses for urgent matters.

I will try to help you in your research and teach you what I know as much as I can. However, I point out that I don’t yet have a lot of experience mentoring people, which means I will most likely make mistakes. I am, however, very eager to improve, and you will thus have more influence on the mentor-mentee relationship. I’m looking forward to input on how we can make the most out of the mentorship.

My projects are very empirical and engineering-heavy. Other mentors might be a better fit if you want to do more conceptual, theoretical, or mathematical work. My projects can involve learning to execute well on scoped-out research projects (as a first step for getting into research). I have a few very concrete projects that I’d be excited for you to work on. However, I want you to be excited about your work since this is key to a successful project. If you have concrete projects that you would like to work on that fit into my stream, I’m happy to mentor you on those. If you know that you definitely want to work on your own project, I will, however, only be able to select you if I am interested/excited about the project and believe that my skills would add value to it.