Neel Nanda

According to our database, Neel Nanda authored at least 58 papers between 2021 and 2025.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of four.

2025

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning.

Helena Casademunt, Caden Juang, Adam Karvonen, Samuel Marks, Senthooran Rajamanoharan, Neel Nanda

CoRR, July, 2025

Reasoning-Finetuning Repurposes Latent Representations in Base Models.

Jake Ward, Chuqiao Lin, Constantin Venhoff, Neel Nanda

CoRR, July, 2025

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.

Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca D. Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Madry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vladimir Mikulik

CoRR, July, 2025

Simple Mechanistic Explanations for Out-Of-Context Reasoning.

Atticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda

CoRR, July, 2025

Thought Anchors: Which LLM Reasoning Steps Matter?

Paul C. Bogdan, Uzay Macar, Neel Nanda, Arthur Conmy

CoRR, June, 2025

Understanding Reasoning in Thinking Language Models via Steering Vectors.

Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda

CoRR, June, 2025

Because we have LLMs, we Can and Should Pursue Agentic Interpretability.

Been Kim, John Hewitt, Neel Nanda, Noah Fiedel, Oyvind Tafjord

CoRR, June, 2025

How Visual Representations Map to Language Feature Space in Multimodal LLMs.

Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, Neel Nanda

CoRR, June, 2025

Convergent Linear Representations of Emergent Misalignment.

Anna Soligo, Edward Turner, Senthooran Rajamanoharan, Neel Nanda

CoRR, June, 2025

Model Organisms for Emergent Misalignment.

Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, Neel Nanda

CoRR, June, 2025

Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models.

Patrick Leask, Neel Nanda, Noura Al Moubayed

CoRR, May, 2025

Towards eliciting latent knowledge from LLMs with mechanistic interpretability.

Bartosz Cywinski, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda

CoRR, May, 2025

Scaling sparse feature circuit finding for in-context learning.

Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Arthur Conmy, Neel Nanda

CoRR, April, 2025

Robustly identifying concepts introduced during chat fine-tuning using crosscoders.

Julian Minder, Clément Dumas, Caden Juang, Bilal Chugtai, Neel Nanda

CoRR, April, 2025

An Approach to Technical AGI Safety and Security.

Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, Rishub Jain, Rory Greig, Samuel Albanie, Scott Emmons, Sebastian Farquhar, Sébastien Krier, Senthooran Rajamanoharan, Sophie Bridgers, Tobi Ijitoye, Tom Everitt, Victoria Krakovna, Vikrant Varma, Vladimir Mikulik, Zachary Kenton, Dave Orr, Shane Legg, Noah D. Goodman, Allan Dafoe, Four Flynn, Anca D. Dragan

CoRR, April, 2025

Learning Multi-Level Features with Matryoshka Sparse Autoencoders.

Bart Bussmann, Noa Nabeshima, Adam Karvonen, Neel Nanda

CoRR, March, 2025

SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability.

Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, Neel Nanda

CoRR, March, 2025

Chain-of-Thought Reasoning In The Wild Is Not Always Faithful.

Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy

CoRR, March, 2025

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing.

Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, Neel Nanda

CoRR, February, 2025

Open Problems in Mechanistic Interpretability.

Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adrià Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, Tom McGrath

CoRR, January, 2025

Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control.

Aleksandar Makelov, Georg Lange, Neel Nanda

Proceedings of the Thirteenth International Conference on Learning Representations, 2025

Sparse Autoencoders Do Not Find Canonical Units of Analysis.

Patrick Leask, Bart Bussmann, Michael T. Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, Neel Nanda

Proceedings of the Thirteenth International Conference on Learning Representations, 2025

Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models.

Javier Ferrando, Oscar Balcells Obeso, Senthooran Rajamanoharan, Neel Nanda

Proceedings of the Thirteenth International Conference on Learning Representations, 2025

2024

Universal Neurons in GPT2 Language Models.

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, Dimitris Bertsimas

Trans. Mach. Learn. Res., 2024

BatchTopK Sparse Autoencoders.

Bart Bussmann, Patrick Leask, Neel Nanda

CoRR, 2024

Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks.

Adam Karvonen, Can Rager, Samuel Marks, Neel Nanda

CoRR, 2024

Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2.

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, Neel Nanda

CoRR, 2024

Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders.

Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, Neel Nanda

CoRR, 2024

Interpreting Attention Layer Outputs with Sparse Autoencoders.

Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, Neel Nanda

CoRR, 2024

Refusal in Language Models Is Mediated by a Single Direction.

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, Neel Nanda

CoRR, 2024

Improving Dictionary Learning with Gated Sparse Autoencoders.

Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda

CoRR, 2024

How to use and interpret activation patching.

Stefan Heimersheim, Neel Nanda

CoRR, 2024

AtP*: An efficient and scalable method for localizing LLM behaviour to components.

János Kramár, Tom Lieberum, Rohin Shah, Neel Nanda

CoRR, 2024

Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs.

Bilal Chughtai, Alan Cooney, Neel Nanda

CoRR, 2024

Confidence Regulation Neurons in Language Models.

Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, Neel Nanda

Proceedings of the Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, 2024

Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders.

Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda

Proceedings of the Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, 2024

Transcoders find interpretable LLM feature circuits.

Jacob Dunefsky, Philippe Chlenski, Neel Nanda

Proceedings of the Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, 2024

Refusal in Language Models Is Mediated by a Single Direction.

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda

Proceedings of the Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, 2024

Explorations of Self-Repair in Language Models.

Cody Rushing, Neel Nanda

Proceedings of the Forty-first International Conference on Machine Learning, 2024

Towards Best Practices of Activation Patching in Language Models: Metrics and Methods.

Fred Zhang, Neel Nanda

Proceedings of the Twelfth International Conference on Learning Representations, 2024

Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching.

Aleksandar Makelov, Georg Lange, Atticus Geiger, Neel Nanda

Proceedings of the Twelfth International Conference on Learning Representations, 2024

2023

Finding Neurons in a Haystack: Case Studies with Sparse Probing.

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, Dimitris Bertsimas

Trans. Mach. Learn. Res., 2023

Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching.

Aleksandar Makelov, Georg Lange, Neel Nanda

CoRR, 2023

Training Dynamics of Contextual N-Grams in Language Models.

Lucia Quirke, Lovis Heindrich, Wes Gurnee, Neel Nanda

CoRR, 2023

Linear Representations of Sentiment in Large Language Models.

Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, Neel Nanda

CoRR, 2023

Copy Suppression: Comprehensively Understanding an Attention Head.

Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, Neel Nanda

CoRR, 2023

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla.

Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, Vladimir Mikulik

CoRR, 2023

Neuron to Graph: Interpreting Language Model Neurons at Scale.

Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay B. Cohen, Fazl Barez

CoRR, 2023

N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models.

Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Fazl Barez

CoRR, 2023

A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations.

Bilal Chughtai, Lawrence Chan, Neel Nanda

Proceedings of the International Conference on Machine Learning, 2023

Progress measures for grokking via mechanistic interpretability.

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt

Proceedings of the Eleventh International Conference on Learning Representations, 2023

Emergent Linear Representations in World Models of Self-Supervised Sequence Models.

Neel Nanda, Andrew Lee, Martin Wattenberg

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2023

2022

Fully General Online Imitation Learning.

Michael K. Cohen, Marcus Hutter, Neel Nanda

J. Mach. Learn. Res., 2022

In-context Learning and Induction Heads.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah

CoRR, 2022

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, Jared Kaplan

CoRR, 2022

Predictability and Surprise in Large Generative Models.

Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Dario Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Chris Olah, Jack Clark

CoRR, 2022

Predictability and Surprise in Large Generative Models.

Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, Jack Clark

Proceedings of the FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21, 2022

2021

An Empirical Investigation of Learning from Biased Toxicity Labels.

Neel Nanda, Jonathan Uesato, Sven Gowal

CoRR, 2021