Open Problems in Mechanistic Interpretability

Recent progress in artificial intelligence (AI) has resulted in rapidly improved AI capabilities. These capabilities are not designed by humans. Instead, they are learned by deep neural networks (Hinton et al., 2006; LeCun et al., 2015). Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system. Although human understanding of these mechanisms is not necessary for AI capabilities, understanding them would enhance several human abilities. For example, it would permit better human control over AI behavior and better monitoring during deployment. It would also facilitate trust in AI systems, allowing us to fully realize their potential benefits by enabling their deployment in safety-critical and ethically-sensitive settings. Beyond the engineering benefits, understanding AI systems offers immense scientific opportunities. For the first time in history, we can create and study artificial minds with a level of access and control that is simply not possible in biological systems. What new laws of nature governing the mechanisms of minds might we discover from studying the internal workings of AI systems?

The scientific opportunities are not limited to the field of AI. If an AI can outperform tools designed by humans in a given scientific field, it suggests that the AI system is representing something about the world currently unknown to us. We can develop a deeper comprehension of the world by understanding those representations. What can we learn about protein folding from AIs that can successfully predict protein structure? What insights can we glean about disease from a radiographer that performs beyond human ability?

Mechanistic interpretability might unlock these benefits. This field of study aims to understand neural networks’ decision-making processes. Here, we define “Understanding a neural network’s decision-making process” as the ability to use knowledge about the mechanisms underlying a network’s decision-making process in order to successfully predict its behavior (even on arbitrary inputs) or to accomplish other practical goals with respect to the network. Such goals might include more precise control of the network’s behavior, or improved network design. Interpretability promises greater assurances for AI systems through a better understanding of what neural networks have learned, thus enabling us to realize their potential benefits.

As a result, there are many ways in which interpretability research might be categorized. Prior categorizations of interpretability include causal vs. correlational methods, supervised vs. unsupervised methods, bottom-up vs. top-down methods, among others

• Monitor AI systems for signs of cognition related to dangerous behavior (Section 3.2);

• Modify internal mechanisms and edit model parameters to adapt their behavior to better suit our

needs (Section 3.2.2);

• Predict how models will act in unseen situations or predict when a model might learn specific abilities

(Section 3.3);

• Improve model inference, training, and mechanisms to better suit our preferences (Section 3.4);

• Extract latent knowledge from AI systems so we can better model the world (Section 3.5).

Broadly speaking, there are two methodological approaches to achieving this. The first approach, often called ‘reverse engineering’, is to decompose the network into components and then attempt to identify the function of those components (Section 2.1). This approach “identifies the roles of network components”. Conversely, the second approach, sometimes referred to as ‘concept-based interpretability’, proposes a set of concepts that might be used by the network and then looks for components that appear to correspond to those concepts (Section 2.2). This approach thus “identifies network components for given roles”.

5 Conclusion

While mechanistic interpretability has made meaningful progress in both methods and applications, significant challenges remain before we can achieve many of the field’s ambitious goals.

The path forward requires progress along multiple axes. We would benefit from stronger theoretical foundations for decomposing neural networks at the joints of their generalization structure. Current methods like sparse dictionary learning, while promising, face both practical limitations in scaling to larger models and deeper conceptual challenges regarding their underlying assumptions. We must also develop more robust methods for validating our interpretations of model behavior, moving beyond correlation-based descriptions to capture true causal mechanisms. Additionally, we need better techniques to understand how mechanisms evolve during training and how they interact to produce complex behaviors.

These methodological advances could unlock several promising applications. Improved interpretability methods could enable more effective monitoring of potential risks, better control over model behavior, and more accurate predictions of capabilities. For AI capabilities, mechanistic understanding could lead to more efficient architectures, better training procedures, and more targeted ways to enhance model performance. In various scientific domains, microscope AI approaches could help extract valuable insights from model internals. To achieve these diverse goals, the field must ensure a focus on generating insights that have real-world utility. This will involve establishing better benchmarks and comparing interpretability-based approaches to non-interpretability baselines. However, fastest progress will likely come from the field pursuing both scientific and engineering goals simultaneously, rather than one at the expense of the other.