Rethinking Interpretability in the Era of Large Language Models

Interpretable machine learning has exploded as an area of interest over the last decade, sparked by the rise of increasingly large datasets and deep neural networks. Simultaneously, large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks, offering a chance to rethink opportunities in interpretable machine learning. Notably, the capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. However, these new capabilities raise new challenges, such as hallucinated explanations and immense computational costs. In this position paper, we start by reviewing existing methods to evaluate the emerging field of LLM interpretation (both interpreting LLMs and using LLMs for explanation). We contend that, despite their limitations, LLMs hold the opportunity to redefine interpretability with a more ambitious scope across many applications, including in auditing LLMs themselves. We highlight two emerging research priorities for LLM interpretation: using LLMs to directly analyze new datasets and to generate interactive explanations.

Introduction. Machine learning (ML) and natural language processing (NLP) have seen a rapid expansion in recent years, due to the availability of increasingly large datasets and powerful neural network models. In response, the field of interpretable ML* has grown to incorporate a diverse array of techniques and methods for understanding these models and datasets1–3. One part of this expansion has focused on the development and use of inherently interpretable models 4, such as sparse linear models, generalized additive models, and decision trees. Alongside these models, post- hoc interpretability techniques have become increasingly prominent, offering insights into predictions after a model has been trained. Notable examples include methods for assessing feature importance5, 6, and broader post-hoc techniques, e.g., model visualizations7, 8, or interpretable distillation9, 10. Meanwhile, pre-trained large language models (LLMs) have shown impressive proficiency in a range of complex NLP tasks, significantly advancing the field and opening new frontiers for applications11–13.

Discussion / Conclusion. In this paper, we have explored the vast and dynamic landscape of interpretable ML, particularly focusing on the unique opportunities and challenges presented by LLMs. LLMs’ advanced natural language generation capabilities have opened new avenues for generating more elaborate and nuanced explanations, allowing for a deeper and more accessible understanding of complex patterns in data and model behaviors. As we navigate this terrain, we assert that the integration of LLMs into interpretative processes is not merely an enhancement of existing methodologies but a transformative shift that promises to redefine the boundaries of machine learning interpretability. Our position is anchored in the belief that the future of interpretable ML hinges on our ability to harness the full potential of LLMs. To this end, we outlined several key stances and directions for future research, such as enhancing explanation reliability and advancing dataset interpretation for knowledge discovery.