Deep Learning Opacity in Scientific Discovery

Philosophers have recently focused on critical, epistemological challenges that arise from the opacity of deep neural networks. One might conclude from this literature that doing good science with opaque models is exceptionally challenging, if not impossible. Yet, this is hard to square with the recent boom in optimism for AI in science alongside a flood of recent scientific breakthroughs driven by AI methods.

In this paper, I argue that the disconnect between philosophical pessimism and scientific optimism is driven by a failure to examine how AI is actually used in science. I show that, in order to understand the epistemic justification for AI-powered breakthroughs, philosophers must examine the role played by deep learning as part of a wider process of discovery. The philosophical distinction between the ‘context of discovery’ and the ‘context of justification’ is helpful in this regard. I demonstrate the importance of attending to this distinction with two cases drawn from the scientific literature, and show that epistemic opacity need not diminish AI’s capacity to lead scientists to significant and justifiable breakthroughs.

Introduction. The recent boom in optimism for the use of deep learning (DL) and artificial intelligence (AI) in science is due to the astonishing capacity of deep neural networks to facilitate discovery [DV+18], overcome the complexity of otherwise intractable scientific problems [SE+20], as well as to both emulate and outperform experts on routine [CL+14], complex, or even humanly impossible [DF+22] tasks. In fact, nearly every empirical discipline has already undergone some form of transformation as a result of developments in and implementation of deep learning and artificial intelligence [ST+20]. To scientists and science funding agencies alike, artificial intelligence both promises and has already begun to revolutionize not only our science, but our society, and quality of life.

Yet, someone reading the recent philosophical literature on deep learning might be forgiven for concluding that doing good science with deep neural networks must be exceptionally challenging, if not impossible. This is because philosophers have, of late, focused not on the enormous potential of DL and AI, but on a number of important epistemological challenges that arise from the uninterpretability of deep neural networks (DNNs). Given that DNNs are epistemically opaque [Cre20, Hum09, Zer22, Lip18], it is, in many instances, impossible to know the high-level, logical rules that govern how the network relates inputs to outputs. It is argued that this lack of transparency severely limits scientists’ ability to form explanations for [Cre20, Zer22] and understanding of [Sul19] why neural networks make the suggestions that they do. For instance, Creel states that “access to only observable inputs and outputs of a completely opaque black-box system is not a sufficient basis for explanation[.]” [Cre20, pg.573]. So, this presents an obvious epistemic challenge when explanations of neural network logic are required to justify claims or decisions made on the basis of their outputs.

As a result, reading the recent philosophical literature can leave one wondering on what basis (beyond mere inductive considerations) neural network outputs can be justified [Bog21]. While lack of interpretability is of particular concern in high-stakes decisionmaking settings where accountability and value-alignment are salient (e.g., medical diagnosis and criminal justice) [BC+22, FS+21, Hof17], the opacity of deep learning models may also be of concern in basic research settings where explanations and understanding represent central epistemic virtues and often serve as justificatory credentials [Kha17]. Even if inductive considerations such as the past success of the model on out of sample data can help to raise confidence in the scientific merit of its outputs, in general, without additional justification, it is unclear how scientists can ensure that results are consistent with the epistemic norms of a given discipline.1 Or, so we are led to conclude.

There is, then, a sharp contrast between the relative optimism of scientists and policymakers on the one hand, and the pessimism of philosophers on the other, concerning the use of deep learning methods in science. This disconnect is due, I believe, to a failure on the part of philosophers to attend to the full range of ways that deep learning is actually used in science.2 In particular, while philosophers are right to examine and raise concern over epistemological issues that arise as a result of neural network opacity, it is equally important to step back and analyze whether these issues do, in fact, arise in practice and, if so, in what contexts and under what conditions.

In this paper, I argue that epistemological concerns due to neural network opacity will arise chiefly when network outputs are treated as scientific claims that stand in need of justification (e.g., treated as candidates for scientific knowledge, or treated as the basis for high-stakes decisions). It is reasonable to think that this must happen quite a bit, particularly outside of scientific settings. After all, the promise of deep learning is the rapid discovery of new knowledge. Of course, philosophers are correct that, if neural network outputs are evaluated in, what has often been referred to as, the “context of justification”, then access to the high-level logic of the network (e.g., interpretability) will, in most cases, be required for validation.3 While this certainly happens, I will show that scientists can make breakthrough discoveries and generate new knowledge utilizing fully opaque deep learning without raising any epistemological alarms. In fact, scientists are often well aware of the epistemological limitations and pitfalls that attend the use of black-box methods. But, rather than throw up their arms and embrace a form of pure instrumentalism (or worse, bad science), they can carefully position and constrain their use of deep learning outputs to what philosophers of science have called the “context of discovery”.

Related work. Zerilli [Zer22] describes the opacity of deep learning models (DLMs) by bringing out the distinction between “Tractability”, “Intelligibility”, and “Fathomability”, a distinction echoed in [Lip18]. Here, the idea is that any working machine learning model is tractable in so far as it can be run on a computer. However, intelligibility comes in degrees that are modulated by model fathomability. Fathomability is understood to be the extent to which a person can understand, straight away, how the model relates features to produce outputs. As a result, the more complex a model (e.g., increased dimensionality, extreme nonlinearities, etc.), the less fathomable it becomes. Many highly complex but linear models (e.g., random forests) remain “intelligible” in so far as all of the relationships between elements of the model can, in principle, be semantically deciphered even though the model as a whole (its overall decision logic) remains unfathomable due to complexity.4 Zerilli’s three aspects of epistemic access to neural network logic mirror Creel’s three levels or granular scales of transparency [Cre20]. For Creel, the transparency of a complex, computational model can be assessed “Algorithmically”, “Structurally”, and at “Runtime”. Most relevant to the issues of this paper are algorithmic and structural transparency. For Creel, a model is algorithmically transparent if it is possible to establish which high-level, logical rules (e.g., which algorithm) govern the transformation of input to output. In the case of a deep neural network, it is not possible to know which algorithm is implemented by the network precisely because the algorithm is developed autonomously during training. As a result, DNNs also lack what Creel calls “structural” transparency in that it is not clear how the distribution of weights and (hyper)parmeterization of the neural network implements (realizes) the algorithm that it has learned. Therefore, for Creel, DNNs are opaque —neither “fathomable” nor “intelligible” in Zerilli’s sense.5 Following Humphreys [Hum04, Hum09], a process is said to be epistemically opaque when it is impossible for a scientist to know all of the factors that are epistemically relevant to licensing claims on the basis of that process, where factors of ‘epistemic relevance’ include those falling under Creel’s algorithmic and structural levels and Zerilli’s intelligibility and fathomability criteria. As such, DLMs are “epistemically opaque”.

Method. Deep learning is a machine learning technique based on artificial neural networks that is widely used for prediction and classification tasks [LBH15]. The goal of deep learning is to automate the search for a function ˆf that approximates the true function f that generates observed data. The fundamental assumption that motivates the use of deep learning is that f is in the set of functions F representable by a neural network given some particular architecture and (hyper)parameterization. Of course, for any given parameterization k, we have no way of knowing a priori whether f ∈Fk. However, deep neural networks are universal approximators [HSW89], so the assumption is at least principled.

When it comes to justifying belief or trust in the outputs of deep learning models, their epistemic opacity is straightforwardly problematic. This is due to the fact that it is not possible to evaluate all of the epistemically relevant factors that led to the output. In high-stakes settings such as medical diagnosis, where the output of an epistemically opaque model forms the basis of a decision, a decision maker’s inability to explain why the model prompts the decision that it does (and not, say, some other decision) can raise reasonable doubt as to whether the decision is, in fact, justified. Here is Creel on why we Why might scientists (and others) all need and require transparency? The reasons Creel and most philosophers6 concerned with the epistemology of deep learning give are that, without transparency, scientists are unable to understand the outputs of their models, are powerless to explain why the models perform the way they do, cannot provide justification for the decisions they make on the basis of the model output, are uncertain whether and to what extent the models reflect our values —on and on. What all of these reasons have in common is a commitment to the idea that neural network transparency is epistemically essential to effectively use and gain knowledge from powerful artificial intelligence applications in scientific and societal settings [GS+19].

Be that as it may, the outputs of epistemically opaque models need not be treated in this way. Rather, they can serve as aspects or parts of a process of discovery. While the process ultimately leads to claims that stand in need of justification, the part played by an opaque model in that process can, itself, be epistemically insulated from the strong sort of evaluation that is applied to findings in the context of justification. In this way, neural network outputs can serve to facilitate discovery without their outputs or internal logic standing in need of justification. That is, neural network outputs that serve as parts of a process of discovery (similar to abduction [DE21, Han65] and problem-solving heuristics [Wim07, Sim73]) can be treated as situated in the “context of discovery”.8 In the context of discovery, the outputs of neural networks can be used to guide attention and scientific intuition toward more promising hypotheses but do not, themselves, stand in need of justification. Here, outputs of opaque models serve to provide reasons to or evidence for pursuit of particular paths of inquiry over others (see: Figure 1). As such, they both provide and are subject to forms of preliminary appraisal [Sch93], but, as the cases in Section 4 will bring out, the mere inductive support DLMs provide is epistemically sufficient to guide pursuit.

Here, I consider a case from low-dimensional topology in which researchers use deep learning to guide mathematical intuition concerning the relationship between two classes of properties of low-dimensional knots. Knots are particularly interesting topological objects because the relationships between their numerous properties are not well understood, and their various connections to other fields within mathematics are plausible but unproven.

Discussion. One might object to the claim that the opacity of the network in this case was epistemically irrelevant. After all, the gradient based saliency method used to isolate the contribution to accuracy of the various inputs might be viewed as an interpretive step. While this objection is well taken, it is important to note that the saliency procedure In this case, a fully opaque DLM has had profound implications for our theoretical understanding of earthquake dynamics. Namely, the ability to accurately predict phenomena orients scientific attention to empirical desiderata necessary for more accurate theory building. Moreover, it is epistemically irrelevant to justifying the improved theory that we cannot verify whether and how any of the geophysical quantities that were determined to be of relevance are, in fact, represented in the network. This is because it is not the network’s predictions that stand in need of justification but, rather, the theory’s itself. The reworked theory is justified in ways that are consistent with the norms of the discipline —it relates known geophysical properties in ways that are consistent with first principles, it aids in the explanation and understanding of aftershock dynamics, and it outperforms extant theory in prediction. Yet, none of this depends on the neural network that was used to lead attention to relevant revisions of the theory for justification.11

Conclusion. What I hope to have shown in this paper is that, despite their epistemic opacity, deep learning models can be used quite effectively in science, not just for pragmatic ends but for genuine discovery and deeper theoretical understanding, as well. This can be accomplished when DLMs are used as guides for exploring promising avenues of pursuit in the context of discovery. In science, we want to make the best conjectures and pose the best hypotheses that we can. The history of science is replete with efforts to develop processes for arriving at promising ideas. For instance, thought experiments are cognitive devices for hypothesis generation, exploration, and theory selection. In general, we want our processes of discovery to be as reliable or trustworthy as possible. But, here, inductive considerations are, perhaps, sufficient to establish reliability. After all, the processes by which we arrive at our conjectures and hypotheses do not typically serve also to justify them. While philosophers are right to raise epistemological concerns about neural network opacity, these problems primarily concern the treatment and use of deep learning outputs as findings in their own right that stand, as such, in need of justification which (as of now) only network transparency can provide. Yet, when DLMs serve the more modest (though no less impactful) role of guiding science in the context of discovery, their capacity to lead scientists to significant breakthroughs is in no way diminished.