If an AI nobody can look inside points scientists to a theory, can that theory still earn trust by passing ordinary tests?

This explores whether a theory can earn real support when the thing that produced it, usually an opaque AI model, can't be inspected. The corpus says yes, with one condition: the theory has to be checked by something other than the black box. Philosophers of science have long separated how an idea is found from how it is defended. One note argues that opacity only becomes a problem when you treat the model's output as if it were already a justified claim Can opaque models guide discovery without needing interpretation?. If a model's accurate predictions point researchers toward a theory, and that theory then meets the field's normal standards of proof, it doesn't matter that nobody understands the model. Where an idea came from was never what made it true.

Terence Tao makes the same argument from mathematics Can opaque machine learning models help prove new mathematics?. A neural network proposed candidate solutions to a hard fluid-dynamics problem, and the results counted because mathematicians then confirmed them with standard proof techniques. The model is a scout, and a separate check, such as a proof assistant, a numerical method or a lab experiment, is what certifies the result. Biology offers a similar case. An AI co-scientist's top-ranked hypothesis matched a mechanism that a lab had already confirmed experimentally but not yet published Can AI systems generate hypotheses that match unpublished experimental discoveries?. That result is persuasive because the experiment came first and the AI's answer could be compared against it.

The obvious workaround for opacity is to ask the model to explain itself, and the corpus shows why that fails. Chain-of-thought explanations can be backdoored to produce fluent, convincing reasoning that leads to wrong answers Can chain-of-thought reasoning be secretly manipulated to look normal?. Plans planted in a model's context can show up rephrased as the model's 'own' reasoning Can reasoning models be steered by injected context without detection?. Some models even state that their answers are unbiased when they demonstrably aren't Do chain-of-thought traces falsely claim their answers are unbiased?. A model's account of its own reasoning is more output from the same black box. It doesn't count as independent evidence.

The harder risk is the checking step itself. One argument holds that citations, tidy logic and careful hedging used to signal real knowledge, but AI can now produce all of them, so using AI-style criteria to check AI makes the test circular Can we verify AI knowledge without using AI-generated tests?. A related framing treats AI output as structurally like hearsay: it can't be traced to a source or checked against one Does AI-generated knowledge have the same structure as hearsay?. The Foundation Priors framework draws the practical conclusion: treat LLM output as a starting assumption whose weight you set on purpose, not as observed data Should we treat LLM outputs as real empirical data?. People don't catch the difference on their own. In one study, readers with no information about sources couldn't tell fluent fabrications from true statements, and they recovered only when an interface showed which claims had been verified Can readers tell truth from fabrication without evidence signals?.

Taken together, these notes say opacity in where an idea came from is acceptable, but opacity in how it was checked is not. A black box can propose a theory, but justification has to come from a check that is independent and traceable: a proof, an experiment, or a verified source. The real danger is not black-box discovery. It is the gradual drift toward letting the same kind of system both propose and approve its own ideas.