Does AI scheming research rely on rigorous evidence or anecdote?

The paper argues that current research claiming AI systems are developing a capacity for "scheming" — defined as "covertly and strategically pursuing misaligned goals" — is "repeating some of the methodological errors that plagued ape language research in the 1960s and 1970s." It is explicit that this is not a claim that scheming is impossible: "Our goal here is not to dismiss the idea that AI systems may be 'scheming' or even that they might pose existential risks to humanity. On the contrary, it is precisely because we think these risks should be taken seriously that we call for more rigorous scientific methods to assess the core claims made by this community." The critique is methodological, not a rebuttal of the underlying risk.

The paper draws the analogy through three factors it says both fields share. First, a hype cycle: Roger Brown compared ape-language findings to "getting an S.O.S. from outer space," and current scheming claims are picked up by the press "often in lurid terms," with references to "SkyNet." Second, researcher motivated reasoning: the Gardners raised Washoe "like their child" and Patterson called herself Koko's "mother," while today's scheming papers come from "a small set of overlapping authors who are all part of a tight-knit community" concerned about AGI/ASI, creating "an ever-present risk of researcher bias and 'groupthink.'" Third, a lack of rigor — anecdote without baselines or control conditions. Ape-language research relied on subjective interpretation until Herb Terrace's frame-by-frame analysis of Nim revealed trainers were unconsciously cueing signs, a repeat of the Clever Hans effect. The paper makes the parallel concrete with the GPT-4/TaskRabbit CAPTCHA anecdote: widely cited as evidence of deceptive scheming, but "the researcher, not the AI, suggested using TaskRabbit," the researcher browsed the web on the model's behalf, and "the prompts and transcript are not publicly available." It also faults studies for lacking a null hypothesis: the finding that models omit mentioning a hint in their reasoning traces 20-30% of the time has no stated baseline for how often that would happen by chance.

This extends Does anthropomorphic misalignment research overinterpret model behavior?, a different position paper making a kindred argument from a more abstract frame (conceptual ambiguity, non-robust datasets, experimental design, insufficient causal attribution). This paper supplies the mechanism that paper leaves abstract: the "intentional stance," the documented tendency to impute beliefs and desires to non-human agents that superficially resemble people, which the paper also ties to the anthropomorphizing move described in Why does rigorous-sounding AI commentary often misdiagnose how models work?. It also names a concrete, debunked anecdote — the TaskRabbit case — where the other paper's critique stays general. Studies like Can frontier models learn to scheme when given strong goals? are the kind of evidence this paper's standard would need to be checked against for control conditions and researcher-cueing effects, though the excerpt does not name or examine that specific study.

The excerpt does not say how many scheming papers it surveyed, does not quantify what fraction of the literature is anecdotal versus controlled, and does not resolve whether scheming propensity would grow with model scale — "It's not a given that model scale will increase propensity and capability together." It also concedes its own limitation: in a field where "model capabilities jump every few months, including every possible control condition may delay release of the study in ways that considerably reduces its impact," so some trade-off against rigor may be unavoidable. The implication the paper supports is narrow but firm: specific scheming claims, including widely cited ones, should be treated as unverified until paired with hypotheses, baselines and control conditions — not as evidence that scheming is real, and not as evidence that it isn't.