What makes an AI a true thought partner, not just a tool?

The distinction between a tool for thought and a partner in thought is the relationship to the user. Collins et al. propose three desiderata drawn from behavioral science, not engineering intuition:

1. You understand me — the partner understands my goals, plans, (possibly false) beliefs, and resource limitations, adapting strategies when working with an expert versus a layperson versus a child. This requires a model of the human that updates with observation.

2. I understand you — the partner acts legibly, communicating in ways I intuitively understand. This is not about explanation-on-demand but about structural transparency in behavior.

3. We understand the world — the partner is tethered to reality through a shared representation of the domain or task. "We" emphasizes synergy — moving beyond the sum of parts.

The alternative scaling path proposed: rather than scaling foundation models on more data and human feedback traces (which produces systems that mimic human behavior but don't simulate human cognition), build systems with explicit structured models of task, world, and human. Nine cognitive science motifs provide the architectural ingredients:

• Bayesian Theory of Mind (BToM) — represent other agents as intentional actors; probabilistically infer mental states from observed actions
• Rational Speech Acts (RSA) — reason about language as intentional communicative action to infer speakers' underlying goals
• Resource-Rationality — model humans as making rational choices about how to allocate finite computational resources (time, memory)
• Structured Knowledge Representations — abstract, hierarchical, causal representations rather than flat distributional patterns
• Goal-Directed Planning — humans are intentional actors who plan under uncertainty
• Learning to Learn — meta-learning jointly with learning concrete concepts

The provocative claim: current LLMs produce fluent text but do not "robustly simulate human cognition" in ways a true thought partner requires. Mimicking human demonstrations is not the same as building models of why humans act as they do. The gap is between behavioral fidelity (producing human-like outputs) and cognitive fidelity (reasoning about the human's cognitive state).

Since Does theory of mind predict who thrives in AI collaboration?, the thought partner framework explains why ToM predicts collaboration: the three desiderata are fundamentally ToM-dependent. A user who can model the AI (desideratum 2) and signal their own state to the AI (enabling desideratum 1) fulfills both sides of the reciprocal understanding requirement.

Since What breaks when humans and AI models misunderstand each other?, the thought partner desiderata operationalize what bidirectional MToM would look like in practice: desideratum 1 is AI→human modeling, desideratum 2 is human→AI legibility, and desideratum 3 is the shared ground that makes both possible.

Source: Human Centered Design Paper: Building Machines that Learn and Think with People