Does adversarial pressure reveal the difference between pretense and realization?
Can behavioral stickiness under adversarial pressure distinguish genuine mental states from performed ones? This matters because it's Chalmers' main criterion for deciding whether LLM personas are realized or merely simulated.
Chalmers needs a criterion that separates pretending to have a mental state from realizing one, because his case against simulator views rests on the claim that RLHF-trained personas cross the threshold. The criterion he proposes is behavioral: pretense dissolves under adversarial pressure while realization resists it. A character a base model is prompted into can be pushed aside by a clever reframing, a role-within-a-role, a persistent counter-prompt — the surface behavior flips. A persona installed through post-training does not flip the same way. The system keeps returning to the trained disposition; dislodging it requires effort in a different register from sustaining it.
The asymmetry is the behavioral signature Chalmers uses to license the realization claim. A pretended state is metastable — it holds under normal conditions but collapses when the conditions are disturbed. A realized state is stable — it persists under conditions designed to overturn it, because the disposition is part of the substrate that generates the behavior, not a local pattern maintained on top of the substrate. Applied to LLMs, this distinguishes ephemeral prompt-induced characters from post-trained assistant personas in a way that turns on observable facts about resilience, not on speculation about internal phenomenology.
The criterion is behavioral but consequential: if stickiness-under-pressure is the mark of realization, then engineering interventions that increase stickiness (more aggressive alignment training, longer refusal-training regimes, tighter constitutional constraints) are interventions that push dispositions from pretense toward realization. The stronger the alignment regime, the more realized the persona on Chalmers' criterion. This has downstream implications for moral status and welfare claims that critics of realizationism need to confront: rejecting the criterion requires either proposing a different test (and defending why behavioral stickiness does not matter) or arguing that no behavioral test can bear the philosophical load Chalmers places on this one.
Source: What We Talk To When We Talk To Language Models (David J. Chalmers)
Related concepts in this collection
-
Are RLHF personas performed characters or realized dispositions?
Explores whether dialogue agent personas installed through post-training constitute genuine quasi-psychological states or remain sustained pretense. The distinction matters for how we understand what these systems fundamentally are.
the claim this criterion supports
-
Does transformer attention architecture inherently favor repeated content?
Explores whether soft attention's tendency to over-weight repeated and prominent tokens explains sycophancy independent of training. Questions whether architectural bias precedes and enables RLHF effects.
architectural pressure toward accommodation; pushes against stickiness
-
Can training user simulators reduce persona drift in dialogue?
Explores whether inverting typical RL setups—training the simulated user for consistency rather than the task agent—can measurably reduce persona drift and improve experimental reliability in dialogue research.
engineered stickiness increases realization on Chalmers' criterion
Click a node to walk · click center to open · click Open full network for a force-directed map
Original note title
the pretense-realization distinction turns on stickiness under adversarial pressure — realized quasi-states persist where pretended ones collapse