Pretrained Persona Mixture Models and Tandem Models for Human Simulation

We argue here that the current dominant practice in LLM human simulation— prompting instruction-tuned “assistant” language models to role-play personas—is inaccurate and produces stereotyped predictions (lacking natural diversity). It has previously been shown that LLMs can be “bound” to personas using naturalistic, freetext dialog avoiding stereotyping. Here we show that binding can also be achieved using short, individual samples of dialog from specific people. Demographics can be added later without negative effects by simply querying the model. We use the term Persona Mixture Models (PMMs) for well-calibrated human models, currently realized as pretrained base models. We show that they produce more accurate predictions than instruction-tuned models and retain more of the lexical, semantic, and pragmatic diversity found in human dialog. We measure realism and diversity of LLMs simulating human interlocutors across a diverse set of corpora spanning open-domain text, human-AI chat, and task-oriented dialogue between human speakers. However, base pretrained models can produce out-of-domain dialog and may lose some of the human’s internal state over long contexts. We propose and explore tandem models which combine a pre-trained model with an instruction-tuned supervisor.

Introduction. Large language models (LLMs) are increasingly used as proxies for human users. For example, recent work uses LLMs as simulated users for training and evaluating LLMs: rather than recruiting human participants for every iteration, a simulated user “interacts” with a model in multi-turn dialogue, providing automatic feedback at scale. Other work in computational social science uses LLMs as synthetic survey respondents and study participants [1–4]. The practice of building classical task-oriented dialogue systems through user simulations [5–7] has continued to prove effective even as the underlying technology has moved to LLMs, for problems including cooperative planning [8], cooperative task completion [9, 10], end-to-end evaluation of dialogue agents [11–15], and in reinforcement learning pipelines where simulator fidelity determines the quality of the reward signal [16–18]. Most human simulation work using LLMs uses instruction-tuned (IT) “chat” models, prompted with persona descriptions [19–24].

Discussion / Conclusion. In this work, we have argued that user simulation is a very different task from dialog assistance. The optimizations applied for the assistant task systematically degrade performance for human simulation, and IT models are as well-adapted for human simulation as unaligned pretrained models are for the assistant task. We adopted the terms Persona Mixture Models (PMMs) to better capture the distinction between these models (currently realized as pretrained models) and ALMs (IT assistant models). LLM pretraining corpora are themselves large, poorly characterized, and subject to their own distributional biases [84]. The user models that emerge from pretraining reflect the demographics and discourse norms captured in web-scale data. IT models typically introduce additional biases from post-training data which are based on behavioral norms adopted by their designers. These may or may not represent the full user community.