The Consensus Game: Language Model Generation via Equilibrium Search

When applied to question answering and other text generation tasks, language models (LMs) may be queried generatively (by sampling answers from their output distribution) or discriminatively (by using them to score or rank a set of candidate outputs). These procedures sometimes yield very different predictions. How do we reconcile mutually incompatible scoring procedures to obtain coherent LM predictions? We introduce a new, a training-free, game-theoretic procedure for language model decoding. Our approach casts language model decoding as a regularized imperfect-information sequential signaling game—which we term the CONSENSUS GAME—in which a GENERATOR seeks to communicate an abstract correctness parameter using natural language sentences to a DISCRIMINATOR. We develop computational procedures for finding approximate equilibria of this game, resulting in a decoding algorithm we call EQUILIBRIUM-RANKING. Applied to a large number of tasks (including reading comprehension, commonsense reasoning, mathematical problem-solving, and dialog), EQUILIBRIUM-RANKING consistently, and sometimes substantially, improves performance over existing LM decoding procedures—on multiple benchmarks, we observe that applying EQUILIBRIUMRANKING to LLaMA-7B outperforms the much larger LLaMA-65B and PaLM- 540B models. These results highlight the promise of game-theoretic tools for addressing fundamental challenges of truthfulness and consistency in LMs.

Further complicating matters, LMs offer multiple affordances for solving factual generation tasks. They may be used both generatively (e.g. by asking for the most probable answer to a question) or discriminatively (e.g. by presenting a (question, answer) pair and asking whether the answer is acceptable) and, these two procedures do not always produce consistent results: generative procedures may fail when probability mass is spread across multiple contradicting answers (Wang et al., 2022; Mitchell et al., 2022), while discriminative procedures may fail due to miscalibration (Han et al., 2022; Chen et al., 2022) or subtle dependence on question wording (Jiang et al., 2020). Given these noisy and often-conflicting signals, how should we distill out an LM’s best guess at the truth?

This paper presents an approach for reconciling generative and discriminative LM decoding procedures by formulating decoding as a signaling game (Lewis, 2008) that we call the CONSENSUS GAME. At a high level, this game features a GENERATOR agent that must communicate an abstract correct or incorrect value to a DISCRIMINATOR agent, but may only do so using a set of candidate natural language strings (Fig. 1). Intuitively, an effective strategy for this game (i.e. a joint policy) is one in which the GENERATOR and DISCRIMINATOR agree on the assignment of strings to correctness values. Given such a strategy, we may inspect it to identify candidates agreed by consensus to be correct.