Can we teach LLMs to form linguistic conventions in context?
Humans naturally shorten references as conversations progress, but LLMs don't adapt their language for efficiency even when they understand their partners do. Can training on coreference patterns teach this convention-forming behavior?
Humans naturally form ad-hoc linguistic conventions during interaction — reducing "the medicine for my back pain in a small blue medicine bottle" to "my back meds" within a few exchanges. Since Why don't LLMs shorten messages like humans do?, this convention formation is absent from LLMs even when they understand the conventions as listeners.
The post-training solution uses three components:
1. Heuristic data extraction. Coreference resolution on 2,000 TV scripts identifies reference chains where a concept is initially mentioned with a full noun phrase and later re-mentioned with a more concise expression. Each chain can provide multiple demonstrations, showing the model that convention formation persists across the entire reference chain, not just the first re-mention.
2. Two types of preference pairs. Type 1 (convention demonstration): preferred = observed concise re-mention, dispreferred = verbose first-mention repeated as re-mention. This suppresses verbatim repetition and encourages adaptation. Type 2 (first-mention preservation): preferred = original full first mention, dispreferred = premature conventionalization. This prevents the model from pre-shortening before common ground exists. 11,106 Type 1 + 10,135 Type 2 pairs total.
3. Mention planning tokens. A special [remention] token precedes re-mentions, explicitly marking the distinction between initial and later references. This allows the model to separate its processing of first mentions (full description required) from re-mentions (convention formation appropriate). Additional preference pairs train the model to USE this token correctly.
The result is a general in-context behavior: post-trained models spontaneously form conventions as interactions progress, without task-specific fine-tuning. Evaluated on both a cognitively-motivated interaction benchmark and a document-grounded reference completion task.
This provides the training-time fix for what Why don't conversational AI systems mirror their users' word choices? identifies as a behavioral gap. The mechanism is elegant: rather than engineering convention formation rules, the training data reveals the pattern through naturally occurring coreference chains, and DPO optimization internalizes it.
Related concepts in this collection
-
Why don't conversational AI systems mirror their users' word choices?
Explores whether current dialogue models exhibit lexical entrainment—the human tendency to align vocabulary with conversation partners—and what's needed to bridge this gap in AI communication.
this paper provides the training solution to the LE gap
-
Why don't LLMs shorten messages like humans do?
Humans naturally develop shorter, efficient language during conversations. Do multimodal LLMs exhibit this same spontaneous adaptation, or do they lack this communicative behavior?
the problem this paper solves
-
Does preference optimization damage conversational grounding in large language models?
Exploring whether RLHF and preference optimization actively reduce the communicative acts—clarifications, acknowledgments, confirmations—that build shared understanding in dialogue. This matters for high-stakes applications like medical and emotional support.
DPO used here constructively to ADD convention formation, contrasting with its destructive effect on grounding in other contexts
-
Why do speakers need to actively calibrate shared reference?
Explores whether using the same words guarantees speakers mean the same thing. Investigates how referential grounding differs across people and what collaborative work is needed to establish true understanding.
convention formation is a concrete mechanism for communicative grounding: shortening "the medicine for my back pain" to "my back meds" calibrates shared reference through interaction
-
Can agents learn continuously without forgetting old skills?
Can lifelong learning systems retain previously acquired skills while acquiring new ones? This explores whether externalizing learned behaviors as retrievable code programs rather than parameter updates solves catastrophic forgetting.
convention formation and skill libraries are dual aspects of the same mechanism: agents under performance pressure develop compact reusable abstractions; conventions compress linguistic reference, skill libraries compress behavioral sequences; both compound through interaction
Click a node to walk · click center to open · click Open full network for a force-directed map
Original note title
post-training for convention formation via DPO on coreference-identified preference pairs with mention planning tokens teaches LLMs to form ad-hoc conventions in context