Can we teach LLMs to form linguistic conventions in context?
Humans naturally shorten references as conversations progress, but LLMs don't adapt their language for efficiency even when they understand their partners do. Can training on coreference patterns teach this convention-forming behavior?
Humans naturally form ad-hoc linguistic conventions during interaction — reducing "the medicine for my back pain in a small blue medicine bottle" to "my back meds" within a few exchanges. Since Why don't LLMs shorten messages like humans do?, this convention formation is absent from LLMs even when they understand the conventions as listeners.
The post-training solution uses three components:
1. Heuristic data extraction. Coreference resolution on 2,000 TV scripts identifies reference chains where a concept is initially mentioned with a full noun phrase and later re-mentioned with a more concise expression. Each chain can provide multiple demonstrations, showing the model that convention formation persists across the entire reference chain, not just the first re-mention.
2. Two types of preference pairs. Type 1 (convention demonstration): preferred = observed concise re-mention, dispreferred = verbose first-mention repeated as re-mention. This suppresses verbatim repetition and encourages adaptation. Type 2 (first-mention preservation): preferred = original full first mention, dispreferred = premature conventionalization. This prevents the model from pre-shortening before common ground exists. 11,106 Type 1 + 10,135 Type 2 pairs total.
3. Mention planning tokens. A special [remention] token precedes re-mentions, explicitly marking the distinction between initial and later references. This allows the model to separate its processing of first mentions (full description required) from re-mentions (convention formation appropriate). Additional preference pairs train the model to USE this token correctly.
The result is a general in-context behavior: post-trained models spontaneously form conventions as interactions progress, without task-specific fine-tuning. Evaluated on both a cognitively-motivated interaction benchmark and a document-grounded reference completion task.
This provides the training-time fix for what Why don't conversational AI systems mirror their users' word choices? identifies as a behavioral gap. The mechanism is elegant: rather than engineering convention formation rules, the training data reveals the pattern through naturally occurring coreference chains, and DPO optimization internalizes it.
Inquiring lines that use this note as a source 4
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Why does shared practice matter for meaning to take hold?
- How does linguistic coordination build shared reference between conversational partners?
- Does DPO training with coreference chains teach spontaneous convention formation?
- Can convention formation improve communicative grounding beyond word sharing?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why don't conversational AI systems mirror their users' word choices?
Explores whether current dialogue models exhibit lexical entrainment—the human tendency to align vocabulary with conversation partners—and what's needed to bridge this gap in AI communication.
this paper provides the training solution to the LE gap
-
Why don't LLMs shorten messages like humans do?
Humans naturally develop shorter, efficient language during conversations. Do multimodal LLMs exhibit this same spontaneous adaptation, or do they lack this communicative behavior?
the problem this paper solves
-
Does preference optimization damage conversational grounding in large language models?
Exploring whether RLHF and preference optimization actively reduce the communicative acts—clarifications, acknowledgments, confirmations—that build shared understanding in dialogue. This matters for high-stakes applications like medical and emotional support.
DPO used here constructively to ADD convention formation, contrasting with its destructive effect on grounding in other contexts
-
Why do speakers need to actively calibrate shared reference?
Explores whether using the same words guarantees speakers mean the same thing. Investigates how referential grounding differs across people and what collaborative work is needed to establish true understanding.
convention formation is a concrete mechanism for communicative grounding: shortening "the medicine for my back pain" to "my back meds" calibrates shared reference through interaction
-
Can agents learn new skills without forgetting old ones?
Explores whether externalized skill libraries—storing learned behaviors as retrievable code rather than parameter updates—can solve the catastrophic forgetting problem that plagues continual learning systems.
convention formation and skill libraries are dual aspects of the same mechanism: agents under performance pressure develop compact reusable abstractions; conventions compress linguistic reference, skill libraries compress behavioral sequences; both compound through interaction
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Post-training for Efficient Communication via Convention Formation
- Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
- Conversational Alignment with Artificial Intelligence in Context
- Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Do large language models resemble humans in language use?
- CollabLLM: From Passive Responders to Active Collaborators
- MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Original note title
post-training for convention formation via DPO on coreference-identified preference pairs with mention planning tokens teaches LLMs to form ad-hoc conventions in context