How do LLMs balance remembering context versus keeping it separate?

Successful conversation requires keeping track of common ground, scoreboard updates, discourse referents, and topic shifts. Humans do this through structured memory — episodic, semantic, procedural — that compartmentalizes contexts into separate frames. LLMs do not have this structure. They process context as a single long string of tokens, with no native distinction between conversational threads, communicative roles, or topic boundaries.

This forces a dilemma. If the model retains too much, it suffers context collapse: a technical-support thread blurs into a billing thread, a philosophy conversation contaminates a vacation discussion, and the model produces responses that mix references from incompatible frames. If it retains too little — for example because the conversation overflowed the context window — it loses anaphoric reference, drifts off topic, and contradicts its own earlier commitments. Diachronic consistency breaks: the model that recommended one solution may unknowingly suggest a conflicting one once the prior turn has scrolled out of attention.

Mitigations exist — context compression, longer windows, retrieval-augmented memory — but each introduces its own failure mode. Compression is lossy and biased toward what the model judges salient. Larger windows raise cost without solving prioritization. RAG depends on retrieval quality. None of these reproduces the human capability to maintain separate mental contexts that can be entered and exited deliberately. This is not a tunable parameter problem. It is a structural mismatch between transformer attention and the layered, compartmentalized memory that pragmatic competence requires.

Source: Conversation Topics Dialog