Do LLMs in conversational recommendation systems use collaborative or content knowledge?

There are two kinds of knowledge an LLM might use to recommend in conversation. Collaborative knowledge maps "users who liked A also liked B" — the standard collaborative-filtering signal embedded in interaction patterns. Content/context knowledge matches recommendations against descriptive context — genres, director names, mood, situational fit — using world knowledge the LLM acquired during pretraining.

He, Xu, Tang et al. probe which knowledge LLMs actually use by perturbing the conversation context in three ways. ItemOnly keeps only the item mentions and removes natural language. ItemRemoved keeps the language and removes the item mentions. ItemRandom replaces mentioned items with random items to control for sentence structure.

The result is asymmetric. Replacing original context with ItemOnly drops Recall@5 by more than 60% on average across models — losing the natural language is catastrophic. But replacing with ItemRemoved or ItemRandom drops GPT-based models less than 10% — losing the items is mild. The ItemRemoved condition still preserves enough content/context information for recommendations close to original quality.

This means LLMs in CRS settings primarily exercise content/context knowledge. They are more like situated content-based recommenders than collaborative-filtering systems with linguistic interfaces. This diverges from how traditional recommenders work, where user-interacted items are the foundation of every prediction. It also explains why LLM-CRS underperforms ItemCF baselines by 30% when only ItemOnly context is provided — without the content channel they have nothing.

The strategic implication: deploying LLMs as CRS works best in domains where content/context is rich (movies have well-known genre/cast/plot vocabulary in pretraining) and worst in domains where the only signal is co-purchase patterns LLMs never saw.

Source: Recommenders Conversational