Do LLM explanations faithfully describe their recommendation process?

When LLMs are asked to make group recommendations from individual member preferences, the outputs converge on Additive Utilitarian aggregation — picking items with the highest sum of all members' ratings. This is the consensus-based strategy from social choice theory. The behavior is consistent across uniform and divergent group structures.

The disconnect is in the explanations. Asked to explain its recommendation procedure to a layperson, the LLM doesn't say "I summed the ratings" — it cites averaging (which is similar to but not identical to ADD), user or item similarity, diversity, undefined popularity metrics, and ad-hoc thresholds. Different LLMs invent different procedures: Llama tends to cite user similarity, while Mistral and Phi cite diversity in the recommendation list. These claimed procedures don't match the behavioral output.

This makes LLM explainers unreliable narrators. They generate recommendations one way and explain them another way, and the explanation is plausible enough that a user accepts it. As item set size grows, the mention of similarity and diversity in explanations increases (suggesting the LLM is performing post-hoc justification harder when more items make the choice less defensible) while the use of "undefined popularity" decreases. The implication for group recommender systems built on LLMs: the explanation layer cannot be trusted to faithfully describe what the model did, even though that's its stated purpose.

Source: Recommenders Architectures