Why do language models ignore temporal order in ranking?

When LLMs are formatted as conditional rankers given a sequence of historical interactions, they can extract user preferences but treat the sequence as a set, ignoring temporal order. Order matters: recent interactions reflect current taste; older ones reflect past taste; the trajectory between them is informative. The LLM disregards this without explicit cuing.

Two interventions recover order sensitivity. Recency-focused prompting explicitly draws attention to the most recent items, signaling that recency carries weight. In-context learning provides examples of order-sensitive ranking, demonstrating the kind of inference the model should perform. Both work, indicating the issue is activation rather than capability — the LLM has the latent ability but doesn't deploy it without prompting.

Two systematic biases also appear: position bias (preferring candidates appearing early in the candidate list regardless of relevance) and popularity bias (preferring popular items). Both can be alleviated by prompting strategies — shuffling candidate orders across queries and aggregating, for instance, or explicit bootstrapping.

The empirical bottom line: LLMs outperform existing zero-shot recommendation methods, especially when ranking candidates retrieved by multiple candidate-generation strategies. The work needed to unlock that performance is not training but prompting. Many LLM capabilities require explicit cuing — they are present but not active by default. Treating LLMs as black-boxes whose performance reflects raw capability misses the activation gap; thoughtful prompting reveals capabilities undeployed by naive use.

Source: Recommenders LLMs