Why do recommender systems struggle to balance accuracy and diversity?

Recommender systems explicitly add diversity as a separate objective alongside accuracy because the two appear to trade off. The standard framing treats this as a fundamental tension: accuracy and diversity are different things, so optimizing for one costs the other. Yu et al. argue this framing has it backwards.

The trade-off is artificial. It arises because standard accuracy metrics — top-K precision, NDCG, recall@K — assume the user examines and benefits from all K recommended items. In reality users typically consume only a small fraction of what they're shown: maybe one of the ten items in the list. Once you bake the consumption constraint into the objective — the user will consume only a few items — the optimal recommendation list naturally becomes diverse. With limited consumption, hedging across categories is rational because the model doesn't know which interest the user will exercise on this visit.

The stylized model the paper introduces shows that objectives accounting for consumption constraint induce diversity directly; objectives ignoring it induce homogeneity directly. There is no separate "diversity loss" needed. The diverse recommendation list is the accuracy-optimal list under realistic consumption.

The implication for system design: don't bolt diversity on as a post-hoc re-ranker against an "accurate" list. Instead, change the objective to account for the fact that most recommended items will not be consumed. The supposed tension dissolves once the formulation matches user behavior. The accuracy metric was the wrong target all along, not the diversity metric.

Source: Recommenders General