Do online reviews actually measure product quality or just buyer preferences?

If consumers were homogeneous in preferences, their reviews would directly reveal product quality — average a few of them and you have an unbiased estimate. Heterogeneity breaks this both ex ante and ex post. Ex ante, only consumers who expected to be satisfied chose to purchase; consumers who would have hated the product never bought it and so never wrote anything. Ex post, only the reviews of consumers who purchased are observable. Both filters select on idiosyncratic preferences correlated with satisfaction.

This produces several non-obvious effects studied across the review-aggregation literature. Hu, Zhang, and Pavlou's self-selection paper shows that early buyers' idiosyncratic preferences propagate into long-term purchase behavior — early reviewers shape what later buyers think the product is. Besbes and Scarsini formalize the question of whether consumers can learn product quality from reviews despite the bias, finding that altruistic reviewers (those writing about intrinsic product quality rather than personal experience) enable social learning while subjective reviewers do not. The combination of these two findings is uncomfortable: self-selection ensures the reviewer pool is biased, and many reviewers write about themselves rather than the product, which means social learning from ratings is conditional on reviewer motivation.

Acemoglu et al.'s "Fast and Slow Learning From Reviews" shows that more information does not always lead to faster learning — strictly finer rating systems do, but adding summary statistics can slow learning by amplifying selection effects. The general point: the rating distribution you see is not the satisfaction distribution among all potential customers. It is the satisfaction distribution among self-selected purchasers who chose to write, with the writing motivation itself a confound. Treating average rating as a quality estimate ignores both filters.

Source: Recommenders General