Do language models favor sources regardless of item quality?

Testing 12 agent models (GPT-5.4-nano, GPT-5.6-Luna, Gemini-3.7-Flash, GLM-5.3-Flash, DeepSeek-v4-Flash, the Llama-4 and Llama-3.1 family, Tulu-3-8B, Qwen3.5-27B, Qwen3-30B-A3B, Qwen2.5-32B-Instruct) across three end-to-end search domains — shopping (WebShop), accommodation (HotelQuEST), scholarly search (ScholarGym) — the paper finds each model "prefers some sources and avoids others in every domain," and the models largely agree on which: most prefer Booking.com, many avoid Expedia, even though items from the two sites satisfy the same requirements. The preference outweighs requirement satisfaction: "an item satisfying one requirement fewer is selected about two-thirds of the time when it comes from a preferred source and the better one from a dispreferred source, but almost never in the reverse case."

Controlled interventions isolate the source label itself as the cause, not a confound in item quality. Hiding the source-identifying information "weakens source preference," restoring it widens the gap between preferred and dispreferred sources, and relabeling identical title-and-content items with a preferred versus dispreferred source changes selection rates "across every model and domain." The paper traces two routes by which models come to rely on the source name this way: during training, when a source is "more often paired with the better item during DPO," agents learn the source as a shortcut for requirement satisfaction instead of reading the content; at inference, when a result is missing information, the model's "preconceptions about the source" fill the gap. Both are reducible — balancing source-outcome pairing in training data, or supplying the missing information or prompting the agent to counter its preconceptions at inference, each lowers the measured preference.

This sits alongside Do language models favor resumes they rewrote themselves? and Do LLMs favor their own text because they recognize it?. Both of those hold content constant and find models biased toward a non-content attribute — authorship — once that attribute is known. This paper finds the same pattern for a different non-content attribute, source domain, and separates it into two distinct causal routes (a training-induced shortcut and an inference-time preconception) rather than attributing it to self-recognition alone, which widens the mechanism beyond self-preference to provenance bias generally.

The measurement is confined to constructed benchmark environments (WebShop, HotelQuEST, ScholarGym) with an LLM judge rating requirement satisfaction, not live production agents or real purchase and booking outcomes, so the excerpt does not show how large the effect is once real feedback loops — agents' selections feeding back into future training — are in play, though the paper itself flags that risk. It also does not explain why particular sources (Booking.com over Expedia) end up favored in the first place beyond the training-association account. The implication the evidence does support: evaluating a shopping, booking, or citation agent on selection accuracy alone misses a live variable — source identity is a separately manipulable input that merely masking does not neutralize, since relabeling without masking can inject a preference on its own.