Why do similar user profiles produce worse personalization errors?

PRIME's controlled profile-replacement experiment reveals a counterintuitive U-shaped error curve in personalization fidelity:

Performance is highest with the target user's own profile (Self). When profiles are swapped, the drop is steepest with the most similar replacement user, then partially recovers as replacements become more dissimilar. The most similar user's profile produces worse outcomes than a mid-range or even dissimilar user's profile.

The mechanism is confidence-driven misdirection. PRIME learns fine-grained, user-specific preferences — effectively a dedicated bias toward which responses that specific user finds compelling. Two users with superficially similar posting histories may differ sharply in the specific responses they find persuasive. When a similar-but-not-identical profile is substituted, the model confidently applies the wrong preferences. A dissimilar profile, being obviously mismatched, triggers weaker and less harmful predictions because the model's learned biases don't activate as strongly.

This is an "uncanny valley" effect for persona similarity: nearly-right is more dangerous than clearly-wrong because the system cannot distinguish genuine user-specific preferences from similar-user preferences. The closer the profile approximation, the more confident — and more specifically wrong — the model becomes.

Data scarcity amplifies the effect. On the CMV (Change My View) forum, limited active users mean that two users with similar posting histories may have genuinely different persuasion profiles. The fine-grained preference learning that makes PRIME effective on correct profiles makes it maximally vulnerable to similar-but-wrong profiles.

This has direct design implications for persona-level personalization. Since How do personalization granularity levels trade precision against scalability?, persona-level approaches that group "similar" users together may systematically trigger this confidence-misdirection failure mode. The efficiency gain from persona grouping comes at the cost of confidently applying nearly-right-but-wrong preferences — potentially worse than no personalization at all.

The finding also connects to simulation fidelity. Since How do we generate realistic personas at population scale?, similar profiles may amplify rather than reduce systematic bias, because the simulation confidently generates behavior for the "wrong similar" user.

Source: Personalization