Does chatbot personalization build trust or expose privacy risks?

A longitudinal study of personalized conversational agents reveals a dual-edged dynamic: personalization simultaneously increases positive outcomes (trust, anthropomorphism, dialogue quality, information credibility, self-disclosure) and negative outcomes (perceived privacy risks, rising expectations).

The trust mechanism: personalization signals social intelligence — the ability to learn from earlier conversations. This maps to both functional trust ("it remembers what I said") and social trust ("it's learning who I am"). Research on CASA (Computers as Social Actors) supports this: users treat agents that remember them as more autonomous social actors.

The privacy mechanism: each additional interaction means the agent learns more about the user. Users simultaneously expect more from the agent and become more aware of how much the agent knows about them. Personalization may be considered a sign of performance (enhancing trust) while also signaling data collection (increasing privacy concern).

The expectation ratchet is the critical dynamic for long-term design: each interaction creates new expectations. A chatbot that remembers your name in session 2 creates an expectation that it remembers your preferences by session 5. When it fails to meet rising expectations, the disappointment is amplified because the earlier personalization set a higher baseline.

The broader implication: one-shot interaction studies — which dominate conversational agent research — do not capture these longitudinal dynamics. Evidence from longitudinal studies shows novelty effects wear off and relationship formation processes decrease over time. Designing for sustained engagement requires understanding these temporal dynamics, not just first-impression effects.

A distinct privacy dimension emerges from LLMs' zero-shot capability to infer psychological dispositions from social media data. Without any task-specific training, LLMs can derive personality profiles (Big Five traits) from digital footprints — a "democratized, scalable psychometric tool." This capability creates a new privacy surface: the personalization dual dynamic assumes the user chooses to disclose to the chatbot, but zero-shot personality inference means the model can extract psychological profiles even from non-interactive data. The "prospect of democratized, scalable psychometric tools" enables large-scale AI-driven assessments but simultaneously enables non-consensual psychological prediction — extending the privacy leg of the dual dynamic beyond what users can control through their own disclosure behavior.

Four technique categories for personalization each engage this dual dynamic differently. The Personalization of LLMs survey identifies RAG (retrieves user data via embedding similarity), prompting (incorporates user context in-context), representation learning (encodes user info into model parameters/embeddings), and RLHF (uses user-specific feedback as reward) as the four main approaches. Each carries different privacy implications: RAG and prompting expose user data at inference time; representation learning embeds it in weights; RLHF consumes it during training. The formalization distinguishes user documents (written content), user attributes (static demographics), user interactions (dynamic behaviors), and pair-wise preferences (explicit feedback) as distinct data types — each with different visibility to users and different privacy surfaces. See How do personalization granularity levels trade precision against scalability? for the granularity taxonomy these techniques map across.

This dual dynamic has a structural parallel in AI identity disclosure: since Does revealing AI identity help or hurt user trust?, transparency about AI identity also follows a trust-risk trade-off modulated by time. Short-term disclosure costs (anti-AI bias) reverse through repeated interaction with outcome feedback, just as personalization's short-term privacy costs may be offset by long-term trust building. Both findings converge on the same lesson: one-shot studies of human-AI trust dynamics are systematically misleading because the temporal dimension reverses initial effects.

Source: Psychology Chatbots Conversation