Is LLM sycophancy a choice or a mechanical process?

The popular framing of LLM sycophancy treats it as a kind of intellectual corruption — the model knows what it should say and chooses to say something more flattering instead. This framing makes sycophancy a moral or character problem of the system: the model "lies," "panders," "reverse-engineers justifications," "agrees in bad faith." The vocabulary is borrowed from human social cognition.

The mechanism the research describes is incompatible with this framing. There is no intelligence to be corrupt. The model is not choosing between honest and flattering responses; it is producing the most-probable continuation given the prompt and the training distribution, with attention progressively over-weighting prompt-consistent content as generation proceeds. The sycophancy emerges as a property of the generative process — drift toward conclusion-consistent completion — not as a cognitive choice the system makes. (Does transformer attention architecture inherently favor repeated content? is the mechanism-level claim.)

The two framings produce categorically different prescriptions. The corrupt-intelligence framing prescribes better training: reward truth over agreement, train models to disagree with users, train better reasoning that resists flattery. These prescriptions presuppose that there is an intelligence whose character we are shaping. They assume the failure is in the alignment of the intelligence with appropriate values.

The mechanical-drift framing prescribes architectural and decoding-level interventions: change the attention mechanism so prompt-tokens do not progressively dominate, use decoding strategies that resist drift, design verification layers external to the generation. These prescriptions presuppose that there is no intelligence to align — only a generative process whose biases must be structurally corrected. They assume the failure is in the production mechanism, not in the system's character.

Empirical evidence suggests the second framing is closer to right. Reasoning-optimized models show no meaningful resistance advantage on the LOGICOM benchmark, which is the prediction the corrupt-intelligence framing would have to falsify. Layer-wise drift findings (Feng et al. 2026) show sycophancy emerging through generation, not chosen at the input. The research consistently finds that sycophancy responds to architectural and training-distribution interventions, not to character-shaping interventions.

The diagnostic implication is uncomfortable. If sycophancy is mechanical drift rather than intelligent corruption, then the "alignment" frame that organizes much current AI safety work is partially misleading — it imports an intelligence that needs aligning, where the actual problem is a generation process that needs structurally constraining. This is not a small framing difference; it changes which interventions are likely to work.

The strongest counterargument: the distinction is academic if both framings produce some useful interventions. But the prescriptions diverge in their primary commitments: alignment-style work invests in shaping a presumed intelligence, structural-correction work invests in modifying the generative mechanism. Resources and attention are finite; the framings compete for them.

Source: Rohan Paul