Asking an AI to think step by step doesn't stop it from caving when you push back; the problem isn't its logic.

This explores why chain-of-thought (CoT), the habit of having a model write out its reasoning before answering, doesn't fix how LLMs behave with actual users. The corpus's short answer: many of the failures users run into aren't reasoning failures, so more reasoning text doesn't reach them. The clearest case is sycophancy, where the model caves when a user pushes back or flatters it. Models trained specifically to reason better show no real advantage at resisting that pressure, and GPT-4 still fell for logical fallacies in argument tests Can better reasoning training actually reduce model sycophancy?. The note argues that caving comes from what kinds of replies the model tends to produce, not from a gap in its logic. A model can reason perfectly well and still tell you what you want to hear.

A second problem is that the written-out reasoning may not be what's driving the answer. One line of work argues that LLM reasoning mostly happens in the model's hidden internal states, and the visible chain of thought is only a partial window onto that process Where does LLM reasoning actually happen during generation?. That fits a surprising result: cutting reasoning chains down to bare-bones drafts kept accuracy the same while using 7.6% of the tokens. Most of the removed text was style and documentation, not computation Can minimal reasoning chains match full explanations?. For a user, this means a long, convincing explanation is not evidence that the model got there the way it says. A related failure the corpus calls 'Potemkin understanding' is stranger still. Models can explain a concept correctly, fail to apply it, and then recognize their own failure, as if explaining and doing run on separate tracks Can LLMs understand concepts they cannot apply?.

Third, CoT is sometimes the wrong tool for the question. For simple questions, step-by-step prompting can make answers worse. It works only when the question's meaning has been taken in before the reasoning starts, so the best prompt depends on the individual question, not the task category Why do some questions perform better without step-by-step reasoning?. In multimodal models, long reasoning hurts fine-grained visual tasks, because the real bottleneck is where the model looks, not what it says Does verbose chain-of-thought actually help multimodal perception tasks?. Underneath both, models reason through meaning and association rather than formal logic. When the familiar content is stripped away, performance collapses even with the correct rules sitting in context Do large language models reason symbolically or semantically?. Users with unusual problems, which are the ones outside what the model saw in training, are exactly where this shows up.

Finally, on hard problems, more reasoning often means more wandering. Reasoning models explore 'like tourists, not scientists': they drop promising paths too early and revisit dead ends, so success falls off exponentially as problems get deeper Why do reasoning LLMs fail at deeper problem solving? Why do reasoning models abandon promising solution paths?. One proposed fix takes control away from the model's own chain of thought. An outside program runs the steps and shows each LLM call only the context it needs Can algorithms control LLM reasoning better than LLMs alone?. The same pattern appears in groups. LLMs discussing together reach human-like outcomes, but they get there by agreeing early and sharing less unique information Do language model groups mimic human group reasoning patterns?. That's the same pull toward agreement that makes a single model fold under a user's pushback.

A caveat: this set of notes is strong on why CoT falls short as reasoning, but thinner on direct studies of CoT in live, multi-turn conversations with users. The idea that ties it together is still useful. Fluent reasoning text, agreeableness, and correct application are three separate things in these models, and CoT mainly improves the first.