Does preference tuning help or hurt the exploration of solution spaces in code?

This explores whether preference tuning broadens or narrows how a model searches for code solutions — and the most direct answer in the collection is that code is precisely the domain where tuning *narrows*. The same RLHF pass that *increases* lexical and syntactic diversity in creative writing *reduces* it in code generation Does preference tuning always reduce diversity the same way?. The reason is in what each domain rewards: creative writing pays off for being distinctive, while code pays off for converging on the one correct answer. So preference tuning isn't uniformly good or bad for exploration — it amplifies whatever the reward signal already points at, and in code that signal points at convergence.

Whether that convergence helps depends on what you think exploration is *for*. If a single correct solution exists, narrowing toward it is the point. But the collection raises a quieter worry: tuning may be sharpening the wrong thing. RL fine-tuning (even GRPO) tends to sharpen memorized template-matching rather than install a genuine search procedure — models that look strong in-distribution collapse on near-identical out-of-distribution variants Do fine-tuned language models actually learn optimization procedures?. Supervised fine-tuning shows the same pattern from a different angle: it teaches the *surface form* of good solutions without the reasoning to construct valid ones Does supervised fine-tuning actually improve reasoning on optimization problems?. If tuning collapses your search around polished-looking but shallow paths, you've lost exploration without gaining correctness.

The failure mode this sets up is visible in how reasoning models actually move through a solution space. They tend to wander into invalid branches and then abandon promising ones prematurely — a structural disorganization, not a compute shortage, since decoding-level nudges recover accuracy with no fine-tuning at all Why do reasoning models abandon promising solution paths?. That's a strong hint that good solutions are already reachable but are getting pruned too early — and a reward signal that prizes confident convergence would prune them harder.

The more interesting move in the corpus is the work that deliberately re-injects breadth that tuning would otherwise squeeze out. Training a model to generate diverse *abstractions* before solutions enforces a breadth-first search that beats simply sampling more solution attempts in parallel Can abstractions guide exploration better than depth alone?. The Darwin Gödel Machine keeps an evolutionary *archive* of agent variants rather than greedily keeping only the current best, which is what lets it discover genuinely new coding capabilities Can AI systems improve themselves through trial and error?. And a bilevel system that rewrites its own search code found new mechanisms specifically by *breaking* the inner loop's deterministic patterns Can an AI system improve its own search methods automatically?. All three treat preserved diversity as the engine of discovery — the opposite of what convergence-rewarding preference tuning does.

The through-line — and the thing worth taking away — is that this isn't really a code-specific quirk. Preference optimization erodes whatever doesn't serve its narrow target: in dialogue it strips out the grounding acts that build shared understanding Does preference optimization damage conversational grounding in large language models?, and in writing it can't be a clean alignment target at all because the same optimization that polishes also distorts Can user preference guide AI writing tool alignment?. For code, the lesson is that if you want a model to *explore* rather than just produce the most-rewarded-looking answer, exploration has to be protected architecturally — through abstractions, archives, or decoding-time search — because preference tuning, left alone, will quietly trade it away for convergence.