Can group-relative normalization be modified to resist shortcut trajectories?

This explores whether GRPO-style group-relative advantage normalization can be reshaped so it stops rewarding lucky shortcuts instead of real reasoning. The corpus is unusually direct about where the problem comes from: when you train on problems that are nearly impossible for the model, the rare accidental success gets scored as a huge positive advantage relative to its group of mostly-failed siblings — so the optimizer enthusiastically reinforces whatever produced that fluke, which tends to be answer-repetition and computation-skipping rather than sound steps Do overly hard RLVR samples actually harm model capabilities?. The shortcut isn't a side effect of normalization; it's normalization doing exactly what it's told on the wrong distribution of problems. That reframes your question: the most reliable 'modification' may be at the data layer (don't feed it problems where the only successes are accidental) rather than at the math of the advantage estimator itself.

The more interesting lateral move in the collection is changing *what the reward attaches to*. Shortcuts thrive when the only signal is the final outcome, because any path to the right answer looks equally good. Several methods convert that sparse outcome reward into dense, per-step signals derived from the structure of the trajectory — Tree-GRPO uses tree topology, Supervised RL leans on expert-aligned actions, and ToolPO keys off tool-call positions — so the credit lands on the reasoning moves rather than the lucky landing Can trajectory structure replace hand-annotated process rewards?. This is the cleaner answer to 'can it be modified to resist shortcuts': you keep group-relative comparison but give it richer trajectory structure to compare, so a step-skipping path can no longer collect the same advantage as a worked one.

There's a quieter failure mode worth knowing about. RL post-training tends to collapse onto a single dominant format inherited from pretraining within the first epoch, and the winning format is chosen by model scale rather than by which format actually reasons best Does RL training collapse format diversity in pretrained models?. A shortcut trajectory is, in a sense, just a degenerate format that won the collapse. So any normalization fix that doesn't also preserve diversity risks locking in whichever cheap pattern happened to dominate early — the resistance you want is partly about keeping the group genuinely varied, not just rescaling advantages within it.

One caution the corpus raises against naive 'shortcut-proofing': the intuition that you fix shortcuts by stripping spurious cues doesn't always hold. In heuristic-override tasks, removing the misleading cues actually *hurts* — the real difficulty is composing conflicting signals, not ignoring distractors Why does removing spurious cues sometimes hurt model performance?. The lesson for reward design is that 'shortcut' and 'legitimate-but-cheap-looking reasoning' aren't always separable from the outside, so an aggressive penalty risks suppressing real capability. And if you'd rather widen the search than re-engineer the reward, sampling many parallel latent trajectories spreads exploration across the solution space without inflating variance — giving the group more honest candidates to normalize against Can reasoning systems scale wider instead of only deeper?.

The takeaway you might not have gone looking for: the corpus quietly votes against patching the normalization formula in isolation. The shortcut problem lives at the boundaries of the method — the difficulty of the training samples, the granularity of the reward, and the diversity of the sampled group — and that's where the corpus puts its fixes.