How should multi-objective post-training balance competing behavioral goals?
This explores how to train a model toward several behavioral goals at once — accuracy, calibration, diversity, planning — when pushing on one tends to erode another, and what the corpus says actually keeps them in balance.
This reads the question as: when post-training pulls toward multiple goals that fight each other, how do you keep one from quietly eating the others? The corpus's first surprise is that 'balance' is rarely a matter of picking the right fixed weights. The most direct take on weighting argues you shouldn't hand-tune scalarization constants at all — How should multiple reward objectives be weighted during training? weights each objective by how much signal it actually carries per batch, automatically turning up high-information objectives and damping noisy ones. That reframes the whole problem: the competition between goals is partly an artifact of treating every reward term as equally trustworthy when they aren't.
The deeper lesson is that the worst conflicts are invisible if you only watch your headline metric. Optimizing a single crisp objective tends to silently destroy a behavior you never put a reward on. Binary correctness rewards look fine on accuracy while wrecking calibration, because nothing penalizes confident wrong answers — Does binary reward training hurt model calibration? shows adding a proper scoring rule (Brier) as a second term jointly optimizes both with no trade-off. The same hidden-collapse pattern shows up as lost diversity: RL converges policies onto one narrow strategy in reasoning, in search agents, and even onto a single output format inherited from pretraining (Does reinforcement learning squeeze exploration diversity in search agents?, Does RL training collapse format diversity in pretrained models?). A striking partial fix: Does negative reinforcement alone outperform full reinforcement learning? finds that punishing wrong answers while never over-rewarding right ones preserves diversity that positive-only training crushes. So one goal — exploration breadth — is best protected not by adding a reward but by changing which direction you push.
The corpus's biggest move is to stop thinking about balance as simultaneous and start thinking about it as sequenced. Competing objectives often shouldn't be optimized at the same time at all. Does sequencing imitation then exploration training improve reasoning? runs imitation first to build reasonable behavior, then verifiable-reward RL to sharpen it, beating either alone. Does training order reshape how models handle different task types? shows why order matters mechanically: structured tasks lower output entropy and creative tasks raise it, so training structured-first avoids entropy collapse poisoning open-ended skills. And Does RL training follow a predictable two-phase learning sequence? reveals models naturally pass through phases — execution correctness first, strategic planning second — meaning the 'right' objective to emphasize is itself a moving target during training.
There's also a counterintuitive thread on a goal everyone wants but that fights ordinary training: making a model better at *deciding* can make it worse at *learning*. Can utility-weighted training loss actually harm model performance? finds utility-weighted loss strengthens decisions while starving feature learning, and that training with symmetric loss then adjusting predictions afterward beats baking utility in directly. The unifying principle across all of this: scalar rewards are too thin to carry multiple goals. Can scalar rewards capture all the information in agent feedback? shows feedback splits into 'how well' (evaluative) and 'how to change' (directive) information a single number can't hold both of, and Can natural language feedback overcome numerical reward plateaus? shows language critiques break plateaus numbers can't — because they explain *why*. Should successful and failed episodes be processed differently? extends this asymmetry to memory itself, treating successes and failures as different kinds of evidence.
The thing the reader may not have expected to learn: the best answer to 'how do you balance competing goals' is often 'don't balance them at once.' Sequence them by phase, protect fragile behaviors (diversity, calibration) with the right *kind* of signal rather than a bigger weight, let variance decide how much each objective is trusted, and reach for richer-than-scalar feedback when two goals genuinely can't be compressed into one number.
Sources 12 notes
DVAO weights objectives by their within-group variance, automatically up-weighting high-signal objectives and suppressing noise without hyperparameter tuning. This keeps advantage magnitudes bounded and replaces fixed scalarization constants with data-driven weighting.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Asymmetric loss functions correctly incentivize choosing but degrade representation learning by reducing gradient signals for substantive feature acquisition. Training with symmetric loss then adjusting predictions post-hoc outperforms direct utility-weighted training on the same utility objective.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.