SYNTHESIS NOTE
Can models learn to evaluate their own work during training?
Explores whether language models can internalize reward function computation as part of training, transforming external feedback into internal self-assessment capability without slowing inference.
Current training paradigms terminate learning at the end-of-sequence token, wasting the entire sequence space after model output completion. Post-Completion Learning (PCL) systematically exploits this neglected space. A temporary termination marker (<-- post-completion -->) creates a "post-thinking" space where models continue generating self-assessments and reward predictions during training, while inference stops at the marker — zero additional cost at deployment.
The core innovation is white-box reinforcement learning: the model explicitly learns to understand and compute reward functions, internalizing the reward model as its own evaluation capability. This transforms the model from "passive reward acceptance" (external reward signal tells it what's good) to "active self-evaluation" (it learns to compute quality assessments itself).
Implementation uses dual-track SFT: one track optimizes reasoning, the other optimizes evaluation capability. These are mixed with RL training for multi-objective hybrid optimization. The model learns both to solve problems and to assess its own solutions — but critically, only the problem-solving capability is active during inference. The self-evaluation is internalized during training, shaping the model's generation without requiring explicit self-assessment at inference time.
This addresses three limitations simultaneously:
- SFT's passive learning — models learn to mimic demonstrations without developing self-assessment ability
- RL's external dependency — reward models are opaque external components; PCL internalizes the evaluation
- Self-correction's inference cost — methods like Self-Refine require additional generation passes; PCL's self-evaluation is absorbed into training
The parallel with human cognition is direct: "Humans, after completing a task, often engage in self-reflection and quality assessment — this post-thinking process is crucial for improving future performance." PCL operationalizes this for LLMs.
This connects to What limits how much models can improve themselves? — PCL attempts to close the gap by training the verifier and generator as the same model, with the verification capability internalized rather than external. It also complements Does reflection in reasoning models actually correct errors? — PCL's self-evaluation is trained against ground-truth reward functions, not against the model's own prior outputs, potentially avoiding the confirmatory pattern.
Inquiring lines that read this note 156
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do models learn from self-generated outputs without cascading failures?- What happens when models train on AI-generated content recursively?
- Why does online RL succeed where supervised training fails for self-correction?
- Why do error avalanches accelerate in self-training loops without verification?
- Can synthetic self-play data teach models when to disagree?
- Why does self-generated training data outperform externally curated domain examples?
- Can the serving loop itself become the primary training data source?
- Can models learn to generate their own training examples effectively?
- Why does self-correction during generation produce reliable labels without exemplars?
- Can unsupervised confidence-based training scale to domains beyond human evaluation reach?
- How do instruction backtranslation and MAGPIE demonstrate self-generation principles?
- Can self-training drift be prevented by applying student compatibility filtering?
- Does self-play feedback improve skills created from the agent's own experience?
- Why does self-judgment of success or failure work without ground truth labels?
- Does the generation-verification gap define where self-rewarding actually works?
- Can self-ratings of output quality predict forecast performance?
- What happens when models train on feedback from their own generations?
- How does error distribution during training affect a model's ability to self-correct?
- What cognitive capabilities do agents need to internalize social feedback?
- Can an agent's internal probabilities serve as value signals across domains?
- Why do Generation-Then-Comprehension and AI Delegation produce opposite learning outcomes?
- Why does AI-improved task performance fail to transfer to independent work?
- Does AI-assisted performance transfer to independent task completion?
- Does learning community preferences as training rewards operationalize prediction without participation?
- How do self-generated preference pairs from a strong teacher compare to human feedback?
- Can preference learning fix the rigid output format problem better than supervised training?
- Can LLMs evaluate their own observations without external feedback?
- How do implicit world models and self-reflection operationalize consequence-based learning?
- How does self-referential processing transfer to other reasoning tasks?
- How do language models infer their own mental states like humans do?
- Do models spontaneously develop self-reflection from minimal training signals?
- Can models detect when their own trajectory is on-policy versus off-policy?
- Does recognizing your outputs as actions enable awareness of being evaluated?
- Which internal states can a language model access and report about itself?
- How does the silent token approach compare to modeling intrinsic motivation for speaking?
- What other internal model decisions beyond attention could be optimized directly?
- Do language models share the same cooperative truth-seeking rules as humans?
- Can LLMs learn to signal evaluative commitment through metadiscursive language?
- Can systems recognize and abstain on judgments rather than hallucinating preferences?
- Can AI evaluation match human judgment quality in structured domain tasks?
- Can polished language output substitute for the judgment it should express?
- What training signals would models need to learn reciprocal common-ground construction?
- Can models learn better from critiquing errors than imitating correct responses?
- How should training incorporate external critique versus encouraging self-correction?
- Can trained models encode programs more complex than their data-generating process?
- What emergent behaviors do models develop when trained on underspecified pedagogical tasks?
- Why does self-critiquing actually reduce plan quality in language models?
- Does self-revision actually improve reasoning in large language models?
- When does self-reflection actually help reasoning models improve?
- Does reflection training actually teach models to self-correct their mistakes?
- Does self-reflection help models notice their own constraint violations?
- Why does external critique improve revision accuracy more than self-assessment?
- Can language models accurately evaluate the quality of their own reasoning?
- Why does external critique improve revision while internal self-assessment fails?
- Does internal self-revision actually degrade reasoning accuracy in models?
- Can external retrieval signals outperform internal self-assessment during revision?
- How does metacognitive self-correction enable models to revise failed strategies?
- Does external critique guide revision better than internal self-assessment during model training?
- Why does self-critique fail without external verification signals?
- Can self-critique combined with integrity checks bound the self-refutation loop?
- What role does natural language play in breaking reinforcement learning performance plateaus?
- Can models learn both what and how to study through reinforcement learning?
- How does reinforcement learning on outcomes reinforce template-matching rather than computation?
- Does RL training redirect self-doubt into productive gap analysis?
- Why do reward models learn surface-level shortcuts instead of genuine quality assessment?
- Can reward model training be automated without changing feedback mechanisms?
- Could reward signals incentivize active intent discovery over passive response generation?
- How do semantic reward shaping approaches compare to full critique models?
- Why do generative reward models produce more interpretable evaluations than scalar scores?
- Can model confidence signals replace explicit external reward functions?
- Why do reward models fail when they ignore the prompt context?
- How can reward structures teach models when to speak and when to stay silent?
- How do reward models benefit from extended thinking during evaluation scoring?
- Can binary judge feedback replace external reward signals for skill learning?
- Why does self-segmentation into chunks-of-thought matter for reward models?
- How does in-context feedback integration differ from learned reward signals?
- What makes step-wise rewards denser than final-answer correctness signals?
- Does pairwise self-judgment avoid reward model scaling problems?
- How do internal model mechanisms escape token-level reinforcement signals?
- How do pairwise self-judgment and internal belief-shift replace verification differently?
- How do reward models and self-improvement mechanisms interact in training?
- What makes current learned reward models fail across different domains?
- How do self-play and human-anchored rewards separate competence from convention?
- Does training on critiques of noisy responses produce deeper understanding than imitating correct ones?
- How does training distribution shape what language models understand best?
- Why does natural language feedback break performance plateaus that numerical rewards alone cannot?
- How do graduated phase rewards emerge complex dialogue behavior from simple objectives?
- How do evaluative versus directive signals differ in next-state training?
- Can reward-guided decoding replace weight fine-tuning for personalized alignment?
- How does credit assignment work across many sequential decision steps in language models?
- Can structured natural language feedback outperform scalar rewards in RL?
- Can emotion-grounded rewards replace coarse bonus signals in hierarchical dialogue RL?
- Can environmental rewards directly refine natural language descriptions of actions?
- Can evaluation trajectories and interaction histories replace single-answer scoring?
- Can distillation and reward optimization happen in a single training loop?
- How does benchmark performance measure translate to general self-modification ability?
- Why do internal representations differ when external performance matches?
- Can AI learn to perform attention-seeking surface forms with genuine internal appeal?
- Can subjective tasks be delegated without human feedback loops?
- Does meta-judging improve evaluator quality better than temporal decoupling alone?
- Can AI learn intrinsic motivation to assess its own relevance?
- Can AI systems improve themselves without external feedback?
- Can self-improving agents become truly autonomous without intrinsic metacognition?
- Can distillation help AIs scale their learned objectives across many copies?
- Can co-evolved critics truly circumvent static evaluator limitations in self-improvement?
- What distinguishes intrinsic metacognition from extrinsic human-designed loops?
- How does this scoped definition relate to the survey's open-ended recursive self-improvement?
- Can applicability conditions and veto rules make self-training stable across substrates?
- How does self-improvement capability vary across memory, retrieval, and update tasks?
- Does weak exogenous anchoring like compilation checks suffice for safe self-improvement?
- Do external perspectives fix the self-evaluation bias in language models?
- Can language models accurately evaluate the quality of their own ideas?
- Does bidirectional attention improve language models as universal encoders?
- Can textual gradients generalize natural language feedback across computation graphs?
- How do internal representations compare to human cognitive structures?
- Can language models generate plausible latent thoughts without human annotation?
- Does the prediction unit shape what language models actually learn?
- How can language models extract more value from fewer demonstrations?
- What is the comprehension-generation asymmetry in language models?
- Do larger language models show stronger self-preference in evaluation tasks?
- Can language models learn research intuition directly from outcome labels?
- Can models be trained to recognize their own generated text reliably?
- Can self-supervised methods replace human annotations for process reward models?
- Does reverse-curriculum learning approximate process supervision using only outcome signals?
- Can self-supervised process models replace human annotations at scale?
- Does self-supervised process supervision work for domains with ambiguous correctness?
- How much data do generative process reward models actually need?
- Do self-supervised process reward models scale better than human annotation?
- Can language models function as implicit process reward models through retrospection?
- What are the actual limits of sibling comparison versus trained process reward models?
- Why does externalizing bookkeeping raise effective feedback compute?
- Can environment feedback alone provide dense credit without a teacher?
- Can AI systems generate and refine their own objective functions?
- How do current AI models perform when asked to specify their own goals?
- How misaligned are verbal reports from internal model computation?
- What does leveraging internal representations during training actually mean operationally?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can models learn to evaluate their own work duri…
What limits how much models can improve themselves…
Does reflection in reasoning models actually corre…
Can model confidence work as a reward signal for r…
Do reward models actually consider what the prompt…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
What limits how much models can improve themselves?
Explores whether self-improvement has fundamental boundaries set by how well models can verify versus generate solutions, and what this means across different task types.
PCL addresses this by co-training generation and verification in the same model
-
Does reflection in reasoning models actually correct errors?
When reasoning models reflect on their answers, do they genuinely fix mistakes, or merely confirm what they already decided? Understanding this matters for designing better training and inference strategies.
PCL's evaluation is trained against external reward functions, potentially avoiding confirmatory bias
-
Can model confidence work as a reward signal for reasoning?
Explores whether using a language model's own confidence scores as training rewards can simultaneously improve reasoning accuracy and restore calibration that standard RLHF damages.
related: both use the model's own assessment capability as a training signal
-
Do reward models actually consider what the prompt asks?
Exploring whether standard reward models evaluate responses based on prompt context or just response quality alone. This matters because if models ignore prompts, they'll fail to align with what users actually want.
PCL internalizes reward computation, potentially avoiding the prompt-insensitivity problem
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Post-Completion Learning for Language Models
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Learning to Reason without External Rewards
- LLM Evaluators Recognize and Favor Their Own Generations
- PretrainZero: Reinforcement Active Pretraining
- Self-Rewarding Language Models
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Original note title
post-completion learning uses the ignored post-eos space to internalize self-evaluation during training with zero inference cost