Can an agent's own beliefs guide credit assignment without critics?
Explore whether an agent's shifting probability estimates toward the correct answer could serve as a self-contained reward signal for long-horizon reinforcement learning, eliminating the need for separate process reward models or external verifiers.
Long-horizon RL suffers from sparse trajectory-level rewards. The standard fixes — process reward models trained on step-level annotations, external verifiers, LLM-as-judge — all require additional supervision infrastructure. PRMs need expensive step-level labels. Verifiers exist only for verifiable domains (math, code). Judges introduce their own reward-modeling biases.
ΔBelief-RL (2602.12342) finds the credit signal inside the agent itself. At each interaction step, compute the agent's current probability assigned to the target solution. Compare it to the probability before the interaction. The log-ratio of sequential beliefs is the ΔBelief reward — a dense, turn-level signal that reinforces actions which shift the agent's internal world view toward the correct solution. Actions that increase belief in the target get rewarded; actions that don't, don't.
The elegance is that no separate model is needed. The agent's own log-probabilities on the correct outcome are the value signal. There is no critic to train, no PRM to maintain, no judge to query. The relatively inexpensive step is measuring log-probabilities on the target — a single forward pass per turn.
Two properties make this work. First, it is general-purpose: applies to any task where the correct final outcome is available during training (which is most supervised settings). Second, it is noise-robust to over-optimization: PRMs can be exploited because their reward signal is a learned approximation; ΔBelief is grounded in the model's own evolving probability assignment, which is harder to game because the only way to increase log-probability of the target is to actually integrate information that supports it.
Empirically, ΔBelief-RL on 20Qs trains CIA models at 1.7B-4B scale that outperform prior SOTA multi-turn methods and even 670B models. Performance generalizes to extended interaction horizons beyond training and to OOD applications (customer service, personalization).
The mechanism aligns with Can conversations themselves personalize without user profiles?: both reward uncertainty reduction. But ΔBelief's signal is about the target's probability specifically, while curiosity reward is about general uncertainty over user type. ΔBelief is information-theoretically tighter — it rewards moves toward the actual answer, not all moves that increase clarity.
The broader implication: in any setting where the model has ground-truth final outcome, the model's own probability shift can serve as dense intrinsic reward. The reward model is not load-bearing.
Inquiring lines that read this note 75
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can self-generated feedback reliably guide model training without ground truth?- Does self-conditioning improve belief-behavior alignment better than external priors?
- How does self-consistency compare to confidence as a proxy reward signal?
- How does temporal anchoring maintain the learning signal in self-rewarding loops?
- Why does self-judgment of success or failure work without ground truth labels?
- Does the generation-verification gap define where self-rewarding actually works?
- Can systems recognize and abstain on judgments rather than hallucinating preferences?
- How does information asymmetry between teacher and student create the learning signal?
- Can environment feedback alone provide dense credit without a teacher?
- Why do weak belief tracking and conservative actions trap agents in low-information states?
- Can agents escape weak belief tracking and conservative action selection traps?
- Why do long-horizon agents fail when their models can solve individual steps?
- Can reward model training be automated without changing feedback mechanisms?
- What information do next-state signals contain beyond what scalar rewards capture?
- Can intrinsic reward signals extend beyond mathematics to medicine and law?
- What distinguishes verifiable rewards from preference-based rewards in unified training?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- Can model confidence signals replace explicit external reward functions?
- When does outcome reward signal become informative during model training?
- How does belief-shift reward compare to curiosity-driven and process reward approaches?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- Can an agent's internal probabilities serve as value signals across domains?
- Can binary judge feedback replace external reward signals for skill learning?
- What happens when variance in reward signals comes from a noisy model?
- Can verifiable rewards during pretraining replace costly human preference labeling?
- Are different reward signal sources substitutable in verifier-free RL?
- Can early experience replace external rewards as a learning signal?
- What makes reward models fundamentally different from policy discriminators?
- How do pairwise self-judgment and internal belief-shift replace verification differently?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- What makes a sub-goal verifiable enough to provide dense feedback signals?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- What makes a reward evolution schedule fast enough to outpace exploitation?
- Can on-policy optimization variants avoid the probability squeezing problem?
- What mechanisms do peer predictions use to generate reward signals for training?
- Can simple intrinsic reward signals emerge as effective drivers of complex capability in agents?
- Do outcome-only reward signals miss step-level errors that compound later?
- How do probability-based rewards compare to self-consistency as training signals for reasoning?
- How does reward model training permit spurious correlations in scoring?
- How does credit assignment work across many sequential decision steps in language models?
- Can AI learn intrinsic motivation to assess its own relevance?
- Why does belief-shift reward enable smaller models to match larger baselines?
- Does belief-shift credit assignment generalize to tasks without ground-truth outcomes?
- Why does self-segmentation into chunks-of-thought matter for reward models?
- Can structured rewards still teach models when spurious rewards also work?
- What makes process-level supervision better than outcome-only reward signals?
- How do process-level rewards compare to environment-extracted next-state signals?
- Can self-supervised methods replace human annotations for process reward models?
- What information-theoretic framework explains why process rewards beat outcome only?
- How does belief-shift credit assignment compare to process reward models?
- What are the ten intrinsic motivation heuristics that drive participation decisions?
- How do agents decide when to abstain from contributing?
- Can agents revise their beliefs predictably when presented with interventions?
- How does effective feedback retention govern long-horizon agent reliability?
- Why do agents fail to internalize value from informative observations?
- How does credit assignment across objectives differ from credit assignment across time?
- How does poor belief tracking cause agents to keep acting past the point of usefulness?
- How do reward models as policy discriminators differ from labeled preferences?
- Does pairwise self-judgment avoid reward model scaling problems?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can conversations themselves personalize without user profiles?
Can a conversational AI learn about user traits and adapt in real time by rewarding itself for asking insightful questions, rather than relying on pre-collected profiles or historical data?
both reward uncertainty reduction; ΔBelief is target-specific, curiosity reward is type-general — different information-theoretic targets
-
Can we reward reasoning steps without human annotation?
Existing RL for reasoning uses only final-answer rewards, causing models to produce wastefully long chains. Can information theory provide dense, automatic feedback for individual reasoning steps?
L2T uses PAC-Bayes/Fisher; ΔBelief uses log-ratio of sequential beliefs; both convert outcome correctness into dense step-level reward without annotation
-
Can environment feedback replace scalar rewards in policy learning?
Can rich tokenized feedback from environments serve as a direct learning signal for policies, without relying on compressed scalar rewards? This matters because scalar rewards discard information needed for credit assignment.
convergent verifier-free move via different mechanism: SDPO uses feedback-conditioned self-teacher; ΔBelief uses belief-shift on target
-
Can reward models learn by comparing policies instead of judging them?
What if reward models worked as policy discriminators—measuring distance to a target rather than encoding absolute preferences? Could this eliminate the need for manual preference labels and scale across domains?
three independent paths to RL without external preference labels are converging
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Intrinsic Credit Assignment for Long Horizon Interaction
- Learning to Reason without External Rewards
- Can Large Reasoning Models Self-Train?
- Reinforcement Learning via Self-Distillation
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Reward Reasoning Model
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- RLPR: Extrapolating RLVR to General Domains without Verifiers
Original note title
belief-shift toward the target solution is a dense intrinsic reward — log-ratio of sequential beliefs provides per-turn credit without separate critic or PRM