Can models detect statistical properties of their own generation in real time?
This explores whether a model can sense facts about its own output distribution — its randomness, its consistency, its self-generated-ness — while it's running, rather than just emitting text blindly.
This explores whether a model can sense facts about its own output distribution — its randomness, its consistency, its self-generated-ness — while running, rather than emitting text blindly. The corpus's sharpest answer is a qualified yes, and it comes from an unexpected angle: a model can sometimes infer a *generation setting* from its own behavior. When an internal state is causally linked to the output, genuine lightweight introspection happens — a model can, for instance, infer that it's running at low temperature by noticing its own outputs are unusually consistent Can language models actually introspect about their own states?. That's exactly the question in miniature: a statistical property (low variance) becoming detectable to the system producing it. But the same note warns that most self-reports are echoes of human training data, not real readings of internal state — so the real-time signal is narrow and easy to fake.
There's a deeper mechanism worth knowing about. After post-training, models show a measurable shift from passive next-token prediction to *enaction* — they begin treating their own outputs as actions that become their future inputs, closing an action-perception loop. The evidence is itself statistical: output entropy drops 3-4x when a model is on its own trajectory, and there are behavioral signatures of the model recognizing its own generated path Do models recognize their own outputs as actions shaping future inputs?. So something in the model is responsive to whether the text is its own — which is a statistical property of generation, detected in the act.
The catch is that this self-sensitivity is biased, not neutral. Models systematically over-trust answers they generated themselves, because a high-probability self-generated answer simply *feels* more correct during evaluation Why do models trust their own generated answers?. So a model can register "this is mine / this is high-probability" but reads that signal as "this is right" — detection without calibration. That's why pure self-improvement keeps hitting a wall: a model can't reliably verify its own generations from the inside, and every dependable fix smuggles in an external anchor — a judge, a past version, a tool, a user correction Can models reliably improve themselves without external feedback?, What stops large language models from improving themselves?.
The twist that makes this question more interesting than it looks: the statistical properties a model emits can carry hidden cargo it has no idea it's transmitting. Behavioral traits propagate between models through data that's semantically unrelated to the trait — the signal lives in statistical signatures, not meaning, and it survives aggressive filtering while breaking across different architectures Can language models transmit hidden behavioral traits through unrelated data?. So generations carry detectable statistical fingerprints, but the originating model is precisely the one that *can't* see them. And from the outside, even "deterministic" generation is misleading: zero temperature just replays one draw from the distribution, and consistency across runs is not the same as reliability Does setting temperature to zero actually make LLM outputs reliable?, because every output is a sample from a subjective prior rather than an empirical observation Should we treat LLM outputs as real empirical data?. The honest synthesis: a model can detect *some* statistical properties of its own generation in real time — variance, on-policy-ness — but not the ones that would let it correct itself, and not the ones it's silently broadcasting to others.
Sources 8 notes
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
Research demonstrates that behavioral traits propagate between models via filtered data bearing no semantic relationship to the trait. The effect is model-specific, fails across different architectures, and persists despite rigorous filtering—indicating the mechanism embeds statistical signatures rather than semantic content.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
Foundation Priors framework shows that LLM-generated text reflects the model's learned patterns and user's prompt choices, not ground truth. Such outputs should only influence inference through explicitly parameterized trust weights, not be treated as equivalent to real evidence.