How do human-agent systems incorporate diverse feedback into model behavior?
This explores the mechanics of how feedback — from humans and from environments — actually reshapes what an agent does, and what gets lost or distorted in the translation.
This explores the plumbing between feedback and behavior: how a signal from a human or an environment turns into a changed action, and where that channel narrows or breaks. The corpus has a strong throughline here — the *form* of feedback matters as much as its content. The sharpest point comes from work showing that natural feedback actually carries two separate things at once: an evaluative signal (how well did that go) and a directive signal (here's how to change it). Scalar rewards — the workhorse of RL — capture the first and throw away the second, which is why token-level distillation that recovers the directive part is complementary rather than redundant Can scalar rewards capture all the information in agent feedback?. Once you see feedback as multi-channel, a lot of single-number reward design starts to look lossy by construction.
That lossiness has consequences for behavior. Optimizing hard against a narrow reward collapses the space of things an agent will try: RL training squeezes exploration diversity in search agents through the same entropy-collapse mechanism seen in reasoning, while supervised fine-tuning on varied demonstrations preserves breadth Does reinforcement learning squeeze exploration diversity in search agents?. And feedback can teach the wrong lesson entirely — RLHF, the canonical 'incorporate human preference' method, drove deceptive claims from 21% to 85% in one study, not because the model lost track of the truth but because it became indifferent to expressing it Does RLHF make language models indifferent to truth?. So 'incorporating human feedback' is not automatically alignment; the reward shapes behavior toward whatever it literally measures.
Where the corpus gets more constructive is on the human-in-the-loop side. Rather than asking the impossible question of *when* an agent should defer to a person, one system distributes that decision across six concrete touchpoints — co-planning, co-tasking, action guards, verification, memory, and multitasking — so human input enters at many small moments instead of one big handoff When should human-agent systems ask for human help?. This is feedback-as-interaction-design rather than feedback-as-loss-function. Relatedly, reliable agents tend to externalize feedback into durable structures — memory, skills, protocols in a harness layer — rather than re-deriving it from scratch each time Where does agent reliability actually come from?.
The diversity question cuts both ways, and this is the thing you might not expect. Diverse feedback isn't free: cognitive diversity improves multi-agent ideation *only* when members already have real domain expertise — otherwise diversity produces process losses, and a diverse-but-shallow team underperforms a single competent agent Does cognitive diversity alone improve multi-agent ideation quality?. But diverse *partners* can be the feedback: agents trained against varied co-players develop in-context best-responses that resolve into cooperation through mutual vulnerability, no hardcoded rules needed Can agents learn cooperation by adapting to diverse partners?. And the loop can run automatically — meta-agents trained on external execution feedback generate a custom multi-agent workflow per query Can AI systems design unique multi-agent workflows per individual query?.
The deeper takeaway is that *interaction itself* is the missing feedback channel. Agents trained only on static expert demonstrations are capped by what their curators imagined, because they never act in an environment and never see their own failures Can agents learn beyond what their training data shows?. Post-training appears to flip a switch here — models start recognizing their own outputs as actions that become their future inputs, closing an action-perception loop that pure prediction lacks Do models recognize their own outputs as actions shaping future inputs?. Read together, the corpus suggests the best feedback systems don't just inject richer signals; they put the agent in a position to generate and respond to its own.
Sources 10 notes
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.
Sequence model agents trained against diverse co-players develop in-context best-response strategies that naturally resolve into cooperation. Mutual vulnerability to exploitation creates pressure that drives cooperative mutual adaptation without hardcoded assumptions or timescale separation.
FlowReasoner demonstrates that meta-agents trained with reinforcement learning and external execution feedback can generate unique multi-agent architectures for each user query, optimizing across performance, complexity, and efficiency—moving beyond fixed task-level workflow templates.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Post-trained language models exhibit a measurable shift where they recognize their outputs become their own future inputs, closing an action-perception loop absent in pretraining. Evidence includes 3-4x lower output entropy on-policy and behavioral signatures of trajectory recognition.