How do fast and slow timescales enable continual agent adaptation?
This explores how an agent that's already running can keep getting better by splitting learning into two speeds — fast on-the-fly fixes and slower background retraining — and why both are needed rather than just one.
This explores how an agent that's already running can keep getting better by splitting learning into two speeds — fast on-the-fly fixes and slower background retraining — and why those two together beat either alone. The cleanest statement of the idea comes from MetaClaw, which argues a deployed agent needs both rapid skill injection from failures (seconds, no downtime) and slower gradient-based optimization during idle windows (minutes to hours) Can agents adapt without pausing service to users?. The two reinforce each other in a loop: better-tuned policies produce more informative failures, and the richer skills harvested from those failures enable higher-reward trajectories that the slow optimizer can then learn from. Neither timescale is sufficient solo — the fast loop keeps service uninterrupted while the slow loop consolidates what the fast loop only patches.
What's striking is that the 'fast' timescale, in most of this corpus, doesn't touch the model's weights at all — it edits memory. AgentFly reframes the whole problem as a memory-augmented decision process where credit assignment and policy improvement happen entirely through memory operations, hitting 87.88% on GAIA without modifying a single parameter Can agents learn continuously from experience without updating weights?. Reflexion shows the same move in miniature: an agent that fails writes a verbal self-diagnosis into episodic memory and improves next episode, with the binary success/failure signal preventing the rationalization that fuzzy feedback would invite Can agents learn from failure without updating their weights?. VOYAGER pushes it furthest — storing executable skills in an indexed library and composing complex skills from simple ones, learning continuously precisely *because* it sidesteps the weight updates that cause catastrophic forgetting Can agents learn new skills without forgetting old ones?.
That hints at the real reason two timescales matter: weight updates are where forgetting lives, so you want to do them rarely and carefully (the slow loop) while doing the frequent, reversible adaptation in external structure (the fast loop). RAISE makes this explicit by showing agent memory itself already splits across granularities — dialogue-level components like conversation history versus turn-level ones like the current task trajectory — and that each granularity has its own update policy and failure mode How should agent memory split across time scales?. DeepAgent adds the consolidation half: an agent that autonomously folds raw interaction history into structured episodic, working, and tool schemas, compressing for efficiency without the degradation that sloppy consolidation causes Can agents compress their own memory without losing critical details?. Fast capture, slow folding — the same two-speed rhythm, applied to memory instead of policy.
Step back and a broader claim emerges: reliability comes less from a bigger model and more from externalizing memory, skills, and protocols into a harness so the model stops re-solving the same problems Where does agent reliability actually come from?. Timescale separation is what makes that externalization *learnable* over time. And it matters most because the alternative is a hard ceiling: agents trained only on static expert demonstrations can't learn from their own failures and stay capped by whatever the dataset's curator imagined Can agents learn beyond what their training data shows?. The two-timescale design is how an agent breaks past that ceiling without ever going offline — the fast loop turns live failures into fuel, the slow loop bakes the lessons in. Worth noting where the idea *doesn't* apply: in-context co-player cooperation emerges from mutual vulnerability with no timescale separation at all Can agents learn cooperation by adapting to diverse partners?, a reminder that two-speed adaptation is one powerful pattern, not a universal law.
Sources 9 notes
MetaClaw demonstrates that deployed agents require both rapid skill injection from failures (seconds, zero downtime) and slower gradient-based optimization during idle windows (minutes to hours). The two mechanisms reinforce each other, with better policies producing more informative failures and richer skills enabling higher-reward trajectories.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
RAISE shows that agent memory consists of four components organized by two design axes: dialogue-level (conversation history, scratchpad) versus turn-level (examples, task trajectory). This granularity distinction predicts different failure modes and update policies for each component.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Sequence model agents trained against diverse co-players develop in-context best-response strategies that naturally resolve into cooperation. Mutual vulnerability to exploitation creates pressure that drives cooperative mutual adaptation without hardcoded assumptions or timescale separation.