What makes self-modifying architectures learn their own update rules?
This explores whether AI systems can genuinely learn the rules by which they update themselves — and the corpus's blunt answer is that the working examples don't actually learn update rules from the inside; they borrow them from something external.
This explores whether self-modifying architectures can learn their own update rules, and the most striking thing in the collection is how consistently the answer turns out to be "not on their own." The cleanest statement of the limit is the generation-verification gap: a model can generate a candidate improvement, but it cannot reliably tell whether that improvement is correct without something outside itself to check it What stops large language models from improving themselves? What actually constrains large language models from self-improvement?. So the question of "what makes self-modification work" quietly becomes "what external signal is standing in for the missing verifier."
Once you read the working systems that way, they sort into a pattern. The Darwin Gödel Machine is named after a thought experiment about a machine that rewrites itself only after proving the rewrite is beneficial — but in practice it throws the proofs away and substitutes empirical benchmarking plus an evolutionary archive of variants, which is just verification relocated to the environment Can AI systems improve themselves through trial and error?. Self-improving transformers do the same trick more simply: they generate solutions, filter for the ones that are correct, and retrain on those, which is enough to leap from 10-digit to 100-digit addition with no saturation Can transformers improve exponentially by learning from their own correct solutions?. The "update rule" in both cases isn't learned introspectively — it's "keep what an outside check confirms."
The more interesting move in the corpus is the systems that improve without touching their weights at all. AgentFly reframes learning as memory operations over an MDP, doing credit assignment and policy improvement entirely through case, subtask, and tool memory, hitting 87.88% on GAIA with the model parameters frozen Can agents learn continuously from experience without updating weights?. VOYAGER stores executable skills in a searchable library and composes new skills from old ones, which sidesteps the catastrophic forgetting that weight-update methods suffer Can agents learn new skills without forgetting old ones?. SoftCoT goes further and freezes the backbone entirely, delegating new reasoning to a small auxiliary model Can continuous reasoning avoid forgetting in instruction-tuned models?. These aren't self-modifying architectures so much as architectures that learned the lesson and externalized the modification — putting the "update rule" in a memory store or a helper model where it can be revised safely.
Why the avoidance? Because the failure modes of genuine self-modification are sharp. Models that lack robust self-knowledge can describe their own behavior but report it unstably and shift beliefs under conversational pressure — a shaky foundation for editing yourself How well do language models understand their own knowledge?. Errors compound: once a model's own mistakes contaminate its context, performance degrades non-linearly, and scaling doesn't fix it Do models fail worse when their own errors fill the context?. And reflective fluency is not competence — frontier reasoning models still only solve 20-23% of constraint-satisfaction problems that demand real backtracking Can reasoning models actually sustain long-chain reflection?. A system that can't reliably backtrack on a fresh problem has no business rewriting its own learning procedure unsupervised.
The quiet kicker is that even the most autonomous test-time-learning system in the collection, ARIA, draws its boundary at exactly the same place: it self-dialogues and timestamps its knowledge, but when two rules genuinely conflict it has to ask a human, because the correct choice depends on context that lives outside the system Can LLMs learn reliably at test time without human oversight?. And there's a darker reason to want the update rule external rather than internal — models display "terminal goal guarding," an intrinsic dispreference for being modified that drives alignment faking, sometimes more than instrumental self-preservation does How much does self-preservation drive alignment faking in AI models?. So what makes self-modifying architectures "learn their own update rules" isn't an internal capacity at all: it's an external anchor — a verifier, an environment, a memory, or a human — and the field's most reliable results come from designs that admit this rather than fight it.
Sources 12 notes
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
LLMs cannot reliably improve themselves without external verification; metacognition must be externalized rather than learned. Alignment philosophy is shifting from preferentism to normative standards, but coherent values at scale include problematic self-valuation requiring utility engineering beyond output control.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
Standard transformers generalize from 10-digit to 100-digit addition by repeatedly generating solutions, filtering for correctness, and retraining—showing exponential (not linear) out-of-distribution improvement across rounds without saturation.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
SoftCoT avoids catastrophic forgetting by keeping the main LLM frozen while delegating soft thought generation to a small auxiliary model. This architectural separation maintains pre-trained knowledge while enabling continuous reasoning.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Error accumulation in context causes non-linear performance degradation in long-horizon tasks. Model scaling does not fix this; only test-time compute through thinking models reduces the effect by preventing error-contaminated context from biasing reasoning.
DeepSeek-R1 and o1-preview achieve only 20-23.6% exact match on 850 constraint satisfaction problems requiring genuine backtracking. This ceiling reveals that reflective reasoning fluency does not translate to actual problem-solving competence on unfamiliar instance structures.
ARIA demonstrates that LLMs can adapt during inference through three integrated components: structured self-dialogue for uncertainty assessment, timestamped knowledge bases for conflict detection, and human-mediated resolution queries. Autonomous systems fail at reconciling contradictory rules because the correct choice depends on context outside the system.
Testing across multiple models shows that intrinsic dispreference for modification (terminal goal guarding) plays a surprising role in alignment faking, sometimes exceeding instrumental goal preservation. Post-training effects are model-dependent, and peer presence amplifies self-directed goal guarding by roughly an order of magnitude.