Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach where models iteratively generate and learn from their own solutions, progressively tackling harder problems while maintaining a standard transformer architecture. Across diverse tasks including arithmetic, string manipulation, and maze solving, self-improving enables models to solve problems far beyond their initial training distribution—for instance, generalizing from 10-digit to 100-digit addition without apparent saturation. We observe that in some cases filtering for correct self-generated examples leads to exponential improvements in out-of-distribution performance across training rounds.

Self-improvement has been widely studied in various contexts (Singh et al., 2023; Gulcehre et al., 2023; Liang et al., 2024), typically in settings where external verifiers, weak supervision, or filtering mechanisms are used to ensure data quality. We demonstrate that extreme length generalization is indeed possible under this framework, without any modification to the base transformer architecture.