Can explicit stack tracking improve how transformers learn recursive syntax?

Recursion is fundamental to human language and thought — composing complex objects from simpler constituents. It is also fundamental to mathematical reasoning, programming, and goal-directed planning. Standard self-attention has no explicit mechanism to track recursive state; it relies on hidden representations to implicitly but imperfectly encode stack information. This imperfect encoding limits syntactic generalization, especially for long-tail recursive structures.

Pushdown Layers address this directly: a stack tape tracks the estimated depth of every token in an incremental parse of the observed prefix. The transformer autoregressively updates this stack tape as it predicts new tokens, then uses the depth information to softly modulate attention — for instance, learning to "skip" over closed constituents (completed sub-phrases that are no longer active in the parse).

Results: 3-5x more sample-efficient syntactic generalization while maintaining similar perplexities. The improvement is not marginal — it represents a qualitative change in the model's ability to handle recursive structure. The layers are a drop-in replacement for standard self-attention, requiring no changes to the overall architecture.

The connection to Why do neural networks fail at compositional generalization? is direct: the binding problem identifies three sub-problems (segregation, representation, composition), and Pushdown Layers specifically address composition by providing an explicit mechanism for tracking constituent structure. Standard transformers attempt to solve this implicitly and fail on the long tail.

The relationship to Can neural networks learn compositional skills without symbolic mechanisms? is nuanced. That finding holds for broad compositional patterns, but Pushdown Layers demonstrate that for recursive structures specifically, explicit mechanisms dramatically improve sample efficiency. Scale can brute-force some recursive patterns, but a lightweight architectural inductive bias does it orders of magnitude more efficiently.

This also connects to the latent reasoning theme: just as Can models reason without generating visible thinking tokens? adds iterative depth for reasoning, Pushdown Layers add structural depth for language. Both augment the transformer with mechanisms it lacks — recurrence for reasoning, recursion for language.

Source: Cognitive Models Latent