Can formal language pretraining address surface generalization without learning true linguistic structure?

This explores whether formal-language pretraining genuinely installs grammatical structure in a model, or merely sharpens the surface tricks that let models *look* like they know grammar. The corpus stages this as a real tension rather than settling it. On the optimistic side, pretraining 1B models on hierarchical formal languages doesn't just save tokens — it improves *syntactic* generalization, and the attention heads forged on those formal patterns stay load-bearing when the model later handles natural language Can formal language pretraining make language models more efficient?. That persistence is the strongest hint that something structural, not cosmetic, is being learned: the formal scaffolding survives the transfer.

But the skeptical thread cuts hard against reading that as 'true structure.' BabyLM-style evaluations show models routinely produce grammatically correct outputs by leaning on sentence length, word choice, and spelling — surface heuristics that mimic rules without being rules — and that standard benchmarks literally cannot tell the two apart unless they're designed to rule out the shortcuts Can models pass tests while missing the actual grammar?. So the very 'syntactic generalization' that formal pretraining improves may itself be measured by tests that surface heuristics can pass. The improvement is real; what it *is* remains contested.

The place to look for the seam is structural complexity. Top-tier models systematically misidentify embedded clauses, complex verb phrases, and deep nominals, and crucially the failure worsens *predictably* as syntactic depth increases Why do large language models fail at complex linguistic tasks?. That predictable degradation is a signature: genuine rule-knowledge wouldn't fray with depth the way a pattern-matcher does. If formal pretraining taught real recursive structure, you'd expect that curve to flatten — testing it there, rather than on aggregate scores, is where the question actually gets answered.

Step back and there's a deeper ceiling the corpus keeps circling. Even perfect formal structure is structure *over form* — and the form-only argument holds that meaning needs the relation between expressions and communicative intent, which form-to-form prediction can never supply Can language models learn meaning from text patterns alone?. The counterpoint reframes rather than refutes this: models operationalize Saussure's *langue*, learning a fully relational system where structure emerges from how symbols differentiate each other, no external referent required Can language models learn meaning without engaging the world?. Read together, these suggest formal pretraining might genuinely teach *relational* structure — the internal differential system — while telling us nothing about whether the model grasps what language is *for*.

So the honest answer is layered. Formal pretraining demonstrably does more than dress up surface generalization — its learned heads transfer and persist. But 'addressing surface generalization' and 'learning true linguistic structure' aren't a clean binary: the corpus suggests models can acquire real *relational* structure that still degrades with depth and still lacks grounded meaning. The interesting thing you didn't know you wanted to know is that the bottleneck may not be the training signal at all — it's that our benchmarks can't distinguish the two outcomes, so we've been unable to tell which one formal pretraining actually buys.