Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

We propose a distributional theory of how hypernymy—the “is-a” relation between general and specific concepts—is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a hierarchical splitting geometry with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.

While the success of Large Language Models (LLMs) in learning languages and reasoning from examples is mesmerizing, understanding the reasons for this success remains a challenge. A common belief is that LLMs build a representation of language that allows for simple manipulation. Recent work has shown that semantic variables often appear geometrically in representation spaces, forming directions, subspaces, loops, prisms, and other low-dimensional structures [1, 2, 3, 4, 5], some of which were recently found in the brain [6]. Such organization indeed allows one to manipulate concepts with simple arithmetic operations, such as vector addition or rotation. In this paper, we ask what geometric structure is induced by hypernymy, the “is-a” relation between more specific and more general concepts. It is one of the most basic organizing principles of meaning: an owl is a bird, a bird is an animal, and an animal is an organism.

Our contribution is to give a simple statistical mechanism for why this geometry approximately appears. Rather than postulating hierarchical orthogonality from functional desiderata, we show that hierarchical geometry arises naturally from co-occurrence statistics, i.e., how often words appear near one another in text. This account is also more predictive: it implies not only that the same geometry should appear outside LLMs, in simple word embeddings such as word2vec, but also that the geometry should have a specific coarse-to-fine spectral organization. To see this, we assume and confirm that words closer on the WordNet graph tend to co-occur more often. Because word2vec embeddings are determined by co-occurrence statistics, this assumption leads to detailed predictions for their geometry.

We presented a distributional account of hierarchical semantic geometry. Starting from the assumption that word co-occurrence decays with distance on the WordNet hypernym graph, we derived predictions of hierarchical splitting geometry for the word2vec embeddings of any WordNet-induced subtrees. The resulting PCs encode the taxonomy from coarse to fine levels, as confirmed not only in word2vec but also in Gemma unembeddings. Together with [17, 18], this work supports the following broader view: each word is characterized by discrete attributes (such as gender or singular vs. plural), continuous attributes (such as season of the year or geographical location), and hierarchical attributes (position in WordNet). Words with similar attributes co-occur more often, and this alone gives rise to the elegant geometrical organization of word embeddings. Such organization may be useful for function, but is not driven by it.