What do language models actually know?

How language models know what they know, where they fail, and why the limits are structural not accidental.

Topic Hub · 60 linked notes · 8 sections

View as

Sub-Topic-Maps

3 notes

How do LLMs fail to know what they seem to understand?

This explores the specific, repeatable ways LLMs track language patterns without genuine understanding. Why do models explain concepts correctly but fail to apply them, or possess knowledge that doesn't influence their outputs?

How well do language models understand their own knowledge?

Explores whether LLMs have genuine self-awareness about what they know and can do, and how this self-knowledge (or lack thereof) shapes human-AI interaction dynamics and user trust.

What makes a world model actually useful for reasoning?

Exploring whether language models develop genuine world models that simulate possibilities rather than merely predict sequences. The distinction matters because accurate prediction doesn't guarantee the underlying mechanism was learned.

Formal Properties and Limits

7 notes

Do classical knowledge definitions apply to AI systems?

Classical definitions of knowledge assume truth-correspondence and a human knower. Do these assumptions hold for LLMs and distributed neural knowledge systems, or do they need fundamental revision?

Can any computable LLM truly avoid hallucinating?

Explores whether formal theorems prove hallucination is mathematically inevitable for all computable language models, regardless of their design or training approach.

Why can't language models reverse learned facts?

Language models trained on directional statements like "A is B" often fail to answer the reverse query. This explores why symmetric relations aren't automatically learned during training, despite appearing throughout the data.

Can text-trained models compress images better than specialized tools?

Do general-purpose language models trained only on text outperform domain-specific compressors like PNG and FLAC on their native data? This tests whether compression ability is universal or requires domain specialization.

Can LLMs reconstruct censored knowledge from scattered training hints?

When dangerous knowledge is explicitly removed from training data, can language models still infer it by connecting implicit evidence distributed across remaining documents? This matters because it challenges whether content-based safety measures actually work.

Why do neural networks fail at compositional generalization?

Exploring whether the binding problem from neuroscience explains neural networks' inability to systematically generalize. The binding problem has three aspects—segregation, representation, and composition—each creating distinct failure modes in how networks handle structured information.

When do language models stop memorizing and start generalizing?

Can we measure the exact capacity limit where models transition from memorizing training data to learning underlying patterns? Understanding this boundary could reshape how we think about model learning and privacy.

Foundation Priors and Epistemic Status of AI Outputs

3 notes

Should we treat LLM outputs as real empirical data?

Can synthetic text generated by language models serve as evidence in the same way observations from the world do? This matters because researchers increasingly rely on AI-generated content without accounting for its fundamentally different epistemic status.

Do foundation models actually reduce our need for real data?

As AI systems grow more powerful, does empirical observation become less necessary? This explores whether foundation models can substitute for ground truth or whether they instead demand stronger empirical anchoring.

How much should we trust AI-generated data in inference?

Most AI workflows treat synthetic data with implicit full trust, but should there be an explicit parameter controlling how heavily AI outputs influence downstream reasoning and decision-making?

Forward-Looking vs Backward-Looking Knowledge

1 note

Can LLMs predict novel scientific results better than experts?

Do language models excel at forecasting experimental outcomes in neuroscience when given only method descriptions? This challenges the assumption that LLMs are mere knowledge retrievers rather than pattern integrators.

Epistemic Properties of Expertise

4 notes

Can AI distinguish which differences actually matter?

Explores whether AI systems can perform the qualitative judgment that experts use to select relevant observations. Matters because confusing AI outputs with expert observation leads users to trust pattern-matching as if it were reasoning about what's important.

Can AI ever gain expert community trust through participation?

Explores whether AI can accumulate the social capital and track record that human experts build within their communities. Questions whether prediction of social norms equals genuine participation in expert validation processes.

Is expertise really just knowing more than others?

This explores whether expertise is fundamentally about possessing domain knowledge, or whether the ability to deploy that knowledge in the right moment, context, and way with the right audience is equally or more central to what makes someone an expert.

Why does rigorous-sounding AI commentary often misdiagnose how models work?

Expert commentary on AI frequently cites real research and sounds carefully reasoned, yet reaches conclusions built on unwarranted cognitive attributions. What makes this pattern so persistent in AI analysis?

Tokenization of Intelligence Framework

37 notes

Does AI actually commodify expertise or tokenize it?

The standard framing treats AI output like mass-produced commodities, but does AI's contextual, mutable nature fit better with token economics than commodity theory?

Where does the value of AI output actually come from?

If AI-generated intelligence has no intrinsic content-value like physical goods do, what determines whether it's valuable to someone? This explores whether value lives in the token or the receiver.

Is the LLM a tool or a new form of intelligence itself?

Does framing AI as merely delivering pre-existing intelligence miss what's actually happening? This explores whether the model itself constitutes a fundamentally new intelligence-medium with distinct cultural effects.

Is AI fundamentally changing how value gets produced?

Rather than automating commodity production, does AI represent a shift from making identical stockpiled objects to generating contextual tokens on demand? And what makes this genuinely new?

Does Marxist alienation theory explain what AI does to cognitive work?

Marxist alienation frames AI as degrading authentic labor. But does that framework actually describe the shift happening with tokenization, or does it misdiagnose the transformation occurring in intelligence itself?

Why does AI output change with every prompt and context?

Explores whether the variability of AI-generated intelligence across contexts and audiences is a fundamental feature or a flaw to be fixed. Examines what this mutability means for how we should evaluate and understand AI systems.

Can exchange value exist entirely without use value?

Does AI-generated knowledge represent a genuinely new category of goods where exchange-value (market price, social credibility) operates independently of use-value (actual accuracy, practical utility)? This matters because it suggests AI disrupts markets in ways Marx's commodity analysis did not predict.

Does AI abundance actually devalue knowledge itself?

If AI generates vastly more claims than humans can evaluate, does the sheer volume undermine the social processes that normally establish what counts as reliable knowledge? And what would that erosion look like?

Can AI generate knowledge faster than humans can evaluate it?

Explores whether AI-driven content production is outpacing human judgment capacity, mirroring monetary hyperinflation dynamics. Why this matters: understanding this gap reveals whether our evaluation infrastructure can sustain epistemic confidence.

Why do search tools fail against AI generated content?

Internet search worked for finding needles in haystacks of fixed documents. But AI generates new content on demand with no underlying corpus to search. Does this require fundamentally different solutions?

Why can't search tools handle AI-generated content?

Search infrastructure was built for stable, pre-existing items. AI generates ephemeral content on-demand. Can the indexing tools that solved information overload work when there's nothing stable to index?

What actually backs the value of AI-generated intelligence?

If AI produces intelligence tokens at near-zero cost, what constrains their value and prevents inflation? Exploring whether training data, expert validation, or statistical probability can serve as a genuine backing mechanism.

Can we still verify AI knowledge if verification itself is AI-generated?

When the tools we use to distinguish genuine expert knowledge from AI facsimile are themselves AI-generated, does verification become circular? This explores whether expertise can survive the collapse of independent testing criteria.

When do users stop checking whether AI output is actually backed?

What causes users to accept AI-generated content at face value without verifying its basis? Understanding this receiver-side acceptance reveals how intelligence-token systems maintain value despite lacking real backing.

How does AI writing escape the conversations that govern knowledge?

If knowledge claims normally get filtered and refined through social discourse, what happens when AI generates claims outside that governing process? Why does scale matter here?

Does AI generate diverse claims or diverse perspectives?

When AI produces thousands of articles on a topic, does that create genuine argumentative diversity? Or does scaling claim-generation without scaling perspective-generation result in apparent but not real diversity?

Why does AI discourse feel obscene in Baudrillard's sense?

Explores whether AI-generated arguments lack the relational and productive scenes that normally make discourse meaningful, creating a disembedded visibility that resembles obscenity in Baudrillard's technical sense.

How do we learn to read AI-generated text critically?

Publics have developed interpretive postures toward journalism, advertising, and scholarship over time. But AI discourse arrived too suddenly for any cultural discount to form, raising questions about how we might develop one.

Does AI writing collapse the author-to-public relationship?

When AI generates text optimized for a prompter's satisfaction rather than a public audience, what happens to the core practice of writing for readers you don't know? This explores whether AI reorganizes the structural relationship between author, text, and public.

Does AI content displace human influencers on social media?

Explores whether AI-generated posts that circulate without an identifiable author undermine social media's reputation-building function and crowd out human creators competing for attention.

Why do AI posts get likes without inviting conversation?

Exploring why AI-generated social media content accumulates visibility metrics through comprehensiveness and authority, yet fails to generate the reply-and-counter-reply dynamics that normally validate social proof.

Does AI writing lack the internal appeal to attention that humans use?

Explores whether AI-generated text is structurally missing the constitutive property of human communication — an internal gesture that reaches for and holds the reader's attention, not just inheriting visibility from platforms.

Does AI threaten social media's conversational function?

Explores whether AI-generated posts undermine social media's value as a space for dialogue and idea-testing, beyond just sentiment or topic manipulation. Why this structural threat matters more than content-level problems.

Why do LLMs produce such different writing in chat versus posts?

Explores whether the shift from deferential conversation to confident declarations reflects distinct generation modes or stylistic variation, and what training conditions produce this split.

Can language models actually raise alarm about threats?

Explores whether LLMs can perform the social act of raising alarm—which requires interpersonal address, internal concern, and proactive reaching for attention—or whether they can only mimic alarm-shaped outputs when prompted.

Does alignment training suppress socially necessary speech acts?

Current AI alignment optimizes for hedged, neutral output across contexts. But can models trained this way still perform essential social functions like raising alarms or warnings that require taking strong positions?

Is all human language use fundamentally communicative?

Does human language always involve addressing another person, even in private writing or internal thought? This matters because it challenges how we define language use itself.

Does AI repeat the Enlightenment's reversal into its opposite?

Exploring whether AI's design as a cognitive liberation tool structurally produces epistemic regression rather than progress. The inquiry draws on Adorno and Horkheimer's theory that reason contains seeds of its own mythologization.

Does instrumental AI reproduce pre-Enlightenment knowledge structures?

Does AI's optimization-driven design reintroduce the unverifiability, authority-dependence, and cognitive surrender that characterized pre-modern thought? This connects technical architecture to historical patterns of intellectual regression.

Does AI-generated knowledge have the same structure as hearsay?

This explores whether AI output exhibits the core epistemic features that made hearsay unreliable in pre-Enlightenment knowledge systems. The question matters because it challenges whether existing verification institutions can evaluate AI claims.

Does AI homogenize culture the way mass media did?

If AI generates contextually unique outputs, how can its underlying form be homogeneous? This explores whether AI repeats the culture industry's pattern of suppressing novelty under the guise of variety.

Does advanced technology eventually function like cultural myth?

Explores whether the most sophisticated technical systems—particularly AI—end up operating in culture the way traditional myths do: as unquestionable authorities accepted on faith rather than verified on merit.

Is AI returning knowledge to flow-based economies?

Exploring whether AI's on-demand generation mirrors the flow-based knowledge transmission of oral cultures, and how this differs structurally from both print commodification and gift economies.

Why doesn't AI output carry the spirit of a giver?

Does AI-generated output function like a gift in Mauss's sense, where the giver's spirit obligates the receiver? This explores whether statistical residue can replace the moral weight of personal obligation.

Does AI-generated content mirror oral culture's knowledge patterns?

Walter Ong's framework for oral versus literate cultures may describe how AI content functions on social media. Understanding this parallel could explain why AI discourse feels fundamentally different from print-era knowledge.

Where is the speaker when AI produces speech?

Prior forms of orality—from face-to-face speech to broadcast media—always had an embodied speaker anchoring the utterance. Does AI speech without a speaker represent a fundamentally new media condition, and what happens to our frameworks for evaluating it?

Do transformer models store knowledge or generate it continuously?

Explores whether transformer residual streams function as storage-and-retrieval systems or as real-time flow mechanisms. This distinction challenges fundamental assumptions about how language models actually work.