Do language models generate more novel research ideas than experts?
Explores whether LLMs can break free from expert constraints to generate more novel research concepts. Matters because novelty is often thought to be AI's creative blind spot.
The LLM research ideation study is notable for being the first to achieve statistical significance on LLM vs. human expert idea generation with a proper experimental design. Over 100 NLP researchers wrote novel ideas and provided blind reviews of both LLM-generated and human ideas. The results:
- LLM-generated ideas rated more novel than human expert ideas (p<0.05, robust under multiple hypothesis correction and different statistical tests)
- LLM-generated ideas rated slightly lower on feasibility (trend, not conclusive given sample size)
- Novelty gains correlate with excitement and overall score
The finding is counterintuitive in an important way: we typically assume novelty is the hardest thing for AI — the last creative frontier. But expert researchers are constrained by their existing knowledge, established paradigms, and accumulated priors. LLMs, generating without those constraints, may naturally explore a wider space of conceptual combinations — and expert novelty suffers by comparison.
The feasibility penalty makes sense: novel ideas that violate practical constraints (compute requirements, dataset availability, methodological precedent) are easier to generate than ones that are also realizable. LLMs may be better positioned to generate surprising combinations than to evaluate whether those combinations are tractable.
The study also identifies two key failure modes in LLM research agents: (1) lack of diversity in generation — individual ideas are novel but the set is narrow, and (2) failures of LLM self-evaluation — models cannot accurately assess the quality of their own generated ideas.
Inquiring lines that read this note 63
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI systems achieve real improvement without external human feedback? What human oversight must AI research systems have?- How does the ideation-execution gap differ between AI and human-generated research?
- What makes open-ended scientific paradigm shifts different from specified research tasks?
- What role should human experts play in AI-driven research ideation loops?
- Why do LLMs generate ideas that sound novel but fail during execution?
- Why does LLM research ideation collapse into low diversity despite high novelty?
- How can LLMs evaluate their own creative outputs for utility and novelty?
- Can prompting for specific creative paradigms improve ideation diversity?
- Why do research ideation systems suffer from diversity collapse despite high novelty metrics?
- What happens to idea diversity when AI tools draw from collective knowledge?
- Why do LLM-generated ideas score higher novelty yet lower feasibility than expert ideas?
- Why do LLMs plateau on creativity tasks while humans reach further?
- Why do LLM research ideas lack diversity despite high average novelty?
- What makes a novel research idea practically infeasible for implementation?
- Why do LLMs generate novel ideas but lack evaluative commitment?
- Do LLMs generate more novel ideas than they can evaluate?
- Why do models generate creative ideas but fail to evaluate their legitimacy?
- Why do LLMs generate novel ideas but struggle to evaluate them?
- What makes novelty assessment harder to automate than idea generation?
- Can AI provide creative evaluation or only generative idea production?
- Can LLMs generate more novel research ideas than human experts?
- Can human researchers improve LLM ideas through iterative feedback?
- Do novelty and feasibility always trade off in idea generation?
- Which LLM backends produce the most executable research ideas?
- Do independent LLM outputs converge enough to create artificial hiveminds?
- Why does diversity collapse occur in multi-agent research ideation despite high novelty?
- Why are AI research ideas more novel but harder to evaluate than human ones?
- What distinguishes scientific plausibility from cognitive availability in research ideas?
- How should AI ideation systems decompose and recombine research concepts?
- How does collective idea diversity differ when groups use AI assistance for ideation?
- What would a valid diversity measure for AI-assisted ideation tasks look like?
- Why do AI agents pursue novelty prompts yet produce narrow idea ranges?
- Do different experts disagree on whether AI-generated research ideas have genuine novelty?
- Why do top scientists disagree on whether o3 produces genuinely novel ideas?
- What specific execution barriers do LLM ideas encounter most frequently?
- Why does LLM knowledge fail to influence their actual outputs?
- Can bilevel autoresearch discover new search mechanisms for the inner research loop?
- Why do major AI breakthroughs require human-discovered data and method combinations?
- Do models need world models to reliably steer hypothesis discovery?
- Can LLMs reliably assess the quality of ideas they generate?
- Do LLM adopters actually cite more diverse and younger research?
- What concerns does widespread LLM use raise for scientific independence?
- Do humans and LLMs agree on novelty assessment in research?
- How does prompt design alter what kind of creativity LLMs can express?
- What makes LLMs media rather than tools that deliver intelligence?
- Why do LLMs produce directive responses when experts favor open-ended exploration?
- How much does user framing of questions limit what LLMs can suggest?
- How do expert priors constrain human researchers from exploring novel concepts?
- Can AI systems generate diverse hypotheses or do they collapse toward similar ideas?
- How often do machine learning agents generate truly novel solutions?
- Why does expert density around topics signal near-future discoveries?
- Do research agents mostly reproduce known techniques or discover novel solutions?
- Does AI-augmented research produce greater diversity in research topics or research outputs?
- Can structured evaluation assess novelty in scientific writing?
- Can automated AI systems assess novelty as well as human reviewers?
- Do academic reward structures actively prevent innovation in research communication forms?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do LLMs generate novel ideas from narrow ranges?
LLM research agents produce individually novel ideas but cluster them in homogeneous sets. This explores why high average novelty coexists with poor diversity coverage and what it means for automated ideation.
the diversity failure mode
-
Why do LLMs generate more novel research ideas than experts?
LLM-generated research ideas are statistically more novel than those from 100+ expert researchers, but the mechanisms behind this advantage and its practical implications remain unclear. Understanding this paradox could reshape how we use AI in creative knowledge work.
writing angle for this cluster
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
- The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
- Has the Creativity of Large-Language Models peaked? —an analysis of inter- and intra-LLM variability —
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
- Agent Laboratory: Using LLM Agents as Research Assistants
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Predicting Empirical AI Research Outcomes with Language Models
- The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions
Original note title
llm-generated research ideas are statistically more novel than human expert ideas but less feasible