Do language models generate more novel research ideas than experts?

The LLM research ideation study is notable for being the first to achieve statistical significance on LLM vs. human expert idea generation with a proper experimental design. Over 100 NLP researchers wrote novel ideas and provided blind reviews of both LLM-generated and human ideas. The results:

• LLM-generated ideas rated more novel than human expert ideas (p<0.05, robust under multiple hypothesis correction and different statistical tests)
• LLM-generated ideas rated slightly lower on feasibility (trend, not conclusive given sample size)
• Novelty gains correlate with excitement and overall score

The finding is counterintuitive in an important way: we typically assume novelty is the hardest thing for AI — the last creative frontier. But expert researchers are constrained by their existing knowledge, established paradigms, and accumulated priors. LLMs, generating without those constraints, may naturally explore a wider space of conceptual combinations — and expert novelty suffers by comparison.

The feasibility penalty makes sense: novel ideas that violate practical constraints (compute requirements, dataset availability, methodological precedent) are easier to generate than ones that are also realizable. LLMs may be better positioned to generate surprising combinations than to evaluate whether those combinations are tractable.

The study also identifies two key failure modes in LLM research agents: (1) lack of diversity in generation — individual ideas are novel but the set is narrow, and (2) failures of LLM self-evaluation — models cannot accurately assess the quality of their own generated ideas.

Source: Discourses