Can you trust an AI's book summary if you only spot-check a few random details against the original?

This explores how often AI book summaries make things up, and whether randomly checking a few passages tells you enough about the rest. The short answer is that the corpus has no fabrication rate for book summaries. No study here samples summaries at random and counts the invented details. What it does have is more useful than a number: several cases showing why spot-checking works in some settings and fails in others.

The closest case is economist Brad DeLong's experiment. He had an AI build a detailed summary of a book, checked it against the original text for invented content, and found that the checked version gave him roughly the understanding he would have gotten from reading the book. He could discuss it convincingly afterward Can an AI summary substitute for actually reading a book?. The key detail is that the substitution worked only *after* verification. The checking was what made the summary trustworthy, not something extra added on top. DeLong also had the book in hand and the expertise to notice when something sounded wrong. A typical reader with only the summary has neither.

Other material suggests the checking step is where things usually go wrong. Deloitte delivered a paid government report containing invented court quotes and fake academic references, even though it went through review Can AI quality control catch fabricated citations in professional reports?. Writers using AI assistance edited the generated text only 23% of the time, and even then their edits barely changed it Do writers actually edit AI-generated text before publishing?. Sakana AI's automatically generated paper passed a workshop peer review, and its authors found a citation error only afterward Can AI-generated papers pass peer review undetected?. The pattern across these cases is that fabrications rarely look different from accurate content. Invented details carry the same confident tone as real ones. Fake references can even make text *more* convincing, since AI evaluators reliably score responses higher when they include citations, whether or not those citations are real Can LLM judges be tricked without accessing their internals?.

There's a deeper problem with random spot-checks in particular. AI output changes with every run and every small change to the prompt Why does AI output change with every prompt and context?. So a clean sample from one summary tells you little about the next summary, or even a regenerated version of the same one. Quality checks built for fixed products assume that a good sample says something about the whole batch, and that assumption breaks down here. The lawyers in one study found a related cost. Summaries that don't show where each claim came from forced them to re-check so much that it took longer than doing the work by hand Does GenAI actually save lawyers time on fact verification?. The most promising alternative in the corpus is checking that gathers evidence actively instead of sampling: an agent-based evaluator that went looking for supporting evidence was far more consistent than a standard AI judge. Even so, an error in one of its components spread through the rest of the system Can agents evaluate AI outputs more reliably than language models?.

The takeaway is that "how often do summaries fabricate?" may matter less than "can fabrications be traced back to the source?" A summary that points to the exact passage behind each claim makes spot-checking cheap and meaningful. A smooth summary with no sources makes even a careful random check feel like reassurance it hasn't earned.