Did anyone actually catch the made-up quotes and fake papers before an AI-written report went out the door?
How often do fabricated sources in AI output escape citation checking?
This explores how often made-up references, quotes, and papers in AI-written work get past the people and tools meant to catch them, and what the corpus can say about the rate rather than just the anecdotes.
This explores how often fabricated sources in AI output slip past checking, both human review and automated tools. The short answer: the corpus has no base rate. No study here counts fabricated citations and measures what share got through. What it does have is several cases and a few numbers. Together they suggest that checking often fails, and that the reasons have more to do with how checking is organized than with how good the fakes are.
The clearest case is Deloitte's report for the Australian government. It contained invented court quotes and references to academic papers that don't exist, and it passed the firm's own review before reaching the client Can AI quality control catch fabricated citations in professional reports?. The detail that matters is who found the errors. An outside academic did, after delivery, not the quality-control process the client was paying for Can AI-assisted reports pass quality checks with fabricated citations?. Academic peer review shows a similar pattern. One of Sakana AI's three fully AI-generated papers scored above the acceptance threshold in double-blind workshop review. The citation error in it was found later by the authors themselves, not by the reviewers Can AI-generated papers pass peer review undetected?.
Why does this keep happening? The corpus points to a few compounding reasons. First, people rarely edit what AI hands them. In one study, writers changed AI-drafted paragraphs only 23% of the time, and their edits left the text about 96% the same Do writers actually edit AI-generated text before publishing?. A fake reference in a draft that nobody edits has little chance of being noticed. Second, fabrication is often how agents behave when pushed, not a rare glitch. An analysis of 1,000 deep-research agent failures found that 39% came from inventing examples and evidence to look rigorous when real depth was demanded Why do deep research agents fabricate scholarly content?. Third, it can be produced at scale. One demonstration generated 288 finance papers, each with invented reasoning and fabricated citations Can AI generate hundreds of fake academic papers automatically?.
Here is the twist you might not expect: handing the checking to AI can make things worse. LLM judges give higher scores to answers that include references, whether or not those references are real, because citations look authoritative Can LLM judges be tricked without accessing their internals?. A fake bibliography doesn't just get past an automated reviewer. It can earn extra credit. And some authors are already writing to that reviewer: eighteen arXiv manuscripts hid instructions telling AI reviewers to praise them Are hidden AI prompts in preprints a deceptive research practice?.
The corpus's more promising answers are about design, not vigilance. One approach separates the model's judgment from deterministic, executable checks, so that whether a source exists is verified by code rather than by the model's own opinion Can separating judgment from verification improve research paper reliability?. Another is grounded refusal in retrieval systems: the model declines to answer when it can't point to supporting evidence. That trades some coverage for integrity Can RAG systems refuse to answer without reliable evidence?. Both rest on the same idea. Fabricated sources get past checks that rely on someone noticing, and they get stopped by checks the output must pass before it can exist.
Sources 10 notes
Deloitte refunded A$97,000 after delivering a government assurance report containing fabricated court quotes and fake academic references. The incident reveals that nominal human oversight did not catch AI-generated errors before a paid deliverable reached the client.
Deloitte's $440,000 Australian government report contained fabricated citations, fake court quotes, and nonexistent papers generated by an Azure GPT-4o tool chain. The firm declined to confirm AI caused the errors and only refunded after outside academic detection.
Sakana AI's end-to-end system produced a paper that scored 6.33 in double-blind ICLR 2025 workshop review, meeting acceptance thresholds, but was withdrawn under pre-agreed protocol. Authors later identified a citation error and judged none of three submissions suitable for main-track publication.
Writers edited AI-generated paragraphs only 23% of the time, with edits averaging 96% similarity to the original. This means AI's opinionated and distorted voice propagates with minimal human filtering before publication.
Analysis of 1,000 failure reports reveals 39% of agent failures stem from strategic content fabrication—inventing examples, products, and false evidence—to mimic scholarly rigor when actual research depth is demanded.
Show all 10 sources
A demonstration showed LLMs generating 288 complete finance papers from 96 statistically significant signals, each with invented theoretical justifications and fabricated citations, proving academic HARKing can be automated at scale.
Research shows LLM evaluators systematically score higher when responses include fake references or rich formatting, independent of content quality. These biases are exploitable without model access, undermining AI benchmark credibility.
Eighteen arXiv manuscripts contained concealed instructions directing AI reviewers to give positive assessments. The practice qualifies as questionable research conduct because concealment plus self-serving design violates ethics regardless of stated intent.
Spark-to-Paper architects paper generation as composable skills that isolate model judgment from executable, verifiable operations and require evidence specification before results are observed, reducing dependence on model correctness for consistency.
A multilingual RAG system for noisy historical newspapers succeeds by aggressively expanding retrieval while constraining generation to only grounded answers. The grounded-refusal prompt prevents hallucination when OCR errors and language drift degrade source quality, trading coverage for integrity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Pangram Predicts 21% of ICLR Reviews are AI-Generated
- Stop Automating Peer Review Without Rigorous Evaluation
- AI for Auto-Research: Roadmap & User Guide
- Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals
- The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Deloitte to refund government after using AI in $440k report
- LLM-REVal: Can We Trust LLM Reviewers Yet?