Leftover chatbot phrases can catch unedited AI-written papers, but they can't spot the polished ones that slip through.

This explores whether you can catch AI-written scholarship by its tell-tale wording, such as chatbot boilerplate left in the text or oddly mangled stock phrases. The corpus suggests surface phrases are a useful net for one kind of paper: work that was pasted in and never edited. They tell you little about the much larger share that was cleaned up before it went out. Phrase-hunting finds real cases, but it can't tell you how many it missed.

The strongest evidence that phrases work comes from Google Scholar. When researchers searched for common ChatGPT phrases, they found that about 62% of the matching papers never disclosed GPT use How much GPT-written scholarship reaches Google Scholar undetected?. Most of those papers sat in non-indexed journals, but some reached mainstream outlets, and copies stayed on mirror sites even after retraction. A second kind of surface signal, 'tortured phrases' (awkward substitutes for standard technical terms), clustered in one Elsevier journal's 2021 volumes Did automated text tools produce suspicious phrases in one journal?. Note how that case was built, though. The phrases alone weren't treated as proof. They were read alongside abruptly shortened review timelines, odd submission patterns and AI detectors that flagged only some of the articles, and even then the conclusion was only 'possible' automated generation. In practice, phrases work best as a lead that sends investigators looking for other evidence, not as a verdict.

Why does unedited text exist in large enough amounts to catch? One study found that writers edited AI-generated paragraphs only 23% of the time, and their edits left the text about 96% similar to the original Do writers actually edit AI-generated text before publishing?. Low effort leaves fingerprints. But the moment someone does revise, the fingerprints fade. ML-literate readers couldn't reliably tell LLM-written abstracts from human ones, and LLM-edited abstracts were actually rated the clearest Can readers tell LLM abstracts from human ones?. Heavily rewritten text converges toward the same style, which makes it harder to attribute, though whether it also fools AI detectors is claimed but hasn't actually been tested Do rewrites that hide authorship also fool AI detectors?.

The less obvious point is that the surface features that matter most may be the ones signaling *scholarliness*, not machine authorship. LLM judges score responses higher when they include fake references or rich formatting Can LLM judges be fooled by fake credentials and formatting?. People prefer answers with more citations even when those citations are irrelevant Do users trust citations more when there are simply more of them?. Deep research agents often fail by inventing evidence to look rigorous when real depth is demanded Why do deep research agents fabricate scholarly content?. One demonstration produced 288 finance papers, complete with invented theories and fabricated citations Can AI generate hundreds of fake academic papers automatically?. And a fully AI-generated paper cleared double-blind workshop review Can AI-generated papers pass peer review undetected?. So the same surface-level reading that catches careless AI papers is what lets polished ones through.

The corpus has a gap here. No note measures how many machine-written papers phrase-based detection actually catches or how often it accuses human work by mistake. What it does support is a sorting rule. Phrases reliably expose the careless cases. They say almost nothing about edited ones. Treat phrase matches as a sign of sloppiness, not as a measure of how much machine-written scholarship is out there.