Google's AI search summaries already pull the clicks that made letting its crawler index your site worthwhile.

This explores why publishers can't just shut AI out when one crawler, Google's, collects pages for both ordinary search results and AI-generated answers. A plain warning first: nothing in this collection studies crawler policy directly. It doesn't cover robots.txt rules, Google's separate AI opt-out controls, or how much a publisher would lose by blocking the crawler. So the mechanism behind the question isn't documented here. What the collection does document is why the bind matters: the AI layer is already taking the traffic that made letting the crawler in worth it.

The bargain used to be simple. Publishers let Google index their pages, and Google sent readers back as clicks. AI Overviews break that exchange by answering the question on the search page itself. Pew's browsing-panel data shows how big the shift is: when an AI summary appears, people click a result link 8% of the time instead of 15%, and more sessions end with no click at all Do AI summaries on Google reduce clicks to actual websites?. A cleaner natural experiment compared English Wikipedia, where AI Overviews launched, with the German and French editions as controls. It found roughly a 5% drop in search referrals that it attributes to the summary layer answering queries before anyone clicks through Does AI search summaries divert traffic away from Wikipedia?. News executives expect much worse: a 40-43% fall in Google referrals over three years Will AI Overviews reduce search referral traffic to publishers?.

Put those numbers next to the crawler problem and the trap is clear. A publisher who blocks the shared crawler to stop AI reuse also disappears from regular search, which is still their biggest source of readers. A publisher who allows it keeps search visibility but supplies material for the summaries that are eating into their clicks. Both choices cost traffic. The studies above measure only one side of that choice, the traffic lost to AI answers. Nobody here measures the other side.

A different angle in the collection suggests this fight over access may be temporary. One line of argument says the internet created 'stock inflation': a huge but fixed body of documents that search engines could index and rank. Generative AI creates 'flow inflation': content made fresh for each query that never sits anywhere waiting to be indexed Why do search tools fail against AI generated content?. You can't search for something that didn't exist before the question was asked Why can't search tools handle AI-generated content?. If that's right, the publisher-crawler fight is a transition problem. The search setup publishers are trying to protect is becoming less central to how people find information, and the more lasting questions are about provenance and verification, not about who gets to crawl what.

To go deeper on the policy mechanics themselves, such as robots.txt, Google-Extended, or regulatory proposals to separate the two crawling purposes, you'd need sources outside this collection.