Search engines send readers your way; AI answers just... answer, and keep the click for themselves.

This explores how publishers separate 'index me so readers can find me' from 'take my content to train or power an AI', and why they care. The direct answer is that this collection doesn't cover the mechanics. It has no notes on robots.txt rules, crawler identification, licensing deals or blocking tools. What it does have is the economic reason the distinction became urgent, and that may be the more useful thing to understand first.

The old deal was simple. A search crawler took a copy of your page, and in exchange the search engine sent you readers. AI answers break that exchange. A Reuters Institute survey found news executives expect Google search referrals to fall 40 to 43 percent over three years, because AI Overviews answer the question on the results page instead of sending a click Will AI Overviews reduce search referral traffic to publishers?. An eye-tracking study shows why: when an AI Overview appears, attention to the top-ranked link drops from 31% to 9%, and readers trust the AI summary just as much as the ranked results Where do searchers look when AI Overviews appear?. Seen this way, the question isn't really 'search versus training'. It's 'does this request send readers back to me, or replace me?'

That line is blurrier than it sounds. Nielsen Norman Group found people don't swap search for AI chat. They run both side by side, often within the same task Does generative AI chat actually replace traditional search?. So a publisher can't simply block 'AI' and keep 'search', because the same search page now does both jobs. There's also a cost to readers, not just to publishers. In experiments with more than 10,000 people, those who learned a topic through ChatGPT summaries ended up with shallower knowledge and gave sparser advice than people who read web pages through search Does learning from AI summaries produce shallower knowledge than web search?.

One note reframes the whole problem. It argues that the internet created *access* inflation: too much existing content, which search and curation could manage. AI creates *generation* inflation: new content with no fixed collection behind it. That calls for tools like provenance marking (recording where content came from) rather than better filtering Why do search tools fail against AI generated content?. If that's right, deciding which crawler gets in only fixes the input side. The harder question for publishers is whether their work stays traceable once it has been absorbed and reworded in an AI's output.

If you want the technical answer (how crawlers identify themselves, what publishers actually block, which licensing arrangements exist), this collection can't supply it yet. Treat that as a gap, not a settled question.