Search/AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders
● News

AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders

UPDATED Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open web, slurping up sites at a rate that accounts for 80 percent of all AI bot traffic, with the remaining 20 percent used by AI fetchers. Bots and fetchers can hit websites hard, demanding data from a single site in thousands of requests per minute. I can only see one thing causing this to stop: the AI bubble popping According to the report [PDF], Facebook owner Meta's AI division accounts for more than half of those crawlers, while OpenAI accounts for the overwhelming majority of on-demand fetch requests. "AI bots are reshaping how the internet is accessed and experienced, introducing new complexities for digital platforms," Fastly senior security researcher Arun Kumar opined…

Read original article
Source
go.theregister.com
Published
Aug 21, 2025
Mentions
8
language: en
Connections
9 relationships