Cloudflare will block AI bots from crawling websites by default for new customers, and broker pay-per-crawl deals between its customers and bot operators.
They can’t stop bots on any of the other sites they regulate either.
Why not? They are doing edge caching, they can literally just block the connection from visiting the site just like they do with their DDoS mitigation.
block which connection? Again, these AI companies know people don’t want them crawling their sites and they do everything they can to be invisible. This has been an issue for years at this point.
Why not? They are doing edge caching, they can literally just block the connection from visiting the site just like they do with their DDoS mitigation.
block which connection? Again, these AI companies know people don’t want them crawling their sites and they do everything they can to be invisible. This has been an issue for years at this point.
blocking DDoS is trivial by comparison.