After an HN post, nearly half one site's traffic came from a single AI scraper

Sammy Straus runs sprinklz.io, a customisable feed reader. After he posted it on Hacker News a month ago and it got about a hundred upvotes, the site slowed down, and it got worse days later. His logs showed most requests never ran the page's JavaScript, and nearly half of all traffic came from one service that watches websites for changes and turns them into AI newsletters, hitting his site many times a minute. He blocked it, added a robots.txt that turns away AI crawlers but still allows search engines, added rate limits, and is now making the site faster. On Hacker News (about 340 points and about 180 comments), many commenters reported similar bot load, but there is no consensus on the fix. Some now block whole cloud networks such as AWS, Google Cloud and Azure, or put a challenge page in front, such as Anubis or Cloudflare Turnstile; one said over 80% of his traffic fails his challenge. One commenter advised telling bots apart: GPTBot or ClaudeBot only index, while ChatGPT-User or Claude-User visits can come from real people asking an AI about you. Skeptics said reaching #1 on HN barely moved their cheap server, and blamed his single-page app design; make pages cacheable. Sorted take: check your logs by user agent before blocking everything. Cutting abusive scrapers is fine, but AI fetches triggered by a user may be customers.