AI scraping bots are knocking university libraries offline

CBC reports academic libraries are being hit by data-scraping bots that slow or crash open archives meant for students and the public. Librarians say whole collections go down under persistent bot traffic; UNC Chapel Hill reported more than 11,000 searches for Finnish music from thousands of IPs in one day, periodically locking out campus users. Project Gutenberg says AI scrapers often ignore robots.txt and download every format at once, even when a full-library download already exists. Major AI firms recently backed a voluntary robots.txt opt-out, but that leaves third-party scrapers uncovered. HN (123 points, 55 comments): No consensus. One camp shares rate-limit and WAF tactics. Another says 11k searches is not much and libraries have weak infrastructure. A third asks who is scraping and maps a residential-proxy value chain. Sorted take: if you publish open content, plan for aggressive scrapers; robots.txt alone will not stop IP-rotated bots.