Open repositories are being profoundly impacted by AI bots and other crawlers: Results of a COAR Survey – COAR

"There are a growing number of AI bots crawling repositories. These bots are sufficiently aggressive that they can cause service disruptions and outages in repositories. As a result, some repositories have started to block machine access to their collections, which is also inadvertently blocking other desired network services such as scholarly aggregators, indexing services, and directories.

To get a better understanding about the impact of bots and crawlers on repositories, a short survey was distributed to COAR members in April 2025. There were 66 responses from different countries around the world (22 from Canada/US, 22 from Europe, 9 from Latin America, 6 from Asia, 4 from Australasia, 2 from Africa, and 1 unknown).

The results of the survey found that over 90% of respondents are encountering AI bots, usually more than once a week, and often leading to service disruptions. Respondents also reported using a variety of measures to minimize or stop AI bots from accessing the repository applying a mix of approaches such as rate-limiting, firewall rules, robots.txt rules and shared white-lists. Below is a brief summary of the results of the survey. In the coming weeks, we will provide a more detailed report on the COAR website...."