Internet Archive to ignore robots.txt directives / Boing Boing

idilali's bookmarks 2017-04-24

Summary:

“The Internet Archive isn’t a search engine, but has historically obeyed exclusion requests from robots.txt files. But it’s changing it’s mind,”

Link:

https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.html

From feeds:

Data & Society » idilali's bookmarks
Open Access Tracking Project (OATP) » lkfitz's bookmarks

Tags:

dsreads bots

Date tagged:

04/24/2017, 12:06

Date published:

04/24/2017, 09:55