Internet Archive to ignore robots.txt directives / Boing Boing
idilali's bookmarks 2017-04-24
Summary:
“The Internet Archive isn’t a search engine, but has historically obeyed exclusion requests from robots.txt files. But it’s changing it’s mind,”
Link:
https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.htmlFrom feeds:
Data & Society » idilali's bookmarksOpen Access Tracking Project (OATP) » lkfitz's bookmarks