Google's Got A Secret - Knuckleheads' Club

"Bandwidth costs money, so there’s a limit to how much and how often website operators will let their websites be crawled. This limit means that website operators are picky about who they let crawl their websites. Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that. This isn’t illegal and it isn’t Google’s fault, but this monopoly on web crawling that has naturally emerged prevents any other company from being able to effectively compete with Google in the search engine market.   There Should Be A Public Cache Of The Web All of Google’s competitors in the search engine market have failed in their own way but most of them have complained bitterly about how Google has such an advantage when it comes to web crawling. We think that there is clearly a failure in this market and government intervention is required to break Google’s hold on the natural monopoly of crawling the web...."