Search Engines and Beyond

Use the link above to access the full text article published in the Spring 2012 issue of Issues in Science and Technology Librarianship. [DOI: 10.5062/F4D21VHZ]  The introduction reads as follows: “As many information professionals know, searching the Internet landscape using the Google search engine will find only 20% of the information available on the Internet. Not only are there alternative general search engines beyond Google and Yahoo, there are an abundance of specialized search engines for finding specialized content. Reaching this specialized content requires specific searching skills, knowledge of the structure of the web, and an understanding of how search engines work (Bergman 2001;Cohen 2012; Drake 2008; Fahey 2007; Lederman 2011; Sadeh 2006).  Hidden information treasures can be discovered by using specialized search engines that are able to crawl the remaining 80% of the invisible web. ‘The paradox of the Invisible Web is that it's easy to understand why it exists, but it's very hard to actually define in concrete, specific terms,’ say Chris Sherman and Gary Price in their article, The Invisible Web: Uncovering Information Sources Search Engines Can't See (2003). However, they give a simple definition, ‘The Invisible Web: Text pages, files, or other often high-quality authoritative information available via the World Wide Web that general-purpose search engines cannot, due to technical limitations, or will not, due to deliberate choice, add to their indices of Web pages.’  Federated search engines are the tools used to uncover the invisible web. We decided to discover--and organize--resources valuable to science, technology, and engineering (STE) researchers through the use of such search engines. We further described these tools in the Meta-Search and/or Federated Search Engines section. In addition, we categorized and presented a number of freely available authoritative web resources including various databases from government and educational institutions not only from the United States, but from across the world, as well as open access publications, ranking tools, and many others...”