Search engine selection with hierarchical categorization

Abstract

Search engines are very successful in locating resources on the Internet. However they cover only small and different parts of the vast Internet. While using metasearching to simultaneously engage multiple general search engines can increase the coverage, users encounter difficulties in finding target information in the large set of results returned. In addition, a large amount of valuable resources are only available behind specialty search engines or databases. The problem of covering the Internet has been replaced by the problem of selecting suitable search engines for a given query. This thesis focuses on developing an algorithm to construct a hierarchical category of specialty search engines automatically to assist in solving the coverage problem and the search engine selection problem. We believe that metasearching specialty search engines can effectively discover large amount of hidden Web resources. In particular, we proposed to categorize specialty search engines automatically by sending probe queries to them, fetch and analyze the returned result documents. By determining the relevancies between the returned documents and search categories, specialty search engines can be associated with nodes in a hierarchical search engine category directory for metasearching. By utilizing the category, search engines that have a high possibility of relevant information and resources can be easily selected by a metasearch engine. In this thesis, we present the concepts, designs, implementation details, and validation of the proposed categorization algorithm and a metasearch engine prototype, which seeks to demonstrate such a search engine category can be beneficial in finding essential Web resources.