Finding and estimating near optimal queries

Abstract

The ultimate objective of IR systems is to obtain optimal retrieval effectiveness. However, the best MAP values of the state-of-the-art IR systems are typically below 35% in the ad hoc automatic retrieval of TREC evaluations. This value is still far below the theoretical optimal retrieval effectiveness of 100%. In this study, we investigate whether it is possible to achieve near optimal retrieval effectiveness using the existing IR systems by formulating effective queries. These effective queries are called near optimal queries because they lead the IR systems to achieve near optimal retrieval effectiveness. Our near optimal queries are defined so as not to include the trivially good effective terms. We propose two strategies, the Idealized Relevance Feedback, and the Combinatorial Optimization Search, to find the near optimal queries under some idealized conditions. We have experimented with a substantial number of query-formulating methods based on the strategies and have evaluated these by using TREC test collections. The best MAP values of our near optimal queries for TREC-6, TREC-7 and TREC-8 test collections are 73%, 76% and 75%, respectively. It appears that a suitable choice of terms and a suitable choice of weights can substantially enhance the retrieval effectiveness of the existing IR systems. Based on the observations of the terms in the near optimal queries, we develop a classifier to estimate a near optimal query. The experimental results show that our classifier can improve the retrieval effectiveness of the user query in existing IR systems.