Please use this identifier to cite or link to this item:
Title: Beyond bag-of-words : bigram-enhanced context-dependent term weights
Authors: Dang, EK
Luk, RW 
Allan, J
Issue Date: 2014
Source: Journal of the Association for Information Science and Technology, 2014, v. 65, no. 6, p. 1134-1148
Abstract: While term independence is a widely held assumption in most of the established information retrieval approaches, it is clearly not true and various works in the past have investigated a relaxation of the assumption. One approach is to use n-grams in document representation instead of unigrams. However, the majority of early works on n-grams obtained only modest performance improvement. On the other hand, the use of information based on supporting terms or “contexts” of queries has been found to be promising. In particular, recent studies showed that using new context-dependent term weights improved the performance of relevance feedback (RF) retrieval compared with using traditional bag-of-words BM25 term weights. Calculation of the new term weights requires an estimation of the local probability of relevance of each query term occurrence. In previous studies, the estimation of this probability was based on unigrams that occur in the neighborhood of a query term. We explore an integration of the n-gram and context approaches by computing context-dependent term weights based on a mixture of unigrams and bigrams. Extensive experiments are performed using the title queries of the Text Retrieval Conference (TREC)-6, TREC-7, TREC-8, and TREC-2005 collections, for RF with relevance judgment of either the top 10 or top 20 documents of an initial retrieval. We identify some crucial elements needed in the use of bigrams in our methods, such as proper inverse document frequency (IDF) weighting of the bigrams and noise reduction by pruning bigrams with large document frequency values. We show that enhancing context-dependent term weights with bigrams is effective in further improving retrieval performance.
Keywords: Information retrieval
Publisher: John Wiley & Sons
Journal: Journal of the Association for Information Science and Technology 
ISSN: 2330-1635
EISSN: 2330-1643
DOI: 10.1002/asi.23024
Appears in Collections:Journal/Magazine Article

View full-text via PolyU eLinks SFX Query
Show full item record


Last Week
Last month
Citations as of Feb 15, 2020


Last Week
Last month
Citations as of Feb 13, 2020

Page view(s)

Last Week
Last month
Citations as of Feb 17, 2020

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.