QSRP : efficient reverse k−ranks query processing on high-dimensional embeddings

Abstract

Embedding models represent users and products as high-dimensional embedding vectors and are widely used for recommendation. In this paper, we study the reverse k-ranks query, which finds the users that are the most interested in a product and has many applications including product promotion, targeted advertising, and market analysis. As reverse k-ranks solutions for low dimensionality (e.g., trees) fail for the high-dimensional embeddings generated by embedding models, we propose the QSRP framework. QSRP precomputes the score table between all user and product embeddings to facilitate pruning and refinement at query time. As the score table is usually large, QSRP samples some of its columns as the index to fit in memory. To tackle the problem that naive uniform sampling results in poor pruning effect, we propose query-aware sampling, which conducts sampling by explicitly maximizing the pruning effect for a set of sample queries. Moreover, we introduce regression-based pruning, which fits cheap linear functions to predict the bounds used for pruning. We also design techniques to build the index with limited memory, reduce index building time, and handle updates. We evaluate QSRP under various configurations and compare with state-of-the-art baselines. The results show that QSRP achieves shorter query time than the baselines in all cases, and the speedup is usually over 100x.