Please use this identifier to cite or link to this item:
Title: Optimizing plurality for human intelligence tasks
Authors: Mo, L
Cheng, R
Kao, B
Yang, XS
Ren, C
Lei, S
Cheung, DW
Lo, E 
Issue Date: 2013
Source: Proceedings of the 22nd ACM International Conference on Conference on information & knowledge management - CIKM '13, San Francisco, California, USA, October 27 - November 01, 2013, p. 1929-1938
Abstract: In a crowdsourcing system, Human Intelligence Tasks (HITs) (e.g., translating sentences, matching photos, tagging videos with keywords) can be conveniently specified. HITs are made available to a large pool of workers, who are paid upon completing the HITs they have selected. Since workers may have different capabilities, some difficult HITs may not be satisfactorily performed by a single worker. If more workers are employed to perform a HIT, the quality of the HIT's answer could be statistically improved. Given a set of HITs and a fixed "budget", we address the important problem of determining the number of workers (or plurality) of each HIT so that the overall answer quality is optimized. We propose a dynamic programming (DP) algorithm for solving the plurality assignment problem (PAP). We identify two interesting properties, namely, monotonicity and diminishing return, which are satisfied by a HIT if the quality of the HIT's answer increases monotonically at a decreasing rate with its plurality. We show for HITs that satisfy the two properties (e.g., multiple-choice-question HITs), the PAP is approximable. We propose an efficient greedy algorithm for such case. We conduct extensive experiments on synthetic and real datasets to evaluate our algorithms. Our experiments show that our greedy algorithm provides close-to-optimal solutions in practice.
Keywords: Crowdsourcing
Data quality
ISBN: 978-1-4503-2263-8
DOI: 10.1145/2505515.2505755
Appears in Collections:Conference Paper

View full-text via PolyU eLinks SFX Query
Show full item record


Last Week
Last month
Citations as of Sep 12, 2020

Page view(s)

Last Week
Last month
Citations as of Sep 23, 2020

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.