Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/39866
Title: Optimizing plurality for human intelligence tasks
Authors: Mo, L
Cheng, R
Kao, B
Yang, XS
Ren, C
Lei, S
Cheung, DW
Lo, E 
Keywords: Crowdsourcing
Data quality
Issue Date: 2013
Source: Proceedings of the 22nd ACM International Conference on Conference on information & knowledge management - CIKM '13, San Francisco, California, USA, October 27 - November 01, 2013, p. 1929-1938 How to cite?
Abstract: In a crowdsourcing system, Human Intelligence Tasks (HITs) (e.g., translating sentences, matching photos, tagging videos with keywords) can be conveniently specified. HITs are made available to a large pool of workers, who are paid upon completing the HITs they have selected. Since workers may have different capabilities, some difficult HITs may not be satisfactorily performed by a single worker. If more workers are employed to perform a HIT, the quality of the HIT's answer could be statistically improved. Given a set of HITs and a fixed "budget", we address the important problem of determining the number of workers (or plurality) of each HIT so that the overall answer quality is optimized. We propose a dynamic programming (DP) algorithm for solving the plurality assignment problem (PAP). We identify two interesting properties, namely, monotonicity and diminishing return, which are satisfied by a HIT if the quality of the HIT's answer increases monotonically at a decreasing rate with its plurality. We show for HITs that satisfy the two properties (e.g., multiple-choice-question HITs), the PAP is approximable. We propose an efficient greedy algorithm for such case. We conduct extensive experiments on synthetic and real datasets to evaluate our algorithms. Our experiments show that our greedy algorithm provides close-to-optimal solutions in practice.
URI: http://hdl.handle.net/10397/39866
ISBN: 978-1-4503-2263-8
DOI: 10.1145/2505515.2505755
Appears in Collections:Conference Paper

Access
View full-text via PolyU eLinks SFX Query
Show full item record

SCOPUSTM   
Citations

6
Citations as of Jul 7, 2017

Page view(s)

30
Last Week
2
Last month
Checked on Aug 20, 2017

Google ScholarTM

Check

Altmetric



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.