Please use this identifier to cite or link to this item:
Title: Detecting low-quality workers in qoe crowdtesting : a worker behavior-based approach
Authors: Mok, RKP 
Chang, RKC 
Li, WC 
Keywords: Crowdsourcing
Cheater detection
QoE Crowdtesting
QoE worker behavior
Issue Date: 2017
Publisher: Institute of Electrical and Electronics Engineers
Source: IEEE transactions on multimedia, 2017, v. 19, no. 3, p. 530-543 How to cite?
Journal: IEEE transactions on multimedia 
Abstract: QoE crowdtesting is increasingly popular among researchers to conduct subjective assessments of network services. Experimenters can easily access a huge pool of human subjects through crowdsourcing platforms. Without any supervision, lowquality workers, however, can threaten the reliability of the assessments. One of the approaches in classifying the quality of workers is to analyze their behavior during the experiments, such as mouse cursor trajectory. However, existing works analyze the trajectory coarsely, which cannot fully extract the imbedded information. In this paper, we propose a novel method to detect low-quality workers in QoE crowdtesting by analyzing the worker behavior. Our approach is to construct a predictive model by using supervised learning algorithms. A quality score is computed by applying existing anti-cheating techniques and human inspections to label the workers. We define a set of ten worker behavior metrics, which quantifies different types of worker behavior, including finer-grained cursor trajectory analysis. A multiclass Na " ive Bayes classifier is applied to train a model to predict the quality ofworkers from the metrics. We have conducted video QoE assessments on Amazon Mechanical Turk and CrowdFlower to collect the worker behavior. Our results show that the error rates of the model trained from four metrics are equal or less than 30%. We further find that combining the predictions from the four different 5-point Likert scale ratingmethods can improve the success rate in detecting lowquality workers to around 80%. Finally, our method is 16.5% and 42.9% better in precision and recall than CrowdMOS.
ISSN: 1520-9210
EISSN: 1941-0077
DOI: 10.1109/TMM.2016.2619901
Appears in Collections:Journal/Magazine Article

View full-text via PolyU eLinks SFX Query
Show full item record


Last Week
Last month
Citations as of Oct 6, 2018


Last Week
Last month
Citations as of Oct 19, 2018

Page view(s)

Last Week
Last month
Citations as of Oct 14, 2018

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.