Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/67409
Title: Detecting low-quality workers in qoe crowdtesting : a worker behavior-based approach
Authors: Mok, RKP 
Chang, RKC 
Li, WC 
Keywords: Crowdsourcing
Cheater detection
QoE Crowdtesting
QoE worker behavior
Issue Date: 2017
Publisher: Institute of Electrical and Electronics Engineers
Source: IEEE transactions on multimedia, 2017, v. 19, no. 3, p. 530-543 How to cite?
Journal: IEEE transactions on multimedia 
Abstract: QoE crowdtesting is increasingly popular among researchers to conduct subjective assessments of network services. Experimenters can easily access a huge pool of human subjects through crowdsourcing platforms. Without any supervision, lowquality workers, however, can threaten the reliability of the assessments. One of the approaches in classifying the quality of workers is to analyze their behavior during the experiments, such as mouse cursor trajectory. However, existing works analyze the trajectory coarsely, which cannot fully extract the imbedded information. In this paper, we propose a novel method to detect low-quality workers in QoE crowdtesting by analyzing the worker behavior. Our approach is to construct a predictive model by using supervised learning algorithms. A quality score is computed by applying existing anti-cheating techniques and human inspections to label the workers. We define a set of ten worker behavior metrics, which quantifies different types of worker behavior, including finer-grained cursor trajectory analysis. A multiclass Na " ive Bayes classifier is applied to train a model to predict the quality ofworkers from the metrics. We have conducted video QoE assessments on Amazon Mechanical Turk and CrowdFlower to collect the worker behavior. Our results show that the error rates of the model trained from four metrics are equal or less than 30%. We further find that combining the predictions from the four different 5-point Likert scale ratingmethods can improve the success rate in detecting lowquality workers to around 80%. Finally, our method is 16.5% and 42.9% better in precision and recall than CrowdMOS.
URI: http://hdl.handle.net/10397/67409
ISSN: 1520-9210
EISSN: 1941-0077
DOI: 10.1109/TMM.2016.2619901
Appears in Collections:Journal/Magazine Article

Access
View full-text via PolyU eLinks SFX Query
Show full item record

SCOPUSTM   
Citations

1
Last Week
0
Last month
Citations as of Oct 10, 2017

Page view(s)

48
Last Week
6
Last month
Checked on Oct 15, 2017

Google ScholarTM

Check

Altmetric



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.