Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/73707
Title: Detecting low-quality crowdtesting workers
Authors: Mok, RKP 
Li, W 
Chang, RKC 
Issue Date: 2016
Publisher: Institute of Electrical and Electronics Engineers
Source: 2015 IEEE 23rd International Symposium on Quality of Service, IWQoS 2015, 2016, 7404734, p. 201-206 How to cite?
Abstract: QoE crowdtesting is increasingly popular among researchers to conduct subjective assessments of different services. Experimenters can easily access to a huge pool of human subjects through crowdsourcing platforms. A fundamental problem threatening the integrity of crowdtesting is to detect cheating from the workers who work without any supervision. One of the approaches in classifying the quality of workers is analyzing their behavior during the experiments. A major challenge is to systematically analyze the mouse cursor trajectory. However, existing works usually analyze the trajectory coarsely, which cannot fully extract the information imbedded in the trajectory. In this paper, we propose to use finer-grained cursor trajectory analysis, including submovement analysis, to identify low quality workers. Our approach is to define a set of ten worker behavior metrics to quantify different types of worker behavior. A jQuery-based library was implemented to collect the worker behavior. Moreover, four different 5-point Likert scale rating methods were employed. A number of methods, including question design, instructions, and human inspections, are used to label workers into three categories. We then apply multiclass Naive Bayes classifier to construct different models using all or some of the metrics and the workers' category. Our results show that the error rates of the model trained from four metrics is equal or less than 30% for four rating methods. By combining the predictions from the four rating methods, the successful rate in detecting low-quality workers is around 80%.
URI: http://hdl.handle.net/10397/73707
ISBN: 9781467371131
DOI: 10.1109/IWQoS.2015.7404734
Appears in Collections:Conference Paper

Access
View full-text via PolyU eLinks SFX Query
Show full item record

SCOPUSTM   
Citations

3
Last Week
0
Last month
Citations as of May 15, 2018

Page view(s)

2
Citations as of May 20, 2018

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.