Please use this identifier to cite or link to this item:
Title: Detecting low-quality crowdtesting workers
Authors: Mok, RKP 
Li, W 
Chang, RKC 
Issue Date: 2016
Publisher: Institute of Electrical and Electronics Engineers
Source: 2015 IEEE 23rd International Symposium on Quality of Service, IWQoS 2015, 2016, 7404734, p. 201-206 How to cite?
Abstract: QoE crowdtesting is increasingly popular among researchers to conduct subjective assessments of different services. Experimenters can easily access to a huge pool of human subjects through crowdsourcing platforms. A fundamental problem threatening the integrity of crowdtesting is to detect cheating from the workers who work without any supervision. One of the approaches in classifying the quality of workers is analyzing their behavior during the experiments. A major challenge is to systematically analyze the mouse cursor trajectory. However, existing works usually analyze the trajectory coarsely, which cannot fully extract the information imbedded in the trajectory. In this paper, we propose to use finer-grained cursor trajectory analysis, including submovement analysis, to identify low quality workers. Our approach is to define a set of ten worker behavior metrics to quantify different types of worker behavior. A jQuery-based library was implemented to collect the worker behavior. Moreover, four different 5-point Likert scale rating methods were employed. A number of methods, including question design, instructions, and human inspections, are used to label workers into three categories. We then apply multiclass Naive Bayes classifier to construct different models using all or some of the metrics and the workers' category. Our results show that the error rates of the model trained from four metrics is equal or less than 30% for four rating methods. By combining the predictions from the four rating methods, the successful rate in detecting low-quality workers is around 80%.
ISBN: 9781467371131
DOI: 10.1109/IWQoS.2015.7404734
Appears in Collections:Conference Paper

View full-text via PolyU eLinks SFX Query
Show full item record


Last Week
Last month
Citations as of Feb 21, 2019

Page view(s)

Last Week
Last month
Citations as of Feb 17, 2019

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.