Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/86855
Title: Gender and occupational knowledge : the effects of rater variability on the assessment of ESP oral performance
Authors: Liu, Weiwei
Degree: Ph.D.
Issue Date: 2011
Abstract: The overarching aim of the research is to find out empirically the effects of rater variability on the assessment of ESP oral performance, and provide empirical evidence for the possible interaction between individual rater variables. Specifically, this research intends to explore the effects of rater variability on the assessment of ESP oral performance on eight types of speaking tasks, with a focus on the variability in two rater variables -occupational knowledge and gender. Besides, it intends to examine whether or not there is an interaction between raters' level of occupational knowledge and gender in the assessment of ESP oral performance on each of these types of speaking tasks. It also sets out to investigate whether the impacts of rater variability on the assessment of ESP oral performance are consistent across various types of oral task. The research was motivated by (1) Erdosy's (2004) view that rater variability in performance assessment is inherent as "raters are individual persons with personal characters" (p. 1), (2) Gerrig and Zimbardo's (2009) view that most human behaviours vary with the specific context in which they take place. and (3) the highlighted importance of background knowledge in Douglas' (2000) construct of specific purpose language ability. Based on Bynate's (1987) taxonomy of speaking tasks, the research consists of eight parallel studies, each exploring the issue of rater variability in the assessment of ESP oral performance on one specific type of speaking task. The data for this research were collected from 360 rater participants. Based on the finance or accounting qualifications they held, the 360 participants were categorized into three occupational knowledge groups: good mastery group (raters with an accredited senior qualification), basic understanding group (raters with an accredited junior qualification), and no understanding group (raters with no accredited qualification). Based on raters' gender, the three groups of raters were subsequently divided into six subgroups. Raters assessed 30 Chinese students' ESP oral performance on eight types of speaking tasks. Both holistic and analytical scales (pronunciation, fluency, task fulfillment, grammar and vocabulary) were employed in the research. To determine whether there were statistically significant differences among different groups of raters in the assessment of ESP oral performance, ratings of different rater groups were subsequently analyzed by repeated measure two-way ANOVA with two between-subject factors (raters' gender and level occupational knowledge) and a single within-subject factor at 30 levels (the 30 speech samples). Interviews were also held with some of the participants to provide further information that would help explain and substantiate the quantitative results.
The research showed that both raters' gender and their level of occupational knowledge would affect the assessment of ESP oral performance, depending on the specific testing context. The quantitative analyses of ratings of ESP oral performance revealed that in the assessment of ESP oral performance on each type of oral tasks, significant effects of the two rater variables would be identified only when particular rating scales were used. Meanwhile, the quantitative results also showed that the rating scales that would result in significant effects of the two rater variables on the assessment of ESP oral performance varied with the task type. These findings indicate that the effects of raters' gender and their level of occupational knowledge on the assessment of ESP oral performance are context-specific. The specific testing context can be defined by two factors: the type of oral task that is used to elicit the ESP oral performance and the rating scale that is used to assess the performance. The interaction between raters' gender and level of occupational knowledge in the assessment of ESP oral performance was also found to be context-specific, depending on both the rating scale and the task type. Both the effect size and the effect direction of the interaction were found to vary with the specific testing context. The findings of the present research, based on which an improved model of assessment of ESP oral performance was proposed, may contribute to a better understanding of the impacts of rater variability on the assessment of ESP oral performance. The research may also contribute to the design and improvement of rater training programs and the interpretation and proper use of scores of speaking assessment.
Subjects: English language -- Spoken English -- Examinations.
Language and languages -- Ability testing.
Hong Kong Polytechnic University -- Dissertations
Pages: xxiii, 860 p. : ill. ; 30 cm.
Appears in Collections:Thesis

Show full item record

Page views

39
Last Week
0
Last month
Citations as of Mar 24, 2024

Google ScholarTM

Check


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.