Applying an assessment use argument to investigate a college-level English language test in Universities in Xi'an

Abstract

This study applied an Assessment Use Argument (AUA) to investigate a college-level English language test in universities in Xi'an, namely, the College English test Band Four (CET-4). The overarching research purpose is to investigate to what extent and in what way the CET-4 can serve as a useful indicator of students' overall English proficiency and an effective measure to promote College English teaching and learning. The related research questions were first linked to the corresponding claims on interpretations of scores, decisions made on scores, and consequences of test uses in an AUA. Then the study narrowed down its research foci: 1) to examine the construct of the CET-4 and its content relevance and coverage, 2) to identify factors underlying the multiple decisions made on CET-4 scores, 3) to reveal stakeholders' perceptions of the CET-4 and its washback, and explore the possible relationships between students' perceptions and their test performances. A distinctive feature of this study is the articulation of an AUA for the CET-4 within China's EFL assessment context. There has been thus far a dearth of research that draws on the structure and rationale of an AUA to either develop a test or justify test uses. Therefore, this study offers an exemplary attempt to examine consequences of the CET-4 while weighing the validity of the revised listening and reading components. The AUA offers an overarching logical structure and a conceptual guidance to investigate all the research questions and sub-questions, and the corresponding claims and warrants. A mixed-method approach was employed to collect backing evidence. About 900 students and 200 teachers participated in this study. A quantitative approach was adopted to analyze the large volume of test data and questionnaire surveys, while a qualitative approach was applied to the analyses of test contents and interview data. Evidence from multiple sources was triangulated to strengthen the logic and coherence of the AUA for the CET-4. Data were analyzed in two phases. In the preliminary study, a statistical comparative study was conducted with 188 test takers' valid scores from the old and the new versions of the CET-4. In the main study, correlations and exploratory factor analysis were performed on a larger pool of 2692 valid score cases from the CET-4. Results from both studies evidence that the current CET-4 possesses better internal structure. In addition, the listening and reading components of seven authentic CET-4 papers underwent content analysis with five parameters including text length, readability, genres, topics, and skills coverage. The results demonstrate an overall nice correspondence between test contents and descriptions on characteristics of input and characteristics of expected response in uniform teaching and testing syllabuses, indicating that the revised listening and reading components have a higher degree of content validity. Questionnaires and interviews, employed to investigate decisions made on CET-4 scores and explore the underlying factors in these decision-making processes, reveal that using test scores as a gatekeeper in selection, advancement, or competition takes a deep root in the inherent influences of China's imperial examination system. Institutional decisions manifest a tendency of using large-scale and high-stakes tests as a catalyst or a lever for curriculum innovation. Both test designers and test users should be held accountable for stakeholders to be affected by their decisions. In addition, interview and survey data explore stakeholders' perceptions of test design, test influences, teaching and learning practices, test preparation activities and so on. Multiple regression analyses reveal that the students' motivations, their perceived difficulty factors, and test-taking strategies have influences on their test performances. To sum up, the study reveals a multiplicity of perspectives on the conceptions, analyses and arguments that bear on assessment validity, use and consequences. The study provides the CET stakeholders, especially test users and test developers, with useful insights to help deepen the understanding of the concept of test validity within the framework of an AUA and also shed light on the process of assessment justification in the Chinese EFL context.