208 / 2018-04-10 12:14:07

Implementing continuous rater standardisation on a large English for Academic Purposes assessment at a Sino-foreign university

standardisation,eap,moderation,university,large-scale

全体主题 > Principles and methodology of developing standards for language assessment

Ensuring rating is reliable plays a key role in a university’s quality assurance framework. The moderation of markers is one step in agreeing, assuring and checking standards (Bloxham, 2009) and ensuring that the test takers’ capability is reflected in their test scores. This presentation focuses on the moderation of subjectively scored assessments in a transnational university in China. In English for Academic Purposes exams involving approx. 2000 students and 80 raters, post-standardisation feedback and early moderation ensure raters are marking to standard at the beginning of the rating period. However, initial analyses of student results suggest that this standard may not be maintained throughout the operational assessment period, and there is significant “rater drift” (Van Moere, 2014). This large-scale action research investigates the design and implementation of additional moderation in the form of ongoing rater training with benchmark (or “anchor”) samples throughout the operational assessment period. The new moderation procedures, tailored to the practical constraints and resources of a university environment, will make use of both technological and human intervention to optimise raters’ accuracy and ensure raters are marking consistently throughout the rating process. Using qualitative data in the form of rater comments and quantitative data to compare raters’ scores with those of the benchmark samples, we will evaluate the effectiveness of these additional moderation practices. This presentation will contribute to discussions on the effectiveness of different moderation methods, and add to empirical research on a rarely studied area within the “black box” of higher education assessment (Zlatkin-Troitschanskaia, Shavelson & Kuhn, 2015). Our research will shed light on reducing the many potential causes of grade distortions (Wu and Tan, 2016) and it will have practical implications for similar contexts, especially in the growing number of transnational university campuses in China and elsewhere in Asia.