250 / 2018-04-15 21:01:15

Construct Validation of an Integrated Listening-to-Write Task in a High-Stakes English Proficiency Test: A Multi-Dimensional Analysis of Register Variation

construct validation,integrated writing,EAP test,multidimensional approach,register variation

全体主题 > Standards in validating language assessments

Integrated writing task, in which test takers are required to produce a written response based on audio or visual stimuli, is becoming an increasingly popular test format for many local and international English language assessments. Different from traditional writing tasks, integrated tasks call for the use of two or more language modalities for task completion, thus having the potential to tap into multiple language abilities and skills (e.g., Plakans, 2009). The language constructs that an integrated writing task assesses, however, may vary according to different task requirements. This paper attempts to elucidate the construct of a listening-to-write task in a university-based high-stakes English proficiency test in China, by means of multidimensional approach to register variation.

The integrated writing task in focus requires test takers to listen to an academic lecture on a common topic, and then write an essay consisting of both a summarization of the main ideas in the lecture (i.e., the "summary" part) and a critical evaluation of the opinions in the lecture (i.e., the "comment" part). The writing samples in this study were selected from a live test administration in 2015. Stratified sampling method was adopted through which writing samples at different English proficiency levels were selected. The selected writing samples were transcribed into computer and subsequently analyzed by Multidimensional Analysis Tagger (MAT, version 1.3) to investigate the text type of each written response. In total, MAT generated six dimension scores which were used for the identification of text types. Inferential statistical analyses including Chi-square test, paired-samples t-test and multiple linear regression were performed to investigate the differences in terms of text types, dimension scores and the predictive power of dimension scores.

Analysis results indicated that the "summary" and "comment" writings elicited significantly different text types. General narrative exposition was most frequently adopted by students in their summary writings, whereas involved persuasion was prominent in comment writings. With regard to dimension scores, four out of six dimension scores differed significantly between summary and comment writing, suggesting that these writing types tap into different language abilities. Multiple linear regression analysis revealed that Dimension 1 and 4 explained 20.6% of the variance in students' overall writing performance, indicated by their writing scores. While shedding light on the construct validity of the writing component of the test, the findings of this study have important implications for the development of rating scales for integrated writing assessment.