Diagnosing linguistic problems in Chinese students’ academic writing in English: Specifying the target constructs for assessment design

Defining and specifying target constructs have been a longstanding obstacle for the purposeful development of diagnostic assessment instrument. This paper reports the innovative procedures we adopted to identify and specify the target constructs for developing assessment tools, which can diagnose problems in the English academic writing of undergraduate students. The method harnessed the power of meta-analysis, manual and computer-assisted error analysis and corpus linguistic tools, which, we believe, bear originality and methodological rigor for test development and validation.
To start with, systematic meta-analyses were conducted to survey and analyze the linguistic errors investigated by existing studies in the literature (between 1988 and 2015). This meta-analysis generated a list of errors which either frequently appeared in Chinese students’ academic writing in English or (not of high frequency but) were considered to be serious by target readers. From the list of high-frequency and grave errors, those that could be reliably detected by existing computational technology were excluded. The final list contains 25 linguistic errors at lexical and syntactical levels. A detailed error-tagging manual was developed to guide manual error tagging.
Manual error tagging was conducted by three raters subsequently with 387 essays selected from an in-house written corpus of over 1000 untimed and researched academic essays written by first-year undergraduates in Hong Kong. Rigorous measures were adopted to monitor and control inter-coder reliability and coding validity during the multistage coding process. Finally, error tags were extracted and examined using the software AntConc. Statistics of the error tags were computed for error prevalence and frequency in order to validate the error list regarding their appropriateness for the target student population. The empirically generated rank-ordered error list was also compared with the one generated from the meta-analysis conducted earlier to explore and understand the differences. Target constructs were further specified based qualitative examination of the linguistic properties of the error-tagged sentences and the sub-categories under each error code. This textual analysis prepared rich and authentic linguistic materials for developing test items to diagnose the target linguistic problems in students’ writing.
Besides detailed description and demo of the methods and techniques adopted and their benefits, the issues and difficulties encountered will also be discussed and shared. Towards the end of this presentation, sample diagnostic items developed from this approach will be presented. Potential applications, as well as the further direction of research, will be discussed. Copyright © 2018 40th Language Testing Research Colloquium.