Diagnosing and tracking English academic writing: Is cognitive diagnostic modelling the way forward?

To date, only a handful of studies in language testing and assessment utilized Cognitive Diagnostic Modeling (CDM) for diagnostic assessment of L2 academic writing. Even fewer studies tracked the development of L2 academic writing over time. This study adopted CDM to profile the English academic writing performance of first-year university students and tracked the changes of their writing profiles over an academic writing course. This study involved 472 first-year undergraduate students from a university in Hong Kong. Participants were involved in a 13-week English academic writing course, for which they produced three essays for assessment at the beginning (Time 1: individual, timed essay), during (Time 2: pair co-construction, untimed essay), and the end (Time 3: individual and timed essay). At each assessment point, students’ writing was assessed by nine EAP instructors against a fine-grained diagnostic checklist for academic writing. The diagnostic data were analyzed via the reduced RUM to generate for each student a multi-dimensional diagnostic profile on five writing attributes. Salient writing profiles were identified for each assessment point and their changes were tracked over the three observations. The study identified four major performance patterns out of the 25 posterior probability of mastery profiles at each assessment point, and multiple transition trajectories of such profiles from Time 1 to 3 and from Time 2 to 3. Comparing Time 1 and 3, students’ writing performance improved on the two attributes of ‘task fulfillment’ and ‘vocabulary use’, but deteriorated in terms of ‘grammar’ and ‘organization’. Performance on ‘mechanics’ has marginal but insignificant decreases. Besides generating individualized feedback to students, the CDM-based diagnostic assessment has potential to provide useful information for course planning, instruction, and evaluation. Limitations of this study and further directions will be discussed at the end. Copyright © 2017 AILA2017 - World Congress in Rio - Brazil. All Rights Reserved.