Diagnosing academic writing in English: Investigating Q-matrix with Coh-Metrix

The educational value of diagnostic assessment (DA) is widely embraced, yet surprisingly little research has been conducted to understand the construct of DA or to develop instruments for DA. This project represents one initial attempt to fill in this research gap. The study integrated the strengths of Cognitive Diagnostic Modelling and corpus-based computation instruments (e.g. Coh-Metrix) to develop and validate a coherent suite of language assessment instruments to diagnosis undergraduates’ relative strengths and weaknesses in writing academic essays in English. The suite consists of a fine-grained diagnostic checklist, a set of textual indicators, and a pool of discrete-point items. The project firstly developed and validated a diagnostic checklist based on the Empirically-Derived Descriptor checklist of Kim (2011). With this checklist, 372 essays were marked and the item scores were analysed with the reduced RUM model. The underlying Q-matrix of the checklist were verified and refined to generate individual learner profiles on five writing attributes. From these learner profiles, major deficiency patterns among these students were identified. Afterwards, the essays were subjected to automated linguistic measures to identify and compute their linguistic features and to manual error tagging and analysis. Subsequently, key linguistic indicators differentiating major deficiency patterns were identified, based on which, we developed a pool of discrete-point items for indirect diagnosis of the component skills of academic writing. The project can potentially contribute to diagnostic assessment of writing and automated essay evaluation. Copyright © 2017 AILA2017 - World Congress in Rio - Brazil. All Rights Reserved.