Automated scoring of Chinese compositions

Automated scoring of essays (AES) has earned a great deal of research attention in recent years. Proponents of AES (e.g., Deane, 2013; Gierl, Latifi, Lai, Boulais, & De Champlain, 2014; Ramineni & Williamson, 2013; Shermis, 2014) highlighted its efficiency in scoring, reporting, and feedback, as well as its capacity to augment human raters in the assessment of critical thinking and problem solving. Critics of AES (e.g., Condon, 2013; Les Perelman, 2014) on the other hand showed discontent on the assessment being confined to linguistic features at sentence- or word- levels rather than on the substantive issues of the writing contents. Despite mixed research findings on the effectiveness of AES, commercial packages such as e-rater® and IntelliMetric® are available for AES of essays written in the English language and AES has been applied to the scoring of commercial examinations as SAT, TOEFL. In contrast, nothing is yet in the market for AES in Chinese. This presentation aims to explore the possibility of using AES for Chinese essays, with the focus of providing timely formative feedback in support of self-directed learning of L1 students in Key Stage (KS) 2 (Primary 4 - 6) in Hong Kong in their learning of Chinese writing. The research team first built a corpus comprising 306 articles from 24 text books for KS2 students, totaling 433,134 Chinese characters. Next, 395 student essays were scored independently by teachers and two raters (the fourth grade undergraduate students majored in Chinese language). These scores were mapped onto a common Rasch scale using Facets software to obtain Rasch scores. Meanwhile, 62 indices were calculated by our AES system, developed on the basis of research at the National Taichung University of Education, using Latent Semantic Analysis (Liao, Chen, Kuo, & Pai, 2014). A prediction equation was developed using multiple regression on the Rasch scores by the AES indices; 12 indices were kept in the final equation with a R2of .470. Third, a new set of 2,795 essays from KS 2 students, totaling 816,981 Chinese characters was automatically rated by the AES system using the established equation. Based on the results, features were incorporated in the AES system to provide feedback to Chinese L1 learners on their essays. Copyright © 2018 CSEPT.