A systematic review and meta-analysis of AI-enabled assessment in language learning: Design, implementation, and effectiveness

Background: Language assessment plays a pivotal role in language education, serving as a bridge between students' understanding and educators' instructional approaches. Recently, advancements in Artificial Intelligence (AI) technologies have introduced transformative possibilities for automating and personalising language assessments. 

Objectives: This article aims to explore the design and implementation of AI-enabled assessment tools in language education, filling the research gaps regarding the impact of assessment type, intervention duration, education level, and first language learner/second language learner (L1/L2) on the effectiveness of AI-enabled assessment tools in enhancing students' language learning outcome. 

Methods: This study conducted a systematic review and meta-analysis to examine 25 empirical studies from January 2012 to March 2024 from six databases (including EBSCO, ProQuest, Scopus, Web of Science, ACM Digital Library and CNKI). 

Results: The predominant design in AI-driven assessment tools is the structural AI architecture. These tools are most frequently deployed in classroom settings for upper primary students within a short duration. A subsequent meta-analysis showed a medium overall effect size (Hedges's g = 0.390, p < 0.001) for the application of AI-enabled assessment tools in enhancing students' language learning, underscoring their significant impact on language learning outcomes. This evidence robustly supports the practical utility of these tools in educational contexts. 

Conclusions: The analysis of several moderator variables (i.e., assessment type, intervention duration, educational level and L1/L2 learners) and potential impacts on language learning performance indicates that AI-enabled assessment could be more useful in language education with a proper implementation design. Future research could investigate diverse instructional designs for integrating AI-based assessment tools in language education. Copyright © 2024 John Wiley & Sons Ltd.