Abstract
OpenAI recently launched sophisticated Artificial Intelligence (AI) chatbots, namely ChatGPT and GPT-4, that can process user input and generate human-like responses. As AI technology continues to play an increasingly vital role in education, this study aimed to explore its potential application in facilitating the analysis of classroom discourse. Specifically, we evaluated the accuracy of ChatGPT and GPT-4 hosted on campus in coding classroom discourse data. The setting was a fully online, secondary school mathematics lesson in a Chinese classroom. The performance of the AI coders was compared with that of human coders. The results indicated substantial discrepancies between the human coders and ChatGPT (Cohen’s Kappa = 0.18), the human coders and GPT-4 (Cohen’s Kappa = 0.37), and ChatGPT and GPT-4 (Cohen’s Kappa = 0.21). Despite GPT-4’s superior performance over ChatGPT, the degree of agreement between each pair of coders could not be considered satisfactory. An in-depth analysis of their coding results revealed that ChatGPT and GPT-4 had limitations in fully comprehending the context, discerning the underlying meanings of text, and understanding certain context-specific issues. Nevertheless, their capacity to construct new and practical codes demonstrated their potential as valuable tools for assisting researchers in classroom discourse analysis. Copyright © 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
| Original language | English |
|---|---|
| Title of host publication | Recent trends of AI technologies and virtual reality: Proceedings of 8th International Conference on Artificial Intelligence and Virtual Reality (AIVR 2024) |
| Editors | Kazumi NAKAMATSU, Roumiana KOUNTCHEVA, Srikanta PATNAIK |
| Place of Publication | Singapore |
| Publisher | Springer |
| Pages | 335-348 |
| ISBN (Electronic) | 9789819611546 |
| ISBN (Print) | 9789819611539 |
| DOIs | |
| Publication status | Published - 2025 |