From hype to insight: Exploring ChatGPT's early footprint in education via altmetrics and bibliometrics

Lung-Hsiang WONG, Hyejin PARK, Chee Kit LOOI

Research output: Contribution to journalArticlespeer-review

Abstract

Background: The emergence of ChatGPT in the education literature represents a transformative phase in educational technology research, marked by a surge in publications driven by initial research interest in new topics and media hype. While these publications highlight ChatGPT's potential in education, concerns arise regarding their quality, methodology, and uniqueness. 

Objective: Our study employs unconventional methods by combining altmetrics and bibliometrics to explore ChatGPT in education comprehensively. 

Methods: Two scholarly databases, Web of Science and Altmetric, were adopted to retrieve publications with citations and those mentioned on social media, respectively. We used a search query, “ChatGPT,” and set the publication date between November 30th, 2022, and August 31st, 2023. Both datasets were within the education-related domains. Through a filtering process, we identified three publication categories: 49 papers with both altmetrics and citations, 60 with altmetrics only, and 66 with citations only. Descriptive statistical analysis was conducted on all three lists of papers, further dividing the entire collection into three distinct periods. All the selected papers underwent detailed coding regarding open access, paper types, subject domains, and learner levels. Furthermore, we analysed the keywords occurring and visualized clusters of the co-occurring keywords. 

Results and Conclusions: An intriguing finding is the significant correlation between media/social media mentions and academic citations in ChatGPT in education papers, underscoring the transformative potential of ChatGPT and the urgency of its incorporation into practice. Our keyword analysis also reveals distinctions between the themes of the papers that received both mentions and citations and those that received only citations but no mentions. Additionally, we noticed a limitation that authors' choice of keywords might be influenced by individual subjective judgements, potentially skewing results in thematic analysis based solely on author-assigned keywords such as keyword co-occurrence analysis. Henceforth, we advocate for developing a standardized keyword taxonomy in the educational technology field and integrating Large Language Models to enhance keyword analysis in altmetric and bibliometric tools. This study reveals that ChatGPT in education literature is evolving from rapid publication to rigorous research. Copyright © 2024 John Wiley & Sons Ltd.

Original languageEnglish
JournalJournal of Computer Assisted Learning
Early online dateFeb 2024
DOIs
Publication statusE-pub ahead of print - Feb 2024

Citation

Wong, L.-H., Park, H., & Looi, C.-K. (2024). From hype to insight: Exploring ChatGPT's early footprint in education via altmetrics and bibliometrics. Journal of Computer Assisted Learning. Advance online publication. https://doi.org/10.1111/jcal.12962

Keywords

  • AI in education
  • Altmetrics
  • Bibliometrics
  • ChatGPT in education
  • Probing publication deluge
  • Research impact

Fingerprint

Dive into the research topics of 'From hype to insight: Exploring ChatGPT's early footprint in education via altmetrics and bibliometrics'. Together they form a unique fingerprint.