Reducing the misclassification costs of cognitive diagnosis computerized adaptive testing: Item selection with minimum expected risk

Chia Ling HSU, Wen Chung WANG

Research output: Contribution to journalArticlespeer-review

Abstract

Cognitive diagnosis computerized adaptive testing (CD-CAT) aims to identify each examinee’s strengths and weaknesses on latent attributes for appropriate classification into an attribute profile. As the cost of a CD-CAT misclassification differs across user needs (e.g., remedial program vs. scholarship eligibilities), item selection can incorporate such costs to improve measurement efficiency. This study proposes such a method, minimum expected risk (MER), based on Bayesian decision theory. According to simulations, using MER to identify examinees with no mastery (MER-U0) or full mastery (MER-U1) showed greater classification accuracy and efficiency than other methods for these attribute profiles, especially for shorter tests or low quality item banks. For other attribute profiles, regardless of item quality or termination criterion, MER methods, modified posterior-weighted Kullback–Leibler information (MPWKL), posterior-weighted CDM discrimination index (PWCDI), and Shannon entropy (SHE) performed similarly and outperformed posterior-weighted attribute-level CDM discrimination index (PWACDI) in classification accuracy and test efficiency, especially on short tests. MER with a zero-one loss function, MER-U0, MER-U1, and PWACDI utilized item banks more effectively than the other methods. Overall, these results show the feasibility of using MER in CD-CAT to increase the accuracy for specific attribute profiles to address different user needs. Copyright © 2022 The Author(s).
Original languageEnglish
JournalApplied Psychological Measurement
Early online dateMar 2022
DOIs
Publication statusE-pub ahead of print - Mar 2022

Citation

Hsu, C.-L., & Wang, W.-C. (2022). Reducing the misclassification costs of cognitive diagnosis computerized adaptive testing: Item selection with minimum expected risk. Applied Psychological Measurement. Advance online publication. doi: 10.1177/01466216211066610

Keywords

  • Cognitive diagnosis
  • Computerized adaptive testing
  • Item selection
  • Bayesian decision theory
  • Minimum expected cost

Fingerprint

Dive into the research topics of 'Reducing the misclassification costs of cognitive diagnosis computerized adaptive testing: Item selection with minimum expected risk'. Together they form a unique fingerprint.