A multidimensional and random-effects approach to subjective judgment in multiple rating scales

This study developed a multidimensional random-effects rating scale model (MRE-RSM) to take into account both multidimensionality and subjective judgment in multiple rating scales where each test measures a distinct latent trait with a set of items. A real data set from PISA 2006 was analyzed to demonstrate the advantages of the MRE-RSM over the RE-RSM (Wang, Wilson, & Shih, 2006) and the RSM (Andrich, 1978). The result showed that the MRE-RSM and the RE-RSM had a better fit, indicating subjective judgment in rating scales existed; the MRE-RSM, with the correlation among latent traits, yielded a higher test reliability, suggesting the MRE-RSM was more efficient; and the correlation among latent traits can be directly estimated only through the MRE-RSM.