Modeling selection effects in examinee-selected items

In an examinee-selected-item (ESI) design, examinees are required to respond to a fixed number of items from a set of given items (e.g., responding to 2 items from 5 given items; leading to 10 selection patterns). The ESI design has the advantages of enhancing students’ learning motivation and reducing their testing anxiety. However, these advantages come at a price: scores from different combinations of items are not directly comparable. This ESI design yields incomplete data, which may be missing not at random so that standard IRT models become inappropriate. Recently, Wang et al. (2012) proposed the examinee-selected-item item response theory (IRT) model by adding an additional latent trait to account for such a selection effect. This latent trait could correlate with the target (intended-to-be-measured) latent trait. Along this research line, we developed a general class of IRT models, which include the examinee-selected-item IRT model as a special case. In the most general case, each selection pattern has one random effect to account for its distinct selection effect. Simulation results indicated a good parameter recovery for the new models. We also conducted an experiment to collect real data, in which 462 fifth graders took five pairs of mathematic (dichotomous) items. In each pair of items, students were first asked to indicate which item they prefer to answer and then answer both items. This is referred to as the “Choose one, Answer all” approach. The new IRT models were fit to the real data and the results were discussed.