Implementing the DIF-free-then-DIF Strategy on DIF assessment

Background: A prerequisite of DIF assessment is a pure scale on which the performances of reference and focal groups can be compared. In practice, all items in a test have to be assessed for DIF, such that a pure scale is not feasible before DIF assessment. To tackle this question, the DIF-free-then-DIF strategy was proposed, where a set of DIF-free items are selected first and then all other items are assessed for DIF using the DIF-free items as anchored. Aims and Keywords: This study aims to implement the strategy on three popular DIF detection methods: the Mantel-Haenszel method, the logistic regression method, and Raju’s area method, and compared their performances with traditional approaches. Methods: Monte Carlo simulations were conducted. The five independent variables were: (a) method (Mantel-Haenszel, logistic regression, and area methods); (b) scale purification procedures: yes and no; (c) difference in mean ability between groups: 0 and 1; (d) percentage of DIF items in the test: 10, 20, 30 and 40 %; and (e) number of anchored items: 4 and 8. A total of 100 replications were made in each condition. The data were generated from the Rasch model and the DIF was uniform. The Type I error rate and power were computed. Results: The traditional methods without scale purification procedures worked fairly well only when the percentage of DIF items was as small as 10 %. When scale purification procedures were implemented, the Type I error rates were well-controlled when the percentage was no greater than 30 %. The selection of a set of DIF-free items was very accurate, but not perfect. The DIF-free-then-DIF strategy was effective only when the percentage of DIF items was as high as 30 or 40 %. Conclusion: Scale purification procedures should be implemented. The new strategy is helpful when tests have a high percentage of DIF items.