Evaluation of Model Selection Strategies for Cross-Level Two-Way Differential Item Functioning Analysis
Chalie Patarapichayatham1, Akihito Kamata1,
and Sirichai Kanjanawasee
This study investigated impacts of six model selection strategies on model selection and model parameter estimates.
Cross-level two-way DIF model: a two-way DIF model with two DIF factors, where one DIF factor is an individual characteristic variable and the other DIF factor is a cluster characteristics variable.
Four models were set up.:
ª the full model (Model 1),
ª an incomplete model without the two-way interaction (Model 2),
ª an incomplete model without the cluster-level DIF (Model 3),
ª an incomplete model without both two-way interaction and cluster-level DIF (Model 4).
A total of six model selection strategies were evaluated:
1 Akaike information criterion (AIC),
2 Bayesian information criterion (BIC),
3 Adjusted Bayesian information criterion (ABIC).
The model with the smallest value was selected as the best model.
4 If two out of three information criteria or all three information criteria indicated the same model as the best model, the model was considered as the best model.
5 The fifth model selection strategy evaluated the p-values of the two-way interaction and the cluster-level DIF. If both parameters were statistically significant at the a = .05 level, Model 1 was considered as the best model. If only the two-way interaction was significant, Model 2 was considered as the best model.
6 The sixth model selection strategy used a series of likelihood ratio tests (LRT) for pairs of nested models.
The results show that when the two-way interaction effect, the cluster-level DIF effect, and the cluster size became larger, all model selection strategies tended to select the complete model in all simulation conditions.
The results indicated that BIC is not a recommended criterion by itself for the cross-level two-way DIF detection model.
comments：
If the results of the other 3 models is showing in the appendix, then we can understand the results better.
The other indexes can be used in this study such as bias of the parameter.