Characterizing Sources of Uncertainty in Item Response
Theory Scale Scores
Ji Seung Yang 1 , Mark Hansen 1 , and Li Cai 1
T raditional scoring approaches (e.g., MAP or EAP) fail to acknowledge th e uncertainty in the item parameter estimates carried over from the calibration process , which will lead to incorrect statement of measurement error especially in the situation where a small calibration sample is used.
Three former approaches can be used to solve the problem:
Analytic approximations: c alculation of a number of non – standard derivative matrices that are model specific
Fully Bayesian sampling based approaches: r ely on Markov chain Monte Carlo (MCMC) to sample the intractable posterior distribution
Previous Multiple imputation (MI) – based approaches: limited to single items or the between-item parameter error covariances were not estimated
The paper proposed the Multiple imputation (MI) – based approaches with
supplemented EM which systematizes a number of seemingly disparate methods under a single framework, including Thissen and Wainer’s confidence envelopes and Lewis’s) expected response functions.
A simple data set consisting of 500 simulated responses to three items (2PL, 3PL, and three-category graded response) was firstly used to illustrate the proposed MI-based procedure. Then the method was used to empirical data.
The results show that the impact of item parameter uncertainty on the full-pattern scores appears to be more variable than for the summed score translations.
Another simulation study was conducted to evaluate the conditions in which the uncertainty carried over from item calibration leads to uncertainty in the scoring process.
Comments:
I do not quite understand the imputation process for evaluating uncertainty.
Other multidimensional model and polytomous mode can be considered
The final analysis demonstrated that parameter uncertainty contributes little to total error variance. May be other indicators can be used instead total error variance.