This paper proposed a multiple imputation approach to characterizing the sources of uncertainty of scoring. Although the sources of uncertainty almost have no impact on the point estimates of ability level, it did bring the error to the measurement. By further employing the idea of confidence envelopes by Thissen and Wainer, the authors developed a more superior multiple imputation approach to take into account the sources of uncertainty by using the full error covariance matrix of item parameter estimates. To illustrate the proposed method and investigate the factors which might influence the magnitude of impact of sources of uncertainty, the authors explained the proposed method using three items mixed test and then an empirical dataset from PISA, and a simulation study which manipulated model complexity, test length and sample size. The relative increase in variance was used to measure the impact of item parameter uncertainty.
Questions and comments:
1.I was confused with the calculation of marginal reliability. The sigma square is the variance of prior ability distribution, how to decide this prior? If it dose come from the estimation of ability parameters, is it already involved the sources of uncertainty?
2.Taking into account the confidence interval of SEM in the context of CAT seems an interesting issue for further research.