The Influence of Item Calibration Error on Variable-Length Computerized Adaptive Testing
Variable-length computerized adaptive testing (VL-CAT) is tailored for items and precision of person measures. The authors investigated average test length (ATL) and percentage of correct classification (PCC) under several levels of calibration error. Two terminations were considered: CSE and ACI. The CSE rule is that testing would be terminated when the standard of precision for person measure is achieved or pre-specific information is met. The ACI actually is a rule for classifying examinees into different categories, such like pass and fail for two categories. It’s kind of wired to see that these two rules are discussed simultaneously due to conventional concept. A series of simulations were carried out: two models of the 2PLM and the 3PLM, two terminations of the CSE and the ACI, and four levels of calibration error of sample size of infinite, 2500, 1000, and 500. It was found that the severer the calibration error, the smaller the ATL would be. It means that calibration error would affect ATL to be underestimated due to overestimation of person measure precision for CSE solely but ACI.
Questions:
1) It is naturally to expect that the severer the calibration error, the lower the PCC would be. However, it seems that the aforementioned expectation does not exist for every ability level. In other words, the true item parameters could not demonstrate the best performance on PCC comparing to other item banks with calibration error. Is it reasonable?
2) From c of figure 4, the true one has the worst performance on bias. Is it just because ACI is not precision based?