In recent years, as the computer technology progressed, more attention in developing noncognitive assessment. However, it is difficult to apply in computerized adaptive testing (CAT) because the sample size is small in this context and conducting an item pool containing more than 50 items. Thus, adopting the marginal maximum likelihood estimation (MML) will harmful in accuracy and efficiency for parameter estimates. But recent researches suggested that the first problem is less critical to accurate scoring and decision making thank that in the second problem. Thus, the unidimensional pairewise preference (UPP) items and model underlying adaptive item selection and scoring – Zinnes-Griggs (ZG) were used for resolving the second problem. It was using subject matter experts (SMEs) to find the parameter estimate for each statement. Two simulation studies were conducted to evaluate the performance of SMEs and MML in empirical data and simulation study. The results shown that error in SME-based location estimates had little harmful effect on score accuracy or validity, regardless of whether measures were constructed adaptively or nonadaptively.
Comments & Questions:
1. Although, the results shows that SME-based were similar to MML. But they are maybe different in other conditions. I mean that there is an important feature in IRT is that the invariance of item parameters, if the parameter estimates substituted by SME-based it will produce different parameter estimates for different experts due to experts’ properties such as experts’ severity.
2. Such as in question 1, different examinees’ score can be compared in CAT because it uses IRT model, thus, item parameters and trait parameters are in the same scale. How to ensure the question of scale when this approach is used in CAT scenario (different items were administered to different examinees)?
3. For the ZG model, the authors mentioned that the slope is depending on the distance between two statements; the larger the distance the higher discriminating the item. How to find this in Equation 1? And why can make an assumption?