Adaptive Testing with Multidimensional Pairwise Preference Items: Improving the Efficient of Personality and Other Noncognitive Assessment
The main purpose of the article is to propose a method for administering multidimensional pairwise preference items adaptively, and compared the efficiency to the nonadaptive testing. Four Monte Carlo studies were conducted. In the first and second simulation, the trait score recovery for nonadpative test for multidimensional pairwise preference items were investigated. In the third simulation, the accuracy of trait score were compaired between the nonadaptive and adaptive test in terms of correlations between estimated and known trait scores, absolute bias, and estimated standard errors. The final simulation study investigated the standard errors estimated using a newly developed replication method.
Questions and Comments:
1. In equation 1, it was assumed that the stimulus s and t were evaluated separately and independently. It is truly not realistic.
2. When the dimensions increase, it was found that the standard error from the Bayes modal method increase sharply. Thus, the replication method was developed to try to avoid problems. It is desirable to compare the Bayes modal method and the EAP method with which no iteration is needed. Hence, the inflated standard error could be eliminated. But, the EAP method, like the replication method, will time consuming when the dimensions are high.
3. It is not well stated how to select items during the adaptive testing. Actually, it is crucial since the information provided by a pairwise preference item from different dimensions are rather complicated compared to the information provided by the items from the same dimensions.
4. It would be desirable to compare the efficiency between the nonadaptive testing and adaptive testing by comparing the number of items needed when the same standard error reach.