Adaptive Testing With Multidimensional Pairwise Preference Items : Improving the Efficiency of Personality and Other Noncognitive Assessments
Stephen Stark, Oleksandr S. Chernyshenko, Fritz Drasgow and Leonard A. White
For noncognitive testing, two practical concerns are testing time and response biases and styles that are commonly observed with rating scales. Computerized adaptive forced choice testing could be a promising approach to assess multiple noncognitive constructs in the shortest time with a heightened resistance to response distortion. The purpose of this article is therefore to propose and test a method for administering forced choice tests adaptively using a pairwise preference format. If both statements appearing in a pair represent the same dimension, then the itemis referred to as unidimensiona; if the statements in a pair represent different dimensions, then the item is said to be multidimensional. The multi-unidimensional pairwise preference (MUPP) item response theory (IRT) model was used as the basis for the proposed computerized adaptive testing algorithm.
Four Monte Carlo studies and a field application were conducted to investigate the viability of CAT with pairwise preference items. The results support the adaptive test
Some questions:
Will the combination affect the GUMM parameter?
In the simulation 3, the adaptive and non-adaptive tests have the same length. If it provides results under different test length, the distinct advantage can be more persuasiveness.
Another thing to consider is that if there are some correlations between the items within the same dimension, will it affect the results?