A New Stopping Rule for Computerized Adaptive Testing
Seung W. Choi, Matthew W. Grady and Barbara G. Dodd
Educational and Psychological Measurement 2011
A new stopping rule named predicted standard error reduction (PSER) for CAT is introduced, which uses the predictive posterior variance to determine the reduction in standard error. It seeks to balance the dual concerns of measurement precision and testing efficiency by considering the predicted change in measurement precision that would result from the administration of additional items. Under this approach, an adaptive test is terminated when the predicted gain in measurement precision brought on by the administration of an additional item is below a predetermined criterion.
Two existing procedures were used as comparison: the minimum SE stopping rule and the minimum information stopping rule.
The simulation study was used for demonstrating the performance of the new rule.
The results show that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant.
Some concepts:
The two most commonly used methods for determining when a computerized adaptive test is complete are the fixed length and variable length stopping rules
Under a fixed length stopping rule, an adaptive test is terminated when a predetermined number of items have been administered. It is simple, but the precision of examinees will be measured with different degrees which may limit the efficiency.
The variable length stopping rules typically seek to achieve a certain degree of measurement precision for all examinees. There are two types of variable length stopping rules: the standard error (SE) stopping rule and the minimum information stopping rule. The most commonly used SE stopping rule terminates an adaptive test when a predetermined SE has been reached for the most recent examinee trait estimate.
Both fixed length and minimum SE stopping rules stem from the fact that both rules are relatively insensitive to the relationship between examinee trait level and the item pool information function.
Under the minimum information stopping rule, the CAT is terminated when there are no more available items capable of providing a predetermined minimum level of information for an examinee at the most recent trait estimate.
Comments:
The method is based on the discovery of the problem which exists in the current methods, which inspired me most. I am a bit confused by the setting of the hyper parameter and the hypo parameters. If the value 0.03 for hyper parameterand the value 0.01 for hypo parameter can be simulated to be fixed in different situation, then the result can be generalized to more context.The item exposure control procedure should be implemented in the future study especially for the test like achievement test.