a-Stratified Multistage Computerized Adaptive testing with b Blocking
by Hua-Hua Chang, Jiahe Qian & Zhiliang Ying (2001)
The major advantage of computerized adaptive testing (CAT) is that it can efficiently and accurately obtain the trait estimate. In other words, the CAT uses the shorter test length compared with the paper-and-pencil testing to achieve the similar accurately trait estimate. Another advantage of CAT is that the flexible testing schedule, thus, CAT likes a continuous testing. The flexible testing schedule will cause test security problem exists. That is, examinees can get the test information from the overused (or overexposure) test items prior to the testing. Hence, test security is an important issue in CAT. Many methods were proposed to address test security in CAT. The rationale behind of this paper is that most overexposure problems are caused by over-selecting certain types of items, preventing over-selection and making more efficient use of item banks could be a more direct approach to solving theses problems.
The a-stratified (AS) method (Chang &Ying, 1999) was proposed based on the idea mentioned above and the rationale behind of the AS method is:
1. Trait estimate could be inaccurate in the early testing, it is more appropriate to use low a items. Likewise, high a items can be more efficiently used in the later of testing.
2. Selection based on largely on item information typically leads to the over-exposure of highly a items.
However, the AS method assumes that the examinee’s trait estimate can be matched closely with suitable items at every stage, for example, a and b should be uncorrelated. In fact, a and b parameter estimates often are positively correlated, thus, shortage of lower b items in those strata could cause low b items to be selected more efficiently.
This paper was modified the AS method (Chang &Ying, 1999) and proposed the AS with b blocking (BAS) method. The basic idea is to force each stratum to have a balanced distribution of b values to ensure a good match trait estimates for different examinees. The three major steps different from the AS method are: (1) divide the item bank into M blocks according to b values, (2)partition each of the M blocks into K strata according to their a values, (3) recombine the kth stratum items across M blocks into a single stratum.
The results show that, first, the BAS method provides a simple and effective solution through a two-stage stratification. Second, the BAS method improved item exposure rates and reduced MSE. To apply the BAS method, many other issues need to be addressed: (1) the number of strata to be used, (2) the size of the a values range, (3) the minimum a value that is acceptable for item bank stratification, and (4) a determination of the kinds of item banks that might not work well with the method.
Future study:
The stratification methods (i.e., AS and BAS methods) are proposed to increase the using of underexposre items and decrease the using of the overexposure items. Thus these methods will cause the number of items used increased then resulted in the exposure rates decreased. Hence, using these simple methods in item selection can consider the trait estimate and test security simultaneously. In the future study, we can think about how to apply this idea to cognitive diagnosis CAT to improve the latent class estimates accuracy, or used for track of test security.