Sandy's readings and review

Topic 3: CCT and CAT

Topic 3: CCT and CAT

by HUANG Sheng Yun -
Number of replies: 1

Modern Sequential Analysis and Its Applications to Computerized Adaptive Testing

The classical procedure for classifying which category examinees belong to by the sequential probability ratio test is to compute the ratio of likelihood of indifference region. To obtain upper bound and lower bound (A and B) by pre-specified type I error and type II error. Then, the ratio aforementioned is compared with A and B. Thus, the decision can be determined. However, the nominal value alpha only can be reached when the test-lengthen is variable. Therefore, such situation of a test to have no more than N items, which is so called “truncated SPRT (TSPRT)”, would not meet the nominal value we specified previously. To resolve above problem, equation (3), (4) and (6) have to be revised and the values of A, B, and C have to be determined by Monte Carlo simulation. Two scenarios (with/without content balance and exposure control) and three procedures (TSPRT, modTSPRT, and modHP: the first one is traditional procedure, and the other two procedures are modified one) were adopted in a series of simulations. The results illustrated that modified SPRT can sustain the nominal value and modHP has the best performance on average test-length.

Comments, Questions and Future Study

1) The present paper is to apply modified SPRT on computerized adaptive testing and it also involves classification (decision making). In past studies, researchers usually adopted constant cut-point ( θ 0 ) on SPRT, here the author proposed to use estimated theta to replace constant θ 0 . Indeed, it’s a good idea to adopt estimated for item selection and improving inflation of type one error. However, it seems that estimated theta works much better with the ACI procedure. Hence, using such combination we don't worry that the problem of inflation of type error.

Reference

Chang, Y. (2004). Application of sequential probability ration test to computerized criterion-referenced testing. Sequential Analysis, 23, 45-61.

Chang, Y. (2005). Application of sequential interval estimation to adaptive mastery testing. Psychometrika, 70, 685-713.

Wald, A. (1947). Sequential analysis. New York: Wiley. Reprinted by Dover (1973).

In reply to HUANG Sheng Yun

Re: Topic 3: CCT and CAT

by HUANG Sheng Yun -

A Stochastic Method for Balancing Item Exposure Rates in Computerized Classification Tests

Issues of controlling item exposure rate have been investigated in computerized adaptive testing. There are many methods that have been developed and Sympson-Hetter (SH) method is the most popular one among these methods. It did not draw much attention for such issue in computerized classification testing. In the present study, the authors combined the method of stochastically curtailed SPRTs (SC-SPRTs) which was proposed by Finkelman in 2008 and SH method in computerized classification testing for maintaining identically well of percentage of correct classification with less average test length, decreasing the percentage of items which over maximum exposure rate, and increasing the percentage of items which less popular for being administered, comparing to SH only. A series of simulations were conducted with content balancing and three levels of gamma. Results indicated that the method of combining SC-SPRT and SH performed quite well for controlling item exposure issue.


Questions and comments:
1) Generally speaking, the relationship between accuracy (person measure) and exposure rate controlling is trade-off in computerized adaptive testing. However, such statement seems not exist in the present study. For computerized classification testing (CCT), the percentage of correct classification (PCC) can be referred to be an index of accuracy. I think the main reason is that SC-SPRT itself can avoid administering those useless items, thus average of test length (efficiency) can be efficiently decreased. As a result, some indices about exposure controlling can be improved. I think the paper shed more light on the conventional SPRT wasting too many items rather than issue of exposure controlling due to the fact that the method did not control exposure rate to fulfill pre-specific rate of 0.2 for all items.