This paper mainly discusses the method to assess random guessing in multiple choice items. Three tests are analyzed: the first, tailored and anchored analyses. The first test was simply an analysis of original responses, which may have been affected by guessing, the tailored analysis involved a subsample of responses in which, based on the first analysis, responses with random guessing were removed, and the anchored analysis was of the original responses but with an identifying constraint （fix the mean of the easiest items）obtained from the tailored analysis. The Andersen’s Theorem was used to compute the variance of the difference of location parameter between two tests. The Advanced Raven’s Progressive Matrices (ARPM) test is used to illustrate the procedure and apply the reference parameters used in the simulation. The software used is RUMM2030. The result shows that if guessing involved in the test, the difficulty parameter will be affected, that is it will be estimated easier. So the difficulty in the first analysis is easier than in the tailored test. The article brings up one formula used to eliminate the guessing.
Questions:
1: (Page 8) The estimate of each item’s difficulty is obtained from more responses in the first analysis than in the tailored analysis. Accordingly, the variance of the estimate in the first analysis will be smaller than in the tailored analysis?
In my opinion, the tailored analysis which means the difficulty of the item is matched to the person, the variance should be smaller.
2 After discussed with Nicky, we think the last sentence in the second paragraph in page 9 is wrong. In our opinion, the item’s difficulty will be greater in the tailored analysis than in the anchored analysis. And in Figure 4, the picture is confusion. We think the name of the axis should be exchanged.
3 The author is very sly. In the discussion, the author avoids the problem of the impact on person parameter. In the Rasch model, the total score is the sufficient estimate. If the pattern of one person is 111001101, will have the same estimate with the pattern 111111000, then how can we sure who is guessing if we just use the total scores?
4 The arrangement of the artcle is not very good.
5 If the data do not fit the Rasch,why do we use another model?