This article proposed an optimizing procedure for obtaining more accuracy item difficulty estimates in Rasch model when there are random guessing were assumed. First, the data set was analysis using Rasch model simply and obtain the item parameters and person parameters, then eliminated the responses referring a cut point. So there is a subsample which was assuming no random guessing evolved. Second, a tailored analysis was conducted using the subsample and more accurate item difficulty parameters were obtained. Third, in order involve all the original data, an anchored analysis was conducted. This procedure was employed to the ARPM dataset and a simulated dataset based on the former. In order to simulate data to illustrate the proposed method, a generalization version of 3PL was given by introduced a y parameter. The results show that the tailored analysis yield more accurate item difficulty estimates and the Andersen theorem can be used for testing the item estimates which affected by random guessing.
Besides the procedure descripted above, this article clearly presents how to handle random guessing behavior in a view of Rasch measurement. More specifically, this article assumes that when students encounter the hard items (more than 1 Logit to their ability), they will employ random guessing. This assumption seems reasonable enough. But in practices, we still hard to conclude that which person is guessing on which item since we cannot obtain “true” ability estimates of persons. Moreover, in practices, there are other types of guessing behavior which not related to the students ability, such as low motivation, testing time limitation, ability-based guessing, etc. In these situations, the generalization of 3PL might not appropriate for use, as well as the cut-off criteria developed based on it.
Based on this -1 Logit logic, the author proposed cut-off criteria for eliminating the response from the original dataset. In case of this study, the cut-off criterion is artificial when it applies to the real data. If the cut-off criterion is loose, more response of the original dataset would be eliminated and lose the valuable information for parameter estimation.
In addition, by observing the Table 3, the item difficulty estimates of tailored analysis are not so accurate as well as disorder of item difficulties with the simulated item difficulties which were anticipated to have good parameter recovery. This phenomenon might cause by only one replication of simulation was conducted. But on an intuitive level, the effect of generating data using generalized 3PL (with y) but analyzing data using Rasch model cannot be attributed a fault to only one replication of simulation. So a systematic simulation should be conducted.
Last but not least, as the author mentioned, the estimation of person parameter should be investigated in further studies, as well as the fit statistics.