This study aimed to compare the impact of 3 rounding rules widely used in the Angoff procedure in standard setting. Data in this study, including item parameters and panelist distributions for 20 panelists, were simulated from the 2005 NAEP 12th grade mathematics Angoff standard-setting pilot study. The influence of rounding rules was determined by biased cut-point scores, using criteria such as biases at the individual and group levels as well as the percent above cut-score (PAC). Manipulated conditions included four pools of items and 3 rounding rules, and the total number of conditions was 12.
Results have suggested the possibility to yield more biases when rounding to the nearest whole number with more items and indicated the importance of knowing a panelist’s intended cut-score. However, as the authors stated in the article, it is impractical to know all panelists’ intended cut-scores in advance and to design an ordered item booklet (OIB) to avoid biases. Also, most large-scale testing programs employ multiple cut-scores, which make it more difficult to design multiple OIBs to avoid biases. My primary concern is that the standard setting often takes several rounds, creating a very complicated situation, and so that it takes much effort to controlling nuisance factors when Angoff methods are used in standard setting. This study only provided a beginning to evaluate Angoff methods in standard setting, and further investigations are needed.