Examining Rounding Rules in Angoff-Type Standard-Setting
Methods
Adam E. Wyse1 and Mark D. Reckase
The failure to examine the statistical functioning of a standard-setting method before it is used operationally means that it is possible that a method may result in statistically biased cut-scores.
The purpose of this study is to examine how well different rules used in Angoff-type standard-setting methods affect cut-scores. The authors compare the impact of panelists rounding judgments to two decimals, the nearest whole number, and the impact of rounding using something similar to bubble sheets where judgments are recorded to the nearest 0.05. They examine how rounding would affect cut-scores when judgments are rounded for individual items or across cluster of items and investigate how well the mean and median work for recovering intended group cut-scores. Three questions were brought out .
Data in this study were simulated based on panelist distributions and item parameters from the 2005 NAEP 12th grade mathematics Angoff standard-setting pilot study.
Question:
1 in the study, they simulated the panelist’s ratings on each item by assuming that the panelist’s y cut-score in the last round of NAEP standard setting was his or her intended cut-score? What about not the last round？
2 Whether the different rules used in Angoff-type standard-setting methods have some effects on the classification of subjects?