Five MCAT item selection procedures were compared in this study, including the Volume, Kullback-Leibler information, Minimize the error variance of the linear combination, Minimum Angle and the newly proposed method, Minimize the error variance of the composite score with the optimized weight. The new method V2 was proved to be promising. No one was proved to be universally best, and which approach should be selected depends on the user’s main concern, like content or item usage. The method V2, although proved to be promising, performed better than other methods only in a number of conditions.
1. From Figure 1, V2 is illustrated to perform the poorest for the domain scores, especially for PC, what is the possible cause for this result? As other methods also showed to perform large BIAS on the PC.
2. The intent for this study is said to find an item selection procedure that yields higher precisions for both the domain and composite abilities and a higher percentage of selected items from the item pool. V2 was indicated to fulfill one of the goal when compare with other methods. When conducting MCAT and deciding the item selection approach, what concern is the most important one?
3. Taking all results together, the five approaches provided similar results in varying conditions. Some methods were said to be “better” than others, but only with “slightly better”. Considering this, is it necessary to make the item selection procedure more complicated?