Psychometric Properties of 3-, 4-, and 5-Option Item Tests: Do Test Takers’ Personality Traits Make a Difference?

Abstract

Prior research has yielded mixed results regarding what contributes psychometrically sound multiple-choice (MC) items. The purpose of the present study was, therefore, twofold: (a) to compare 3-, 4-, and 5-option multiple-choice (MC) tests in terms of psychometric characteristics, and (b) to investigate the relationships between three MC tests and five personality traits. To that end, 150 students were asked to answer three stem equivalent MC item tests. A Big Five Inventory was used to find students' personality traits. Moreover, an attitude questionnaire was utilized to seek students' opinions of these three MC tests. The results of one-way repeated measures ANOVA revealed statistically significant differences for item difficulty, while no statistically significant differences were found for item discrimination and reliability across three MC tests. The results of the Pearson correlation showed no correlation between personality traits and three different versions of MC tests. The results of the attitude questionnaire indicated mixed views towards MC tests. The findings of this study suggest that test developers consider statistical, affective, and contextual factors in order to develop different formats of MC tests.