Does Field of Study Matter in Academic Performance: Differential Item Functioning Analysis of a High-Stakes Test Using One-Parameter and Two-Parameter Item Response Theory Models

Abstract

To ensure test fairness and validity in high-stakes tests, in the absence of sufficient evidence for identifying various sources of Differential Item Functioning (DIF), the present study examined the effect of field of study on the reading section of the English subtest of Iranian University Entrance Examination (IUEE)for MA in English majors. 1-parameter and 2-parameter logistic Item Response Theory (IRT) models were employed to investigate DIF for a sample of 3588 applicants sitting for the test in 2017. For data analysis, the difR package developed by Magis, Beland, Tuerlinckx, and De Boeck (2010) was utilized. The 1-parameter DIF analysis results indicated that out of the 20 items of the reading section, only three items showed DIF toward the examinees based on their field of study. However, 2-parameter DIF analysis results demonstrated that all items of the reading section presented DIF toward the examinees. Furthermore, the least discrimination toward the non-English group was observed in the findings of the 2-p IRT model. Hence, the 2-p IRT model was found more accurate than the 1-p IRT model as it could identify more DIF items, and the reading comprehension subtest of IUEE was biased toward examinees from different fields of study. Based on the results, to identify and remove various sources of potential DIF existing in the tests and produce test items which are void of any bias in terms of academic background, the use of IRT models is required; although their level of precision varies.