Effectiveness of A Training Program on Oral Performance Assessment: The Analysis of Tasks Using the Multifaceted Rasch Analysis

Abstract

The current popularity of second/foreign language oral performance assessment has led to a growing interest in tasks as a tool for assessing language learners' oral abilities. However, most oral assessment studies so far have investigated tasks separately; therefore, any possible relationship among them has remained unexplored. Twenty English as a foreign language (EFL) teachers rated the oral performances produced by 200 EFL learners before and after a rater training program using description, narration, summarizing, role-play, and exposition tasks. The findings demonstrated the usefulness of multifaceted Rasch measurement (MFRM) in detecting rater effects and demonstrating the consistency and variability in rater behavior aiming to evaluate the quality of rating. The outcomes indicated that test difficulty identification is complex, difficult, and at the same time multidimensional. On the other hand test takers' ability is a more determining factor in their score variation than other intervening variables. The outcomes displayed no relationship between task difficulty and raters' interrater reliability measures. The findings suggest that tasks have various effects on oral performance assessment tests and most importantly, performance conditions in estimating the oral ability of test takers. Since various groups of raters have biases to different tasks in use, the findings indicated that training programs can reduce raters' biases and increase their consistency measures. The findings imply that decision makers had better not be concerned about raters' expertise in oral assessment, whereas they should establish better rater training programs for raters to increase assessment reliability.