Judging a book by its cover: Investigating the influence of test takers' physical attractiveness in synchronous oral communication assessment

Abstract

The assessment of Oral communication (OC) is an important measure of students' English language proficiency. In OC assessments, raters are commonly exposed to the visual appearance of the test takers, and depending on the test taker's physical attractiveness (PA), they may award students higher or lower scores than their actual ability. However, due to challenges associated with measuring social appearance values, research on this topic has only scratched the surface of this critical issue. Therefore, this study aims to investigate potential impacts of test takers’ physical attractiveness on their oral assessment scores. In phase I, a small exploratory study was conducted to define the PA construct and develop a PA scale. Phase II followed a concurrent embedded research design where quantitative and qualitative measures were applied. Quantitative measures including Many Facet Rasch Measurement and regression were used to investigate several aspects. First, the study investigated whether raters assign different scores to the test takers in two video conferencing modes: when exposed to test takers physical appearance (Face-to-Face) and when not exposed to it (Avatar-Mediated). Second, it explored whether raters show systematic bias in their evaluations based on the presence or absence of the test takers physical appearance. Then, it aimed at finding whether physical attractiveness is a potential explanatory variable that predicts test takers' OC scores in the F2FVC mode. Qualitative methods were conducted to investigate raters' perceptions about their rating process in the two modes (Face-to-Face vs. Avatar-Mediated), and whether they believe that physical attractiveness of the test takers may have affected their rating in the Face-to-Face video conferencing mode. The Many-facet Rasch Measurement analysis showed that 30 test takers received slightly higher scores in the Face-to-Face video conferencing mode compared to the Avatar-Mediated video conferencing mode. Yet, the difference was not significant (X2 = 3.6, d.f. =1, p = 0.06). Although not significant, four out of the 30 test takers failed the exam in Avatar-Mediated video conferencing mode, and would have passed the exam if the score difference between the two modes would have been awarded. Bias analysis found that out of the five expert raters, one showed significant bias towards the F2FVC mode (t (237) = −2.30, p < .05). Simple regression analysis of 41 test takers OC scores and PA scores revealed that test takers' PA significantly predicted their pronunciation scores within the OC construct, with female participants scores being more impacted than male participants. Findings from the qualitative analysis showed that the raters perceived the mode of delivery as having no impact on their rating decisions. They also believed that physical attractiveness of the test takers should not impact their rating, and if it did, it is unintentional. The findings of this research provide evidence of PA being a source of construct irrelevant variance in test takers' scores. Although this research does not end the controversy, it suggests that Patzer’s (1985) statement that “people do judge a book by its cover despite their claim to do the contrary" is true, at least for some people.