Pronunciation assessment: Investigating the role of lexical stress error categories in distinguishing proficiency and comprehensibility levels

Abstract

Lexical stress has been shown to be a good predictor of proficiency levels and raters’ comprehensibility ratings, as higher levels of proficiency and higher ratings of comprehensibility tend to show fewer lexical stress errors. Yet, there is psycholinguistic evidence to suggest that not all lexical stress error categories are equal insofar as nonstandard shifts inducing a change in vowel quality are more impactful than mis-stress with only suprasegmental alterations, and rightward shifts are more damaging than leftward shifts. To date, however, assessment practices and research have generally treated lexical stress as a static and unitary feature. To address this gap, this dissertation study quantitatively investigates the degree of association between IELTS spoken proficiency levels and comprehensibility ratings with the English Word Stress Error Gravity Hierarchy (Ghosh & Levis, 2021), a predictive model of lexical stress error categories and comprehensibility that considers vowel quality and directionality of misplaced lexical stress. It also compares three groups of raters (native English speakers vs. non-native English speakers from a stress-based language vs. non-native English speakers from a tonal language) to examine if the L1 stress phonological system is a source of variance in compressibility ratings. Qualitatively, the study employed raters’ retrospective written reports to assess the saliency of lexical stress when assigning scores and a think-aloud protocol to examine what categories of lexical stress errors raters perceive as problematic. Fifty-seven L2 speakers completed IELTS-style and summary-argument tasks. Four trained raters rated their performances using the IELTS rubric for spoken proficiency, and 15 experienced raters (5 Chinese, 5 Native, and 5 Saudi) rated their comprehensibility and provided written reports regarding the salient features that affected their comprehensibility scoring. Following the rating sessions, three raters participated in a think-aloud protocol. L2 speakers’ performances were annotated using a hybrid approach incorporating Praat acoustic cues of lexical stress and human visual inspection of these cues to identify which syllable received primary stress. Evidence from correlational analyses revealed that categories of errors encompassing vowel changes were more severely linked to deterioration of proficiency and comprehensibility ratings, and while stress categories encompassing no segmental alterations were not associated with proficiency, they correlated with comprehensibility but with a small magnitude. The importance of directionality was found to be only contingent upon the presence of a vowel change. Subsequent univariate ANOVA analyses showed that the measures significantly correlated with proficiency and comprehensibility differentiated High- from Mid- and Low-level speakers but not Mid- and Low-level speakers, a pattern that held for both constructs. Additionally, results showed that raters, to a good extent, uniformly reacted to measures of lexical stress errors. Qualitatively, raters’ written reports indicated that lexical stress was a salient feature to all rater groups, and the think-aloud protocols generally supported quantitative findings. To this end, a modification of the Hierarchy for assessment purposes is presented.