Perceiving emotions in sign language: Lexical (words) vs. non-lexical (tone) channels

Emotion expression is a crucial part of everyday social interaction. In spoken languages, we express
different emotions through linguistic means, such as emotive lexical items, as well as through nonlinguistic means – intonation, face, head and body movements. Deaf signers also express emotions
linguistically and non-linguistically, yet exclusively in the visual modality, which is accessible to
THE 10th CONFERENCE ON COGNITION RESEARCH (AKKO, 2023)
both deaf and hearing people. Moreover, the same visual display, e.g., an eyebrow raise, can serve
both linguistic and non-linguistic functions – signaling yes/no questions or surprise, respectively,
presenting a challenge for analyzing emotion expressions in the deaf population. Studies
examining the perception of emotional expressions within deaf populations show contradictory
results. On the one hand, sign language knowledge contributes to signers’ more accurate
identification of facial expressions [1]. Yet deaf people experience delays in mastering emotional
understanding, which might stem from delays both in exposure to language and to interpretation
of emotional expression [2-3]. Yet, no study to date has tested whether the important factors of
sign language knowledge and hearing status lead to differences between deaf and hearing
individuals’ production of emotional expressions. In this poster, we present our findings from 45
participants from three groups, performing a language-related task: (1) hearing non-signers, (2)
hearing signers, (3) deaf signers. Their facial expressions produced in four emotion conditions --
sad, happy, angry and neutral -- are analysed using OpenFace’s facial tracking technology and
machine learning techniques. The results of the study show that these three groups can be
classified accurately based on emotional expressions.