Sensory Perception & Interaction, Research Group

Computational emotion recognition relies on observable expressions. However, negative situations can evoke regulation mechanisms that obscure and mask emotional experiences, often by smiling. As smiles are typically associated with positive emotions, this mismatch of emotional experience and expression may lead to misinterpretations by most current algorithmic affective computing approaches. To improve computational modeling of real-life experiences and expressions in negative social situations, we explore connections between smile appearance and function, incorporating participants’ rich personal self-reports into ground truth labels for their expressions. We present an empirically grounded smile corpus of 199 smiles that is based on a) recordings of N= 30 participants in negative social situations that are analyzed regarding smile morphology and b) a category system of smile functions based on participants’ self-reports. In a computational model, we used cleaned corpus data of 183 unique smile instances to classify five smile function categories based on observable nonverbal signals, with results benchmarked at above chance. Applying a theory-and datadriven approach, our analyses confirm a complex relationship between internal smile functions and observable signals. Finally, we discuss smile functions in negative social situations, including ‘despising’,‘provoking’, and ‘kiss my ass’-smiles.