Investigating the Perceived Authenticity and Communicative Abilities of Face-Swapped Portrait Videos

Modern deep learning approaches allow for the automatic creation of highly realistic face-swapped videos. In these videos, recordings of two people are combined in a way that the face of a source person is applied to the video of a target person. This way, the resulting video obtains the facial identity of the source while keeping the body appearance, movements, and facial expressions of the target person. Thanks to their high degree of realism and automation of the generation process, face swaps are a valuable tool for creative and communicative scenarios. However, they could also be abused for criminal activities as they allow the impersonation of others and the generation of manipulated video content.

While many works focus on improving algorithms for the creation and detection of face swaps, there is only limited research on the perception of these modern video manipulations. As humans are very sensitive to changes and imbalances in facial representations, in my thesis I set out to investigate the perception of face swaps. Thereby, I focus on two areas: The perceived authenticity and the communicative abilities of face swaps. To assess the quality and detectable cues in face swap videos, I examine whether humans can detect face swaps and which artifacts and facial areas are most important to detect manipulations using self-reports and eye tracking data. Furthermore, I discuss the perception of the conveyed emotions and personalities of face swaps to evaluate their usefulness as digital avatars in communicative scenarios. In order to perform reliable experiments and evaluations, I additionally introduce a novel dataset of face swaps designed for perceptual experiments as well as an eye tracking framework which enables the automatic generation of areas of interest in portrait videos.

The results of the experiments performed in this thesis indicate that modern face swaps are generally convincing and often mistaken for genuine videos. While participants were able to report visible artifacts, they are usually attributed to video quality and did not suspect face swapping. The eye tracking data, on the other hand, revealed significant differences in viewing behavior between genuine and manipulated videos. This may indicate that some differences are perceived, but only subconsciously. Furthermore, my experiments show that face swaps are able to convey emotions and personality which makes them useful in communicative scenarios such as digital avatars.