Facial Expression Editing in Video
Using a Temporally-Smooth Factorization

CVPR 2012


Fei Yang1   Lubomir Bourdev2    Eli Shechtman3   Jue Wang3   Dimitris Metaxas1

1Rutgers University        2Facebook        3Adobe Systems



Abstract

We address the problem of editing facial expression in video, such as exaggerating, attenuating or replacing the expression with a different one in some parts of the video. To achieve this we develop a tensor-based 3D face geometry reconstruction method, which fits a 3D model for each video frame, with the constraint that all models have the same identity and requiring temporal continuity of pose and expression. With the identity constraint, the differences between the underlying 3D shapes capture only changes in expression and pose. We show that various expression editing tasks in video can be achieved by combining face reordering with face warping, where the warp is induced by projecting differences in 3D face shapes into the image plane. Analogously, we show how the identity can be manipulated while fixing expression and pose. Experimental results show that our method can effectively edit expressions and identity in video in a temporally-coherent way with high fidelity.

Citation

@ARTICLE{Yang_CVPR2012_expresison_editing,
author = {Fei Yang and Lubomir Bourdev and Jue Wang and Eli Shechtman and Dimitri Metaxas},
title = {Facial Expression Editing in Video Using a Temporally-Smooth Factorization},
booktitle = {IEEE CVPR},
year = {2012},
}

Video


Download

Paper   PDF (2.8 M)

Poster   PDF (1.1 M)

Video   MP4 (10 M)

Data and results   Exaggeration (18 M)  ID modification (1.3 M)  Replacement (3.7 M)