Template-based AR facial augmentation
The present disclosure describes an online system that applies template-based facial augmentations to images as part of an augmented reality experience. A facial augmentation template is a template for an augmented reality (AR) modification to apply to a user's face. These templates include a structure for how the augmentation should be applied to the face and display parameters that a user can set for how the templates should be rendered (e.g., color, specularity, or opacity). The online system identifies feature points on a 3D model of the user's face that correspond to features on the user's actual face, and localizes the template relative to the 3D model by mapping anchor points of the template to corresponding feature points on the 3D model. The online system renders the facial augmentation template at the localization to generate an augmented image of the user's face.
1 . A computer-implemented method comprising:
accessing an image depicting a face of a user;
generating a 3D face model for the face based on the image, wherein the 3D face model represents a 3D structure of the face;
identifying a set of feature points on the 3D face model, wherein each feature point corresponds to a facial feature of the face of the user;
mapping a set of anchor points of a facial augmentation template to the facial features, wherein the facial augmentation template describes a structure of an augmentation to a depiction of the face of the user;
receiving a set of display parameters for the facial augmentation template from a client device;
rendering an augmented image based on the display parameters and the facial augmentation template; and
causing a client device to display the augmented image.
2 . The method of claim 1 , further comprising:
generating the 3D face model based on sensor data collected by the client device.
3 . The method of claim 1 , wherein generating the 3D face model comprises:
applying a machine-learning model to the image to generate the 3D face model, wherein the machine-learning model is trained to generate 3D face models based on images depicting faces.
4 . The method of claim 3 , wherein identifying the set of feature points comprises:
applying the machine-learning model to the image to generate the set of feature points.
5 . The method of claim 3 , wherein identifying the set of feature points comprises:
applying another machine-learning model to the image and the 3D face model, wherein the other machine-learning model is a machine-learning model that is trained to identify feature points on a 3D face model based on the 3D face model and a corresponding image.
6 . The method of claim 1 , wherein the image is a frame in a video feed captured by the client device.
7 . The method of claim 6 , further comprising:
receiving updated display parameters from the client device;
receiving another image depicting the face of the user, wherein the other image is another frame in the video feed;
rendering another augmented image based on the other image, the facial augmentation template, and the updated display parameters; and
causing the client device to display the other augmentation image.
8 . The method of claim 1 , wherein causing the client device to display the augmented image comprises:
transmitting the augmented image to the client device from an online system.
9 . The method of claim 1 , wherein the 3D face model comprises a mesh of polygons.
10 . The method of claim 1 , wherein the set of display parameters comprises parameters for applying a texture to the facial augmentation template.
11 . A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
accessing an image depicting a face of a user;
generating a 3D face model for the face based on the image, wherein the 3D face model represents a 3D structure of the face;
identifying a set of feature points on the 3D face model, wherein each feature point corresponds to a facial feature of the face of the user;
mapping a set of anchor points of a facial augmentation template to the facial features, wherein the facial augmentation template describes a structure of an augmentation to a depiction of the face of the user;
receiving a set of display parameters for the facial augmentation template from a client device;
rendering an augmented image based on the display parameters and the facial augmentation template; and
causing a client device to display the augmented image.
12 . The computer-readable medium of claim 11 , wherein the operations further comprise:
generating the 3D face model based on sensor data collected by the client device.
13 . The computer-readable medium of claim 11 , wherein generating the 3D face model comprises:
applying a machine-learning model to the image to generate the 3D face model, wherein the machine-learning model is trained to generate 3D face models based on images depicting faces.
14 . The computer-readable medium of claim 13 , wherein identifying the set of feature points comprises:
applying the machine-learning model to the image to generate the set of feature points.
15 . The computer-readable medium of claim 13 , wherein identifying the set of feature points comprises:
applying another machine-learning model to the image and the 3D face model, wherein the other machine-learning model is a machine-learning model that is trained to identify feature points on a 3D face model based on the 3D face model and a corresponding image.
16 . The computer-readable medium of claim 11 , wherein the image is a frame in a video feed captured by the client device.
17 . The computer-readable medium of claim 16 , wherein the operations further comprise:
receiving updated display parameters from the client device;
receiving another image depicting the face of the user, wherein the other image is another frame in the video feed;
rendering another augmented image based on the other image, the facial augmentation template, and the updated display parameters; and
causing the client device to display the other augmentation image.
18 . The computer-readable medium of claim 11 , wherein causing the client device to display the augmented image comprises:
transmitting the augmented image to the client device from an online system.
19 . The computer-readable medium of claim 11 , wherein the 3D face model comprises a mesh of polygons.
20 . The computer-readable medium of claim 11 , wherein the set of display parameters comprises parameters for applying a texture to the facial augmentation template.