VOLUMETRIC AVATARS FROM A PHONE SCAN
A method for generating a subject avatar using a mobile phone scan is provided. The method includes receiving, from a mobile device, multiple images of a first subject, extracting multiple image features from the images of the first subject based on a set of learnable weights, inferring a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject, animating the three-dimensional model of the first subject based on an immersive reality application running on a headset used by a viewer, and providing, to a display on the headset, an image of the three-dimensional model of the first subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method, are also provided.
1 . A computer-implemented method, comprising:
receiving, from a mobile device, multiple images of a first subject;
extracting multiple image features from the images of the first subject based on a set of learnable weights;
inferring a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject;
animating the three-dimensional model of the first subject based on an immersive reality application running on a headset used by a viewer; and
providing, to a display on the headset, an image of the three-dimensional model of the first subject.
2 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving at least a neutral expression image of the first subject.
3 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving at least an expressive image of the first subject.
4 . The computer-implemented method of claim 1 , wherein receiving multiple images of the first subject comprises receiving a sequence of images collected by scanning the mobile device in a selected direction over the first subject.
5 . The computer-implemented method of claim 1 , wherein inferring a three-dimensional model of the first subject comprises biasing the three-dimensional model of the first subject along a direction selected for collecting the images of the second subject.
6 . The computer-implemented method of claim 1 , wherein to form a three-dimensional model of the first subject comprises masking a gaze direction in the three-dimensional model of the second subject and inserting a gaze direction of the first subject.
7 . The computer-implemented method of claim 1 , wherein the image features comprise an identity feature of the first subject, and to form the three-dimensional model of the first subject comprises replacing an identity feature of the second subject with the identity feature of the second subject.
8 . The computer-implemented method of claim 1 , wherein the image features comprise an expression feature of the first subject, and to form the three-dimensional model of the first subject comprises matching the expression feature of the first subject in a latent expression database.
9 . The computer-implemented method of claim 1 , wherein animating the three-dimensional model of the first subject comprises projecting the image features along a direction between the three-dimensional model of the first subject and a selected observation point for the viewer.
10 . The computer-implemented method of claim 1 , wherein animating the three-dimensional model of the first subject comprises including an illumination source for the three-dimensional model of the first subject based on the existing three-dimensional model of the second subject.
11 . A system, comprising:
a memory storing multiple instructions; and
one or more processors configured to execute the instructions to cause the system to perform operations, comprising:
receive, from a mobile device, multiple images of a first subject;
extract multiple image features from the images of the first subject based on a set of learnable weights;
infer a three-dimensional model of the first subject from the image features and an existing three-dimensional model of a second subject;
animate the three-dimensional model of the first subject based on an immersive application running on a headset used by a viewer; and
provide, to a display on the headset, an image of the three-dimensional model of the first subject.
12 . The system of claim 11 , wherein to receive multiple images of the first subject the one or more processors are configured to receive at least a neutral expression image of the first subject.
13 . The system of claim 11 , wherein to receive multiple images of the first subject the one or more processors are configured to receive at least an expressive image of the first subject.
14 . The system of claim 11 , to receive multiple images of the first subject the one or more processors are configured to receive a sequence of images collected by scanning the mobile device in a selected direction over the first subject.
15 . The system of claim 11 , wherein to infer the three-dimensional model of the first subject the one or more processors are configured to bias the three-dimensional model of the first subject along a direction selected for collecting the images of the second subject.
16 . A computer-implemented method for training a model to provide a view of a subject to an auto stereoscopic display in a virtual reality headset, comprising:
collecting, from a face of multiple subjects, multiple images according to a capture script;
updating an identity encoder and an expression encoder in a three-dimensional face model;
generating, with the three-dimensional face model, a synthetic view of a user along a pre-selected direction corresponding to a view of the user; and
training the three-dimensional face model based on a difference between an image of the user provided by a mobile device, and the synthetic view of the user.
17 . The computer-implemented method of claim 16 , wherein collecting multiple images according to a capture script comprises collecting each of the images with a pre-selected illumination configuration.
18 . The computer-implemented method of claim 16 , wherein collecting multiple images according to a capture script comprises collecting images with different expressions for each subject.
19 . The computer-implemented method of claim 16 , wherein training the three-dimensional face model comprises using a metric for a geometric artifact of the three-dimensional face model based on an image of the user.
20 . The computer-implemented method of claim 16 , wherein training the three-dimensional face model comprises using a metric for an identity artifact of the three-dimensional face model.