Generating a graphical representation of a face of a user wearing a head mounted display
A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various facial sensors, such as cameras, that capture images of portions of the user's face outside of the HMD. For example, multiple facial sensors capture images of a portion of the user's face below the HMD. Through image analysis, points of the portion of the user's face are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face. Additionally, a parametric representation of the user's face is determined for each captured image, resulting in various representations indicating the user's facial expressions. From the parametric representations and transforms mapping the captured images to three dimensions, a rendering model is used and applied to the three dimensional model of the face to render the user's facial expressions.
1. A method comprising:
obtaining images of portions of a user's face enclosed by a head mounted device (HMD) captured by one or more image capture devices;
identifying points corresponding to features of the portions of the user's face within the images of the portions of the user's face;
mapping the identified points to a three-dimensional model of a face;
generating a set of animation parameters for each portion of the user's face from identified points within the images captured by the one or more image capture devices;
determining one or more transformations mapping images captured by the one or more image capture devices into three dimensions;
determining a plurality of differential transformations between images captured by pairs of image capture devices, each differential transformation differently attenuating one or more weights associated with one or more expression parameters corresponding to facial expressions;
generating a rendering model based on the sets of animation parameters, the one or more expression parameters, the determined one or more transformations, and the plurality of differential transformations;
repositioning one or more points of the three-dimensional model of the face by applying the rendering model to the three-dimensional model of the face; and
generating content presenting the three-dimensional model of the face with the repositioned one or more points.
2. The method of claim 1 , wherein the features of the portions of the user's face within the images of the portions of the user's face comprise boundaries of one or more parts of the user's face.
3. The method of claim 1 , wherein identifying points corresponding to features of the portions of the user's face within the images of the portions of the user's face comprises:
applying a model trained from various additional users identifying points corresponding to features of faces of each additional user in previously captured images of the additional users' faces while the additional users make different specific facial expressions to the images of the portions of the user's face.
4. The method of claim 1 , wherein mapping the identified points to the three-dimensional model of a face comprises:
selecting the three-dimensional model of the face from a stored library of three-dimensional models of faces based on characteristics of the user; and
identifying points on the three-dimensional model corresponding to the identified points.
5. The method of claim 4 , wherein selecting the three-dimensional model of the face from the stored library of three-dimensional models of faces based on characteristics of the user comprises:
receiving an image of the user's face; and
selecting a three-dimensional model of a face of the stored library of three-dimensional models of faces having at least a threshold similarity to the image of the user's face.
6. The method of claim 1 , wherein the set of animation parameters comprises a parametric representation of human faces that determines a weight of different human facial expressions in a combination.
7. The method of claim 6 , wherein the parametric representation of human faces comprises a vector of blendshape coefficients that determines a weight of different facial expressions in a linear combination.
8. The method of claim 1 , wherein determining one or more differential transformations between pairs of image capture devices comprises:
determining six differential transformations between pairs of image capture devices.
9. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
obtain images of portions of a user's face enclosed by a head mounted device (HMD) captured by one or more image capture devices coupled to the HMD;
identify points corresponding to features of the portions of the user's face within the images of the portions of the user's face;
map the identified points to a three-dimensional model of a face;
generate a set of animation parameters for each portion of the user's face from identified points within the images captured by the one or more image capture devices;
determine one or more transformations mapping images captured by the one or more image capture devices into three dimensions;
determine a plurality of differential transformations between images captured by pairs of image capture devices, each differential transformation differently attenuating one or more weights associated with one or more expression parameters corresponding to facial expressions;
generate a rendering model based on the sets of animation parameters, the one or more expression parameters, the determined one or more transformations, and the plurality of differential transformations;
reposition one or more points of the three-dimensional model of the face by applying the rendering model to the three-dimensional model of the face; and
generate content presenting the three-dimensional model of the face with the repositioned one or more points.
10. The computer program product of claim 9 , wherein the features of the portions of the user's face within the images of the portions of the user's face comprise boundaries of one or more parts of the user's face.
11. The computer program product of claim 10 , wherein identify points corresponding to features of the portions of the user's face within the images of the portions of the user's face comprises:
apply a model trained from various additional users identifying points corresponding to features of faces of each additional user in previously captured images of the additional users' faces while the additional users make different specific facial expressions to the images of the portions of the user's face.
12. The computer program product of claim 9 , wherein map the identified points to the three-dimensional model of a face comprises:
select the three-dimensional model of the face from a stored library of three-dimensional models of faces based on characteristics of the user; and
identify points on the three-dimensional model corresponding to the identified points.
13. The computer program product of claim 12 , wherein select the three-dimensional model of the face from the stored library of three-dimensional models of faces based on characteristics of the user comprises:
receive an image of the user's face; and
select a three-dimensional model of a face of the stored library of three-dimensional models of faces having at least a threshold similarity to the image of the user's face.
14. The computer program product of claim 9 , wherein the set of animation parameters comprises a parametric representation of human faces that determines a weight of different human facial expressions in a combination.
15. The computer program product of claim 14 , wherein the parametric representation of human faces comprises a vector of blendshape coefficients that determines a weight of different facial expressions in a linear combination.
16. A head mounted display (HMD) comprising:
a front rigid body configured to enclose one or more portions of a face of the user;
a set of image capture devices coupled to the front rigid body, each image capture device configured to capture images of one or more portions of the face of the user enclosed by the front rigid body of the HMD; and
a controller coupled to the set of image capture devices, the controller configured to:
obtain images of the one or more portions of the face of the user enclosed by the front rigid body of the HMD from one or more of the image capture devices;
identify points corresponding to features of the portions of the user's face within the images of the portions of the face of the user enclosed by the front rigid body of the HMD;
map the identified points to a three-dimensional model of a face;
generate a set of animation parameters for each portion of the user's face included in at least one image captured by an image capture device of the set from identified points within the images captured by the set of image capture devices;
determine one or more transformations mapping images captured by the set of image capture devices into three dimensions;
determine a plurality of differential transformations between images captured by pairs of image capture devices, each differential transformation differently attenuating one or more weights associated with one or more expression parameters corresponding to facial expressions;
generate a rendering model based on the sets of animation parameters the one or more expression parameters, the determined one or more transformations, and the plurality of differential transformations;
reposition one or more points of the three-dimensional model of the face by applying the rendering model to the three-dimensional model of the face; and
generate content presenting the three-dimensional model of the face with the repositioned one or more points.
17. The HMD of claim 16 , wherein identify points corresponding to features of the portions of the user's face within the images of the portions of the face of the user enclosed by the front rigid body of the HMD comprises:
apply a model trained from various additional users identifying points corresponding to features of faces of each additional user in previously captured images of the additional users' faces while the additional users make different specific facial expressions to the images of the portions of the user's face.
18. The HMD of claim 16 , wherein map the identified points to the three-dimensional model of a face comprises:
select the three-dimensional model of the face from a stored library of three-dimensional models of faces based on characteristics of the user; and
identify points on the three-dimensional model corresponding to the identified points.
19. The HMD of claim 16 , wherein the set of animation parameters comprises a parametric representation of human faces that determines a weight of different human facial expressions in a combination.