Face model capture by a wearable device
Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user.
1. A method of generating a three-dimensional (3D) model of a face of a user, the method comprising:
under control of a wearable device comprising computer hardware, a display device configured to display a 3D environment to the user, an imaging system configured to image a portion of the face of the user, and an inertial measurement unit (IMU) configured to detect movements of the display device:
detecting, by the IMU, a trigger for imaging a face of the user, wherein the trigger comprises determining that an acceleration of the display device passes a threshold acceleration;
capturing, by the imaging system, images of at least a portion of a face of the user;
analyzing the images captured by the imaging system; and
generating the face model based at least partly on the analysis of the images.
2. The method of claim 1 , wherein one or more of the images comprises a portion of a body of the user other than the face.
3. The method of claim 1 , wherein the images comprise first images captured by a first eye camera of the imaging system and second images captured by a second eye camera of the imaging system.
4. The method of claim 3 , wherein analyzing the images comprises converting the first images and the second images to point clouds using a stereo vision algorithm.
5. The method of claim 4 , wherein the stereo vision algorithm comprises at least one of a block-matching algorithm, a semi-global matching algorithm, a semi-global block-matching algorithm, or a neural network algorithm.
6. The method of claim 4 , wherein generating the face model of the face comprises combining the point clouds using an iterative closest point algorithm.
7. The method of claim 4 , wherein analyzing the images comprises identifying keypoints associated with the face of the user in the images, and wherein generating the face model of the face comprises generating the face model with the keypoints using a bundle adjustment algorithm.
8. The method of claim 4 , wherein analyzing the images comprises:
analyzing the first images to generate a first portion of the face model using a visual simultaneous location and mapping algorithm; and
analyzing the second images to generate a second portion of the face model using the visual simultaneous location and mapping algorithm.
9. The method of claim 8 , wherein generating the face model of the face comprises combining the first portion of the face model and the second portion of the face model.
10. The method of claim 1 , wherein the images comprise frames of a video taken by the imaging system.
11. The method of claim 1 further comprising generating a texture map associated with the face model based at least partly on the images.
12. The method of claim 1 , wherein generating the face model comprises:
accessing a pre-existing face model; and
updating the pre-existing face model based at least partly on the analysis of the images.
13. The method of claim 12 , wherein the pre-existing face model comprises at least one of the following: a generic face model or a previously generated face model of the face of the user.
14. The method of claim 1 , wherein generating the face model comprises:
accessing images of the face previously acquired by the wearable device or by another computing device; and
generating the face model based at least partly on the analysis of images captured by the imaging system and the accessed images.
15. The method of claim 1 further comprising:
communicating the face model to another display device; and
displaying, by the other display device, an image associated with the face of the user based at least partly on the face model.
16. A wearable device comprising:
a display device configured to display a 3D environment to a user;
an imaging system configured to image a portion of the face of the user;
an inertial measurement unit (IMU) configured to detect movements of the display device; and
a computer processor configured to execute software instructions to perform operations comprising:
detecting, by the IMU, a trigger for imaging a face of the user, wherein the trigger comprises determining that an acceleration of the display device passes a threshold acceleration;
capturing, by the imaging system, images of at least a portion of a face of the user;
analyzing the images captured by the imaging system; and
generating a face model based at least partly on the analysis of the images.
17. The wearable device of claim 16 , wherein one or more of the images comprises a portion of a body of the user other than the face.
18. The wearable device of claim 16 , wherein the images comprise first images captured by a first eye camera of the imaging system and second images captured by a second eye camera of the imaging system.
19. The wearable device of claim 18 , wherein analyzing the images comprises converting the first images and the second images to point clouds using a stereo vision algorithm.
20. The wearable device of claim 19 , wherein the stereo vision algorithm comprises at least one of a block-matching algorithm, a semi-global matching algorithm, a semi-global block-matching algorithm, or a neural network algorithm.