Face model capture by a wearable device
Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user.
1. A system for generating a three-dimensional (3D) model of a face of a user, the system comprising:
a head-mounted display (HMD) configured to present virtual content to a user;
an inward-facing imaging system, wherein the inward-facing imaging system is configured to image at least a portion of the face of the user while the user is wearing the HMD;
an inertial measurement unit (IMU) associated with the HMD and configured to detect movements of the HMD; and
a hardware processor programmed to:
detect a trigger to initiate imaging of a face of the user by determining that an acceleration of the HMD provided by the IMU passes a threshold acceleration;
activate, in response to detecting the trigger, the inward-facing imaging system to acquire images;
detect a stopping condition for stopping the imaging based on data acquired from at least one of the IMU or the inward-facing imaging system;
analyze the images acquired by the inward-facing imaging system with a stereo vision algorithm; and
fuse the images to generate a face model of the user's face based at least partly on an output of the stereo vision algorithm.
2. The system of claim 1 , wherein the stopping condition is detected when a distance between the HMD and the head of the user passes a threshold distance.
3. The system of claim 1 , wherein the inward-facing imaging system comprises a first eye camera and a second eye camera, and wherein the first eye camera and the second eye camera have an overlapping field of view.
4. The system of claim 3 , wherein the images comprises a plurality of pairs of images, wherein each pair of images comprises a first image acquired by the first eye camera and a second image acquired by the second eye camera.
5. The system of claim 4 , wherein a pair of images is analyzed together with the stereo vision algorithm.
6. The system of claim 4 , wherein the output of the stereo vision algorithm comprises depth assignments to pixels in the plurality of pairs of images.
7. The system of claim 1 , wherein to analyze the images, the hardware processor is programmed to at least:
identify keypoints in the images using a keypoints detector and descriptor algorithm; or
identify facial features from the images and describe the identified facial features with points in a 3D space.
8. The system of claim 7 , wherein to fuse the images, the hardware processor is programmed to combine the keypoints or facial features using a bundle adjustment algorithm.
9. The system of claim 1 , wherein the stopping condition is detected when the acceleration is less than a threshold acceleration.
10. A method for generating a three-dimensional (3D) model of a face of a user, the method comprising:
detecting a trigger to initiate imaging of a face of the user by determining that an acceleration provided by an inertial measurement unit (IMU) of a head-mounted display (HMD) passes a threshold acceleration;
in response to detecting the trigger, acquiring images of the user's head using an inward-facing imaging system of a wearable device;
identifying a plurality of pairs of images from the acquired images;
analyzing the images by applying a stereo vision algorithm to the plurality of pairs of images; and
fusing outputs obtained from said analyzing step to create a face model.
11. The method of claim 10 , wherein the outputs comprise a depth map associated with the user's face, which contains information relating to distances between the face and the wearable device.
12. The method of claim 10 , wherein the images are acquired as the wearable is being put on or taken off from the user.
13. The method of claim 10 , wherein inward-facing imaging system comprises a first eye camera and a second eye camera, and a pair of images comprises a first image and a second image that are acquired at substantially the same time by the first eye camera and the second eye camera respectively.
14. The method of claim 10 , wherein analyzing the images comprise converting the plurality of pairs of images into point clouds.
15. The method of claim 14 , wherein fusing the outputs comprises combining the point clouds using an iterative closest point algorithm.
16. A system for generating a three-dimensional (3D) model of a face of a user, the system comprising:
a head-mounted display (HMD) configured to present virtual content to a user;
an inward-facing imaging system, wherein the inward-facing imaging system is configured to image at least a portion of the face of the user while the user is wearing the HMD;
an inertial measurement unit (IMU) associated with the HMD and configured to detect movements of the HMD; and
a hardware processor programmed to:
detect a trigger to initiate imaging of a face of the user by determining, based at least on images from the inward-facing imaging system, that a distance between the inward-facing imaging system and the head of the user passes a threshold distance;
activate, in response to detecting the trigger, the inward-facing imaging system to acquire images;
detect a stopping condition for stopping the imaging based on data acquired from at least one of the IMU or the inward-facing imaging system;
analyze the images acquired by the inward-facing imaging system with a stereo vision algorithm; and
fuse the images to generate a face model of the user's face based at least partly on an output of the stereo vision algorithm.