Facial expressions from eye-tracking cameras
Systems and methods for synthesizing an image of the face by a head-mounted display (HMD), such as an augmented reality device, are disclosed. The HMD may be able to observe only a portion of the face with an inward-facing imaging system, e.g., the periocular region. The systems and methods described herein can generate a mapping of a conformation of the portion of the face that is not imaged based at least partly on a conformation of the portion of the face that is imaged. The HMD can receive an image of a portion of the face and use the mapping to determine a conformation of the portion of the face that is not observed. The HMD can combine the observed and unobserved portions to synthesize a full face image.
1. A wearable system comprising:
an inward-facing imaging system configured to image a periocular region of a wearer of the wearable system; and
a hardware processor programmed to:
access a periocular image, obtained via the inward-facing imaging system, of the periocular region of the wearer;
determine a fit level of the wearable system on the wearer, wherein the fit level comprises a loose fit or a tight fit;
in response to a determination of a loose fit, generate periocular face parameters that encode a periocular conformation of at least the periocular region of the wearer, the periocular face parameters generated based at least partly on intensity measurements or optical flow of patches in the periocular region; or
in response to a determination of a tight fit, generate the periocular face parameters based at least partly on movement of the wearable system relative to the face of the wearer, the movement responsive to a facial expression of the wearer;
generate, based at least partly on the periocular face parameters, lower face parameters of the lower face of the wearer, wherein the lower face is unobserved by the inward-facing imaging system when worn by the wearer; and
combine the periocular face parameters and the lower face parameters to generate full face parameters associated with a three-dimensional (3D) face model of the wearer.
2. The wearable system of claim 1 , wherein the periocular conformation comprises extensions and compressions of periocular muscles, wrinkles, blemishes, or patches in the periocular region, eyelids, or eyebrows.
3. The wearable system of claim 1 , wherein the inward-facing imaging system comprises an infrared light emitting diode (IR LED) or a photo-reflective sensor configured to detect a light intensity of the periocular region.
4. The wearable system of claim 1 , further comprising an electrooculography (EOG) sensor, wherein the wearable system is configured to determine eye positions from readings of EOG sensor.
5. The wearable system of claim 1 , further comprising an audio sensor, wherein a lower face conformation is checked for consistency against phonemes recorded from the audio sensor.
6. The wearable system of claim 1 , wherein the full face parameters are stored for use by a rendering engine to generate a 3D animation of the face of the wearer.
7. A method comprising:
under control of a head-mounted display system comprising an inward-facing imaging system configured to image a periocular region of a wearer of the head-mounted display system and a hardware processor:
accessing a periocular image, obtained via the inward-facing imaging system, of the periocular region of the wearer;
determining a fit level of the head-mounted display system on the wearer, wherein the fit level comprises a loose fit or a tight fit;
in response to a determination of a loose fit, generating periocular face parameters that encode a periocular conformation of at least the periocular region of the wearer, the periocular face parameters generated based at least partly on intensity measurements or optical flow of patches in the periocular region; or
in response to a determination of a tight fit, generating the periocular face parameters based at least partly on movement of the head-mounted display system relative to the face of the wearer, the movement responsive to a facial expression of the wearer;
generating, based at least partly on the periocular face parameters, lower face parameters of the lower face of the wearer, wherein the lower face is unobserved by the inward-facing imaging system when worn by the wearer; and
combining the periocular face parameters and the lower face parameters to generate full face parameters associated with a three-dimensional (3D) face model of the wearer.
8. The method of claim 7 , wherein a lower face conformation is cross-checked for full face consistency based on an eyebrow position.
9. The method of claim 7 , wherein the fit level of the wearable system on the wearer is determined based on eye positions.
10. The method of claim 7 , wherein calibration data specific to the wearer is obtained.
11. The system of claim 9 , wherein the eye positions are determined from pupil-glint tracking.