Head mounted system to collect facial expressions
A head mounted system (HMS) configured to collect facial expressions of the user wearing the HMS. The HMS includes a frame and at least four cameras coupled to the frame. First and second cameras capture the user's right and left eyebrows, and third and fourth cameras capture the right and left sides of the user's upper lip. An optional computer utilizes the images captured by the cameras to detect facial expressions, microexpressions, and/or to improve the user's emotional awareness.
1. A head mounted system (HMS) configured to collect facial expressions of a user wearing the HMS, comprising:
a frame configured to be worn on the user's head;
first and second cameras coupled to the frame, at locations to the right and to the left of the symmetry axis that divides the user's face to the right and left sides, respectively, which are less than 15 cm away from the user's right and left pupils, respectively; the first and second cameras are oriented such that at least portions of the user's right and left eyebrows are in fields of view (FOVs) of the first and second cameras, respectively, and the user's left and right oral commissures are not in the FOVs of the first and second cameras, respectively; and
third and fourth cameras coupled to the frame, at locations to the right and to the left of the symmetry axis, respectively, and less than 15 cm away from the user's upper lip; the third and fourth cameras are oriented such that at least portions of the right and left sides of the user's upper lip are in the FOVs of the third and fourth cameras, respectively, and the user's left and right eyebrows are not in the FOVs of the third and fourth cameras, respectively.
2. The HMS of claim 1 , wherein the facial expressions are microexpressions, and the third camera is configured to have at least a portion of the user's right cheek in its FOV, and that portion of the user's right cheek enables a microexpression analyzer to identify a raised right cheek.
3. The HMS of claim 2 , wherein the fourth camera is configured to have at least a portion of the user's left cheek in its FOV, and that portion of the user's left cheek enables a microexpression analyzer to identify a raised left cheek.
4. The HMS of claim 1 , wherein at least one of the cameras is configured to have at least a portion of the user's chin cheek in its FOV, and that portion of the user's chin enables a microexpression analyzer to identify a raised chin.
5. The HMS of claim 1 , wherein the facial expressions are microexpressions, and further comprising a processor configured to utilize a machine learning trained classifier to identify a microexpression expressed by the user.
6. The HMS of claim 5 , wherein the processor is further configured to extract vision-related features from data derived from images captured by the first and second cameras, the machine learning trained classifier is trained to identify a microexpression that relates to the upper part of the user's face from the vision-related features, and the vision-related features comprise at least one of temporal features and features derived from locations of facial landmarks identified in the images captured by the first and second cameras.
7. The HMS of claim 5 , wherein the processor is further configured to extract vision-related features from data derived from images captured by the third and fourth cameras, the machine learning trained classifier is trained to identify a microexpression that relates to a lower part of the user's face from the vision-related features, and the vision-related features comprise at least one of temporal features and features derived from locations of facial landmarks identified in the images captured by the third and fourth cameras.
8. The HMS of claim 1 , further comprising a fifth camera coupled to the frame at a location that is less than 10 cm away from the user's right pupil; the fifth camera is oriented such that a lower orbital part of the user's orbicularis oculi muscle that surrounds the user's right eye is in the FOV of the fifth camera, and the user's left oral commissure is not in the FOV of the fifth camera.
9. The HMS of claim 1 , wherein at least one of the first to fourth cameras is selected from at least one of the following types of cameras: (i) a depth camera configured to take measurements indicative of distances of objects relative to the depth camera, (ii) an extended depth of field camera that can capture in focus objects that are 2 to 5 cm from the first camera, (iii) a light field camera, and (iv) a camera that utilizes at least one of the following techniques to achieve an extended depth of field: wavefront coding, diffusion coding, coded aperture, multiple apertures, and lens array.
10. The HMS of claim 1 , further comprising a structured light pattern projector configured to project structured light; wherein the first camera is configured to capture a distorted pattern of reflected structured light, the structured light pattern projector transmits in wavelength longer than 700 nm, and further comprising a processor configured to calculate at least one of depth and movement from the captured distorted pattern in order to identify the facial expressions.
11. The HMS of claim 1 , further comprising an eye tracker and a processor; the eye tracker is configured to track gaze of the user in order to identify an object the user is looking at; the processor is configured to decode a facial expression of the user based on data received from at least one of the first and second cameras, and to associate the decoded facial expression with the object.
12. The HMS of claim 1 , further comprising a wearable wireless transceiver configured to connect the HMS with a computer that is not carried by the user; and further comprising a facial expression compressor configured to receive images captured by the first and second cameras, and to extract points of interest that represent movements of the portions of the user's right and left eyebrows, wherein storing the points of interest requires less than 10% of a storage required to store the images captured by the first and second cameras, and transmitting the points of interest to the computer.
13. The HMS of claim 1 , further comprising a display and a controller; the display is coupled to the frame and configured to present digital content to the user; wherein the controller is configures to command the first and second cameras to capture images at a higher rate when the display presents an object that is expected to cause the user to have a noticeable emotional response, wherein the noticeable emotional response corresponds to a part of the user's face moving within a predetermined distance, compared to a rate of capturing images by the first and second cameras when the display presents an object that is not expected to cause the user to have the noticeable emotional response.
14. The HMS of claim 1 , further comprising a processor configured to receive data derived from images captured by at least one of the first, second, third, and fourth cameras, identify a facial expression expressed by the user, and provide to the user a feedback related to the identified facial expression.
15. The HMS of claim 1 , further comprising brainwave electrodes coupled to the frame and a computer configured to calculate affective response of the user based on data received from the brainwave electrodes and the cameras.
16. The HMS of claim 1 , further comprising a computer configured to identify brow contraction based on data captured by the first and second cameras; the computer is further configured to alert the user to release the contraction after identifying a contraction above a first threshold that is held for duration longer than a second threshold.
17. The HMS of claim 1 , further comprising a fifth camera coupled to the frame and located behind the user's ears, and a computer configured to estimate the user's posture and facial expression based on data captured by the first, second, third, fourth, and fifth cameras.
18. The HMS of claim 1 , further comprising a computer configured to estimate facial expressions of the user based on a model of a human face parameterized by expression and data extracted from images captured by at least one of the first, second, third, and fourth cameras.
19. The HMS of claim 18 , wherein the computer is further configured to render an avatar of the user based on the facial expressions.
20. The HMS of claim 18 , wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user his/her own avatar; whereby seeing his/her own avatar helps the user to understand how he/she looks to others.