Systems and methods for facial expression tracking
A facial tracking device including an illuminator and a photon detector. The illuminator configured to project a light toward a first portion of a head of a user and the photon detector configured to detect light reflected from a second portion of the head of the user. The facial tracking device further including a processor that is connected to the illuminator and photon detector and configured to cause the illuminator to project the light toward the head of the user, receive information from the photon detector, and determine a facial expression of the user.
1. A head wearable display (HWD) comprising:
an element configured to be worn on a head of a user;
a first illuminator disposed with the element and configured to emit first light toward at least a first portion of the head;
a second illuminator disposed with the element and configured to emit second light toward at least a second portion of the head;
a first photon detector disposed with the element without contacting at least the first portion of the head and configured to receive the first light reflected from at least the first portion of the head;
a second photon detector disposed with the element without contacting at least the second portion of the head and configured to receive the second light reflected from at least the second portion of the head;
one or more processors coupled to a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
determine a facial expression of the user based at least on a first distance to at least the first portion determined using the first light from the first photon detector and a second distance to at least the second portion determined using the second light from the second photon detector.
2. The HWD of claim 1 , wherein the first portion of the head and the second portion of the head overlap at least partially.
3. The HWD of claim 1 , further comprising additional photon detectors arranged in an array and configured to receive light reflected from respective portions of the head.
4. The HWD of claim 3 , wherein to determine the facial expression, the non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to further receive information from each of the additional photon detectors.
5. The HWD of claim 1 , further comprising additional illuminators, wherein additional photon detectors are arranged in an array and each of the additional illuminators configured to emit light with spatially encoded characteristics toward respective portions of the head.
6. The HWD of claim 5 , wherein the first photon detector provides characteristics of the first, and wherein the element is a glass frame or a housing.
7. The HWD of claim 6 , wherein the first and second portions of the head are on one side of a nose of the user.
8. A facial tracking device comprising:
an illuminator configured to emit light toward a face of a user;
a first photon detector disposed with an element without contacting at least a first portion of a head and configured to receive first light reflected from at least a first portion of the head;
a second photon detector disposed with the element without contacting at least a second portion of the head and configured to receive second light reflected from at least a second portion of the head; and
a processor coupled to a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by the processor, cause the processor to:
determine a facial expression of the user based at least on a first distance to at least the first portion of the head determined using the first light from the first photon detector and a second distance to at least the second portion of the head determined using the second light from the second photon detector.
9. The facial tracking device of claim 8 , further comprising circuitry configured to indicate the facial expression of the face based on characteristics received by the first photon detector and the second photon detector.
10. The facial tracking device of claim 9 , wherein the circuitry comprises a winner take all circuit configured to output a signal to the processor that is indicative of the facial expression of the face to the processor.
11. The facial tracking device of claim 8 , further comprising more photon detectors, wherein the more photon detectors are arranged in an array and configured to receive light reflected from respective portions of the head.
12. The facial tracking device of claim 11 , wherein the respective portions of the head are all located on a first side of a nose of the face, and wherein the facial tracking device further comprises:
a second illuminator emit light toward at least a portion of a second side of the face;
a second set of more photon detectors, wherein the second set is arranged in a second array and configured to receive light reflected from respective portions of the second side of the face.
13. The facial tracking device of claim 12 , wherein to determine the facial expression the processor is further configured to receive information regarding characteristics of the light from the more photon detectors and the second set of the more photon detectors.
14. The facial tracking device of claim 8 , wherein the first portion of the head and the second portion of the head overlap at least partially.
15. The facial tracking device of claim 8 , wherein characteristics of the light of the illuminator are spatially encoded corresponding to a face of the user.
16. A method of tracking a facial expression of a user comprising:
projecting light using an illuminator of a facial tracking system toward a face of a user;
receiving, via the facial tracking system, information from a first photon detector and information from a second photon detector, the information comprising characteristics of the light reflected from the face; and
determining, via the facial tracking system, a facial expression of the face based on a first distance to at least a first portion of the face determined using first light reflected from the face and a second distance to at least a second portion of the face determined using second reflected from the face.
17. The method of claim 16 , wherein turning on the illuminator further comprises turning on multiple illuminators of the facial tracking system such that each of the multiple illuminators project a light having different characteristics toward respective portions of the face.
18. The method of claim 17 , wherein the determining the facial expression further comprises reconstructing an image of at least a portion of the face based on the information.
19. The method of claim 16 , further comprising:
receiving, via the facial tracking system, information from multiple photon detectors, wherein each of the multiple photon detectors are configured to receive light from respective portions of the face.
20. The method of claim 19 , wherein the determining the facial expression further comprises reconstructing an image of at least a portion of the face based on the information from the multiple photon detectors.