Eye tracking system for use in head-mounted display units and method of operating same
A virtual reality/augmented reality (VR/AR) wearable assembly is described herein. The VR/AR wearable assembly includes a support frame, a display mounted to the support frame, a plurality of light emitters configured to illuminate an area of a subject's eye, a plurality of photosensors configured to receive reflected light from different portions of the illuminated subject's eye, and a controller operatively coupled to the display, the plurality of light emitters, and the plurality of photosensors. The controller includes a processor programmed to execute an algorithm including the steps of alternating illumination of the light emitters to generate predefined lighting patterns, acquiring data from the plurality of photosensors when corresponding light emitters are illuminated, mapping intensities of reflected light based on the acquired data from the plurality of photosensors, and determining a gaze position of the subject's eye based on the mapped intensities of reflected light.
1. A virtual reality/augmented reality (VR/AR) wearable assembly, comprising:
a head mounted display (HMD) unit housing;
a VR/AR display mounted to the HMD unit housing;
a pair of eye tracking assemblies mounted to the HMD unit housing positioned between the VR/AR display and a subject's eyes, each eye tracking assembly being associated with a corresponding subject's eye and including:
a support frame;
a plurality of light emitters mounted to the support frame and configured to illuminate an area of the corresponding subject's eye;
a plurality of photosensors mounted to the support frame and configured to receive reflected light from different portions of the illuminated corresponding subject's eye; and
a thermistor mounted to the support frame for sensing an ambient temperature adjacent the support frame;
wherein each light emitter is positioned adjacent to a corresponding photosensor; and
wherein the plurality of light emitters and the plurality of photosensors are arranged in a first group of photosensors and light emitters positioned along a first side of the corresponding subject's eye, and a second group of photosensors and light emitters positioned along a second side of the corresponding subject's eye opposite the first group of photosensors and light emitters; and
a controller operatively coupled to the thermistor, the plurality of light emitters, and the plurality of photosensors, the controller including a processor programmed to execute an algorithm including the steps of:
alternating illumination of the first and second groups of light emitters to generate predefined lighting patterns;
acquiring data including measured illumination readings from the first and second groups of photosensors when corresponding first and second groups of light emitters illuminate the corresponding subject's eye, with at least one photosensor including a corresponding detection area that overlaps a neighboring detection area of another photosensor;
receiving a measurement of an ambient temperature adjacent the support frame from the thermistor;
determining an estimated photosensor temperature value based on the measured ambient temperature adjacent the support frame;
calculating a compensated photosensor reading for each photosensor based on a measured illumination reading of each photosensor and the estimated photosensor temperature value;
mapping intensities of reflected light based on the acquired data from the plurality of photosensors including the calculated compensated photosensor reading for each photosensor using a neural network; and
determining a gaze position of the corresponding subject's eye based on the mapped intensities of reflected light.
2. The VR/AR wearable assembly of claim 1 , wherein the processor is programmed to execute the algorithm including the steps of:
alternating illumination of the light emitters such that only one group of light emitters is illuminated at a time.
3. The VR/AR wearable assembly of claim 2 , wherein the processor is programmed to execute the algorithm including the steps of:
acquiring data from a corresponding group of photosensors associated with a currently illuminated group of light emitters.
4. The VR/AR wearable assembly of claim 2 , wherein the processor is programmed to execute the algorithm including the steps of:
acquiring data from the first group of photosensors associated with the first group of light emitters when the first group of light emitters is illuminated; and
acquiring data from the second group of photosensors associated with the second group of light emitters when the second group of light emitters is illuminated.
5. The VR/AR wearable assembly of claim 1 , further comprising:
an optical-based sensor mounted to the support frame for detecting movements between the VR/AR wearable assembly and a subject's skin; and
the processor is programmed to execute the algorithm including the steps of:
correcting the determined the gaze position of the subject's eye based on the detected movements between the VR/AR wearable assembly and the subject's skin.
6. The VR/AR wearable assembly of claim 1 , wherein the processor is programmed to execute the algorithm including the steps of:
estimating an inter-pupilar distance (IPD) based on the determined gaze position of the eye.
7. The VR/AR wearable assembly of claim 1 , wherein the processor is programmed to execute the algorithm including the steps of:
rendering an avatar on the display; and
adjusting a gaze position of an eye of the avatar based on the determined gaze position of the subject's eye.
8. A method of operating a VR/AR wearable assembly including a head mounted display (HMD) unit housing, a VR/AR display mounted to the HMD unit housing, a pair of eye tracking assemblies mounted to the HMD unit housing positioned between the VR/AR display and a subject's eyes, and a processor operatively coupled to the pair of eye tracking assemblies, each eye tracking assembly being associated with a corresponding subject's eye and including a support frame, a plurality of light emitters mounted to the support frame and configured to illuminate an area of the corresponding subject's eye, a plurality of photosensors mounted to the support frame and configured to receive reflected light from different portions of the illuminated corresponding subject's eye, and a thermistor mounted to the support frame for sensing an ambient temperature adjacent the support frame, each light emitter is positioned adjacent to a corresponding photosensor, and the plurality of light emitters and the plurality of photosensors are arranged in a first group of photosensors and light emitters positioned along a first side of the corresponding subject's eye, and a second group of photosensors and light emitters positioned along a second side of the corresponding subject's eye opposite the first group of photosensors and light emitters, the method including the processor performing an algorithm including the steps of:
alternating illumination of the first and second groups of light emitters to generate predefined lighting patterns;
acquiring data including measured illumination readings from the first and second groups of photosensors when corresponding first and second groups light emitters illuminate the corresponding subject's eye, with at least one photosensor including a corresponding detection area that overlaps a neighboring detection area of another photosensor;
receiving a measurement of an ambient temperature adjacent the support frame from the thermistor;
determining an estimated photosensor temperature value based on the measured ambient temperature adjacent the support frame;
calculating a compensated photosensor reading for each photosensor based on a measured illumination reading of each photosensor and the estimated photosensor temperature value;
mapping intensities of reflected light based on the acquired data including the calculated compensated photosensor reading for each photosensor from the plurality of photosensors using a neural network; and
determining a gaze position of the corresponding subject's eye based on the mapped intensities of reflected light.
9. The method of claim 8 , including the processor performing the algorithm including the steps of:
alternating illumination of the light emitters such that only one group of light emitters is illuminated at a time.
10. The method of claim 9 , including the processor performing the algorithm including the steps of:
acquiring data from a corresponding group of photosensors associated with a currently illuminated group of light emitters.
11. The method of claim 9 , including the processor performing the algorithm including the steps of:
acquiring data from the first group of photosensors associated with the first group of light emitters when the first group of light emitters is illuminated; and
acquiring data from the second group of photosensors associated with the second group of light emitters when the second group of light emitters is illuminated.
12. The method of claim 8 , including the processor performing the algorithm including the steps of:
operating an optical-based sensor mounted to the support frame to detect movements between the VR/AR wearable assembly and a subject's skin; and
correcting the determined the gaze position of the subject's eye based on the detected movements between the VR/AR wearable assembly and the subject's skin.
13. The method of claim 8 , including the processor performing the algorithm including the steps of:
estimating an inter-pupilar distance (IPD) based on the determined gaze position of the eye.
14. The method of claim 8 , including the processor performing the algorithm including the steps of:
rendering an avatar on the display; and
adjusting a gaze position of an eye of the avatar based on the determined gaze position of the subject's eye.
15. A non-transitory computer-readable storage media having computer-executable instructions embodied thereon for operating a VR/AR wearable assembly including a head mounted display (HMD) unit housing, a VR/AR display mounted to the HMD unit housing, a pair of eye tracking assemblies mounted to the HMD unit housing positioned between the VR/AR display and a subject's eyes, and a processor operatively coupled to the pair of eye tracking assemblies, each eye tracking assembly being associated with a corresponding subject's eye and including a support frame, a plurality of light emitters mounted to the support frame and configured to illuminate an area of the corresponding subject's eye, a plurality of photosensors mounted to the support frame and configured to receive reflected light from different portions of the illuminated corresponding subject's eye, and a thermistor mounted to the support frame for sensing an ambient temperature adjacent the support frame, each light emitter is positioned adjacent to a corresponding photosensor, and the plurality of light emitters and the plurality of photosensors are arranged in a first group of photosensors and light emitters positioned along a first side of the corresponding subject's eye, and a second group of photosensors and light emitters positioned along a second side of the corresponding subject's eye opposite the first group of photosensors and light emitters, when executed by the processor the computer-executable instructions cause the processor to perform an algorithm including the steps of:
alternating illumination of the first and second groups of light emitters to generate predefined lighting patterns;
acquiring data including measured illumination readings from the first and second groups of photosensors when corresponding first and second groups light emitters illuminate the corresponding subject's eye, with at least one photosensor including a corresponding detection area that overlaps a neighboring detection area of another photosensor;
receiving a measurement of an ambient temperature adjacent the support frame from the thermistor;
determining an estimated photosensor temperature value based on the measured ambient temperature adjacent the support frame;
calculating a compensated photosensor reading for each photosensor based on a measured illumination reading of each photosensor and the estimated photosensor temperature value;
mapping intensities of reflected light based on the acquired data including the calculated compensated photosensor reading for each photosensor from the plurality of photosensors using a neural network; and
determining a gaze position of the corresponding subject's eye based on the mapped intensities of reflected light.
16. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of:
alternating illumination of the light emitters such that only one group of light emitters is illuminated at a time; and
acquiring data from a corresponding group of photosensors associated with a currently illuminated group of light emitters.
17. The non-transitory computer-readable storage media of claim 16 , wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of:
acquiring data from the first group of photosensors associated with the first group of light emitters when the first group of light emitters is illuminated; and
acquiring data from the second group of photosensors associated with the second group of light emitters when the second group of light emitters is illuminated.
18. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the processor to perform the algorithm including the steps of:
operating an optical-based sensor mounted to the support frame to detect movements between the VR/AR wearable assembly and a subject's skin; and
correcting the determined the gaze position of the subject's eye based on the detected movements between the VR/AR wearable assembly and the subject's skin.