Eye tracking using eye odometers
Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.
1. A system, comprising:
a head-mounted device (HMD) configured to display visual content for viewing by a user, wherein the HMD comprises:
at least one camera configured to capture images of the user's eyes at a frame rate; and
one or more eye motion sensors configured to detect movement of the user's eyes with respect to the HMD; and
a controller comprising one or more processors configured to:
process images captured by the at least one camera to generate or update one or more 3-dimentional (3D) models of the user's eyes to determine position of the user's eyes with respect to the HMD at the frame rate of the at least one camera; and
provide a relative update to the 3D models of the user's eyes based on processing of signals from the one or more eye motion sensors to update the position of the user's eyes with respect to the HMD that is determined based on the images captured by the at least one camera, in intervals between the processing of images captured by the at least one camera.
2. The system as recited in claim 1 , wherein, to process images captured by the at least one camera to determine position of the user's eyes with respect to the HMD at the frame rate of the at least one camera, the controller is configured to perform a three-dimensional (3D) reconstruction based on at least one image captured by the at least one camera to generate the 3D models of the user's eyes, wherein the 3D models indicate position of the user's eyes with respect to the at least one camera.
3. The system as recited in claim 2 , wherein the controller is further configured to perform a new 3D reconstruction based on at least one subsequent image captured by the at least one camera upon detecting movement of the HMD with respect to the user's head.
4. The system as recited in claim 1 , wherein to provide the relative update to the 3D models of the user's eyes, the controller is further configured to provide the relative update to the 3D models of the user's eyes upon detecting movement of the HMD with respect to the user's head.
5. The system as recited in claim 1 , wherein said processing signals from the eye motion sensors to track relative movement of the user's eyes with respect to the HMD in intervals between the processing of images captured by the at least one camera allows the frame rate of the at least one camera to be reduced to ten frames per second or less.
6. The system as recited in claim 1 , wherein the eye motion sensors include one or more of photosensors and photodiode arrays.
7. The system as recited in claim 1 , wherein the controller is a component of the HMD.
8. The system as recited in claim 1 , wherein the HMD further comprises:
at least one display screen configured to display frames containing the visual content for viewing by the user, wherein the controller is further configured to render the frames containing the visual content for display by the at least one display screen; and
one or more light sources configured to emit light towards the user's eyes, wherein the at least one camera captures a portion of the light reflected off the user's eyes.
9. The system as recited in claim 8 , wherein the HMD further comprises left and right optical lenses located between the at least one display screen and the user's eyes.
10. A method, comprising:
performing, by a controller comprising one or more processors:
processing images captured by at least one camera of a head-mounted device (HMD) to generate or update one or more 3-dimentional (3D) models of the user's eyes to determine position of a user's eyes with respect to the HMD at a frame rate of the at least one camera; and
providing a relative update to the 3D models of the user's eyes based on processing of signals from one or more eye motion sensors of the HMD to update the position of the user's eyes with respect to the HMD that is determined based on the images captured by the at least one camera, in intervals between the processing of images captured by the at least one camera.
11. The method as recited in claim 10 , wherein processing images captured by the at least one camera to determine position of the user's eyes with respect to the HMD at the frame rate of the at least one camera comprises performing a three-dimensional (3D) reconstruction based on at least one image captured by the at least one camera to generate the 3D models of the user's eyes, wherein the 3D models indicate position of the user's eyes with respect to the at least one camera.
12. The method as recited in claim 11 , further comprising performing a new 3D reconstruction based on at least one subsequent image captured by the at least one camera upon detecting movement of the HMD with respect to the user's head.
13. The method as recited in claim 11 , further comprising updating the 3D models of the user's eyes upon detecting movement of the HMD with respect to the user's head.
14. The method as recited in claim 10 , wherein the eye motion sensors include one or more of photosensors and photodiode arrays.
15. The method as recited in claim 10 , wherein the controller is a component of the HMD.
16. The method as recited in claim 10 , further comprising emitting, by one or more light sources of the HMD, light towards the user's eyes, wherein the eye motion sensors sense a portion of the light reflected off the user's eyes.
17. The method as recited in claim 10 , further comprising rendering, by the controller, frames containing visual content for display by at least one display screen of the HMD.
18. One or more non-transitory computer-readable storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
process images captured by at least one camera of a head-mounted device (HMD) to generate or update one or more 3-dimentional (3D) models of the user's eyes to determine position of a user's eyes with respect to the HMD at a frame rate of the at least one camera; and
provide a relative update to the 3D models of the user's eyes based on processing of signals from one or more eye motion sensors of the HMD to update the position of the user's eyes with respect to the HMD that is determined based on the images captured by the at least one camera, in intervals between the processing of images captured by the at least one camera.