Vestibulo-ocular reflex based eye tracking prediction
A vestibulo-ocular reflex (VOR) eye tracking prediction method leverages head tracking functionality of a device to predict and compensate for latency caused by the VOR. Head motion may be accurately detected and predicted by a head tracking system of the device. The head motion prediction may be leveraged to predict where the eye with the assumed VOR motion will be looking when an image is displayed. In the method, head motion is detected, images of the eyes are captured, and the predicted head motion is used to predict VOR motion and thus pupil position and gaze direction so that an image is rendered with corrections and displayed with a more accurate pupil position and gaze direction than can be obtained with naïve gaze direction estimated based on images of the eyes captured by the eye-facing cameras that does not account for VOR, and thus introduces latency.
1 . A device, comprising:
one or more head motion sensors configured to collect head motion data;
a camera configured to capture images of an eye; and
a controller comprising one or more processors configured to:
process head motion data captured by the one or more head motion sensors to generate first device pose information for a first time corresponding to when an image of the eye was captured by the camera and second device pose information for a second time subsequent to the first time; and
process the image of the eye to generate eye pose information including a pupil position and a gaze vector;
estimate vestibulo-ocular reflex (VOR) motion of the eye based on a difference between the first device pose information and the second device pose information; and
adjust the pupil position and the gaze vector based at least in part on the estimated VOR motion of the eye.
2 . The device as recited in claim 1 , wherein, to process the image of the eye captured by the camera to generate eye pose information including a pupil position and a gaze vector, the controller is configured to:
obtain the first device pose information based on head pose data corresponding to a head pose at a time that the image of the eye was captured; and
obtain the second device pose information based at least in part on head pose data corresponding to a head pose at a later time than the time that the image of the eye was captured.
3 . The device as recited in claim 2 , wherein the later time corresponds to a time at which a current frame being composited by the controller is to be displayed by a display subsystem of the device.
4 . The device as recited in claim 1 , further comprising a display subsystem configured to display frames to an eye, and wherein the controller is further configured to:
apply one or more corrections to an image based at least in part on the adjusted pupil position and gaze vector, wherein the pupil position and gaze vector account for VOR motion of the eye between a time that the image of the eye was captured and a time that the current frame is to be displayed; and
provide the corrected image to the display subsystem as the current frame.
5 . The device as recited in claim 4 , wherein the one or more include a correction for distortion caused by an eyepiece located between the eye and a display screen of the display subsystem, wherein the correction to be applied for the distortion caused by the eyepiece depends on the pupil position with respect to the eyepiece.
6 . The device as recited in claim 1 , wherein, to process an image of the eye captured by the camera to generate eye pose information including a pupil position and a gaze vector based at least in part on the device pose information to account for VOR motion of the eye, the controller is further configured to generate the eye pose information based at least in part on a geometric model of the eye.
7 . The device as recited in claim 1 , wherein, to process an image of the eye captured by the camera to generate eye pose information including a pupil position and a gaze vector based at least in part on the device pose information to account for VOR motion of the eye, the controller is further configured to generate the eye pose information based at least in part on an anatomical model of the eye.
8 . The device as recited in claim 1 , wherein, to process an image of the eye captured by the camera to generate eye pose information including a pupil position and a gaze vector based at least in part on the device pose information to account for VOR motion of the eye, the controller is further configured to estimate distance of a focus point, wherein the estimated distance of the focus point is used to adjust an amount of VOR motion.
9 . The device as recited in claim 1 , wherein the one or more head motion sensors include an inertial motion unit (IMU).
10 . The device as recited in claim 1 , wherein the device is a head-mounted device (HMD) of an extended reality (XR) system.
11 . A method, comprising:
performing, by a controller comprising one or more processors:
processing an image of the eye captured by a camera to generate eye pose information including a pupil position and a gaze vector;
processing head motion data captured by one or more head motion sensors to generate first device pose information for a first time corresponding to when an image of the eye was captured by the camera and second device pose information for a second time subsequent to the first time;
processing the image of the eye to generate eye pose information including a pupil position and a gaze vector;
estimating vestibulo-ocular reflex (VOR) motion of the eye based on a difference between the first device pose information and the second device pose information; and
adjusting the pupil position and the gaze vector based at least in part on the estimated VOR motion of the eye.
12 . The method as recited in claim 11 , further comprising:
obtaining the first device pose information based on head pose data corresponding to a head pose at a time that the image of the eye was captured;
obtaining the second device pose information based at least in part on head pose data corresponding to a head pose at a later time than the time that the image of the eye was captured.
13 . The method as recited in claim 12 , wherein the later time corresponds to a time at which a current frame being composited by the controller is to be displayed by a display subsystem of the device.
14 . The method as recited in claim 11 , further comprising:
correcting an image based at least in part on the adjusted pupil position and gaze vector, wherein the adjusted pupil position and gaze vector account for VOR motion of the eye between a time that the image of the eye was captured and a time that the current frame is to be displayed; and
causing the corrected image to be displayed as the current frame.
15 . The method as recited in claim 14 , wherein correcting the image includes correcting for distortion caused by an eyepiece located between the eye and a display screen of the display subsystem, wherein the correction to be applied for the distortion caused by the eyepiece depends on the pupil position with respect to the eyepiece.
16 . The method as recited in claim 14 , wherein correcting the image includes one or more of corrections for distortions, chromatic aberrations, optical crosstalk, foveation, vignetting, and point of view.
17 . The method as recited in claim 11 , wherein the controller and camera are components of a head-mounted device (HMD) of an extended reality (XR) system.
18 . A system, comprising:
a head-mounted device (HMD), comprising
a display subsystem configured to display virtual content to an eye;
one or more head motion sensors configured to collect head motion data;
a camera configured to capture images of the eye; and
a controller comprising one or more processors configured to:
process head motion data captured by the one or more head motion sensors to generate first device pose information for a first time corresponding to when an image of the eye was captured by the camera and second device pose information for a second time subsequent to the first time;
process the image of the eye to generate eye pose information including a pupil position and a gaze vector;
estimate vestibulo-ocular reflex (VOR) motion of the eye based on a difference between the first device pose information and the second device pose information;
adjust the pupil position and gaze vector based at least in part on the estimated VOR motion of the eye;
apply one or more corrections to an image based at least in part on the adjusted pupil position and gaze vector; and
provide the corrected image to the display subsystem as the current frame.
19 . The system as recited in claim 18 , wherein, to process the image of the eye captured by the camera to generate eye pose information including a pupil position and a gaze vector, the controller is configured to:
obtain the first device pose information based on head pose data corresponding to a head pose at a time that the image of the eye was captured; and
obtain the second device pose information based at least in part on head pose data corresponding to a head pose at a later time than the time that the image of the eye was captured, wherein the later time corresponds to a time at which a current frame being composited by the controller is to be displayed by a display subsystem of the device.
20 . The system as recited in claim 18 , wherein the one or more corrections include a correction for distortion caused by an eyepiece located between the eye and a display screen of the display subsystem, wherein the correction to be applied for the distortion caused by the eyepiece depends on the pupil position with respect to the eyepiece.