Systems and techniques for estimating eye pose
An eye tracking system can include eye-tracking camera(s) configured to obtain images of the eye at different exposure times or different frame rates. For example, longer exposure images of the eye taken at a longer exposure time can show iris or pupil features, and shorter exposure, glint images can show peaks of glints reflected from the eye. The shorter exposure glint images may be taken at a higher frame rate than the longer exposure images for more accurate gaze prediction. The shorter exposure glint images can be analyzed to provide glint locations to subpixel accuracy. The longer exposure images can be analyzed for pupil center and/or center of rotation. The eye tracking system can predict gaze direction, which can be used for foveated rendering by a wearable display system. In some instances, the eye-tracking system may estimate the location of a partially or totally occluded glint.
1 . A head-mounted system comprising:
one or more cameras configured to capture images of an eye of a user;
a plurality of light sources configured to illuminate the eye of the user to produce glints in the images of the eye that are captured by the one or more cameras, wherein a total quantity of light sources included in the plurality of light sources is greater than or equal to a particular value; and
one or more processors operatively coupled to the one or more cameras, the one or more processors configured to:
receive an image of the eye from the one or more cameras;
identify a location in the image at which a center of a pupil of the eye is shown;
detect a set of one or more locations in the image at which one or more glints are shown, respectively;
determine whether a total quantity of locations included in the detected set of one or more glint locations is less than the particular value; and
in response to a determination that the total quantity of locations included in the detected set of one or more glint locations is less than the particular value:
apply the identified location of the center of the pupil and the detected set of one or more glint locations to a probabilistic statistical model to estimate a set of one or more glint locations for the image, the estimated set of one or more glint locations being different than the detected set of one or more glint locations; and
determine a pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
2 . The system of claim 1 , wherein the one or more processors are further configured to:
apply the identified location of the center of the pupil of the eye of the user to the probabilistic statistical model to identify a plurality of regions in the image of the eye of the user at which a plurality of glints are expected to be shown, respectively, wherein a total quantity of regions included in the plurality of identified regions is greater than or equal to the particular value,
wherein to detect the set of one or more locations in the image of the eye of the user at which one or more glints are shown, respectively, the one or more processors are configured to detect a set of one or more locations in one or more of the plurality of identified regions, respectively.
3 . The system of claim 1 , wherein to apply the identified location of the center of the pupil of the eye of the user and the detected set of one or more glint locations to the probabilistic statistical model to estimate the set of one or more locations in the image of the eye of the user at which one or more glints are shown, respectively, the one or more processors are configured to:
apply the identified location of the center of the pupil of the eye of the user to the probabilistic statistical model to identify one or more regions in the image of the eye of the user at which one or more glints are expected to be shown, respectively; and
apply the detected set of one or more glint locations to the probabilistic statistical model to estimate a set of one or more locations in the one or more identified regions of the image of the eye of the user at which one or more glints are shown, respectively.
4 . The system of claim 1 , wherein the one or more processors are further configured to: update the probabilistic statistical model based on the identified location of the center of the pupil of the eye of the user and the detected set of one or more glint locations.
5 . The system of claim 1 , wherein the particular value is three.
6 . The system of claim 1 , wherein the plurality of light sources are infrared light sources.
7 . The system of claim 1 , wherein to determine the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, the one or more processors are configured to:
determine a location of a center of a cornea of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
8 . The system of claim 1 , wherein to determine the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, the one or more processors are configured to:
determine a location of a center of rotation of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
9 . The system of claim 1 , wherein to determine the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, the one or more processors are configured to:
determine a gaze direction of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
10 . The system of claim 1 , wherein the one or more processors are further configured to:
in response to a determination that the total quantity of locations included in the detected set of one or more glint locations is greater than the particular value, determine a pose of the eye of the user based at least in part on the detected set of one or more glint locations.
11 . A method performed by one or more processors of a head-mounted system, the method comprising:
receiving an image of an eye of a user, the image captured by one or more cameras of the head-mounted system;
identifying a location in the image at which a center of a pupil of the eye is shown;
detecting a set of one or more locations in the image at which one or more glints are shown, respectively, wherein the one or more glints are produced by a plurality of light sources of the head-mounted system that are configured to illuminate the eye, wherein a total quantity of light sources included in the plurality of light sources is greater than or equal to a particular value;
determining whether a total quantity of locations included in the detected set of one or more glint locations is less than the particular value; and
in response to a determination that the total quantity of locations included in the detected set of one or more glint locations is less than the particular value:
applying the identified location of the center of the pupil and the detected set of one or more glint locations to a probabilistic statistical model to estimate a set of one or more glint locations for the image, the estimated set of one or more glint locations being different than the detected set of one or more glint locations; and
determining a pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
12 . The method of claim 11 , further comprising:
applying the identified location of the center of the pupil of the eye of the user to the probabilistic statistical model to identify a plurality of regions in the image of the eye of the user at which a plurality of glints are expected to be shown, respectively, wherein a total quantity of regions included in the plurality of identified regions is greater than or equal to the particular value,
wherein detecting the set of one or more locations in the image of the eye of the user at which one or more glints are shown, respectively, includes detecting a set of one or more locations in one or more of the plurality of identified regions, respectively.
13 . The method of claim 11 , wherein applying the identified location of the center of the pupil of the eye of the user and the detected set of one or more glint locations to the probabilistic statistical model to estimate the set of one or more locations in the image of the eye of the user at which one or more glints are shown, respectively, further comprises:
applying the identified location of the center of the pupil of the eye of the user to the probabilistic statistical model to identify one or more regions in the image of the eye of the user at which one or more glints are expected to be shown, respectively; and
applying the detected set of one or more glint locations to the probabilistic statistical model to estimate a set of one or more locations in the one or more identified regions of the image of the eye of the user at which one or more glints are shown, respectively.
14 . The method of claim 11 , further comprising:
updating the probabilistic statistical model based on the identified location of the center of the pupil of the eye of the user and the detected set of one or more glint locations.
15 . The method of claim 11 , wherein the particular value is three.
16 . The method of claim 11 , wherein the plurality of light sources are infrared light sources.
17 . The method of claim 11 , wherein determining the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, further comprises:
determining a location of a center of a cornea of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
18 . The method of claim 11 , wherein determining the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, further comprises:
determining a location of a center of rotation of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
19 . The method of claim 11 , wherein determining the pose of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations, further comprises:
determining a gaze direction of the eye of the user based at least in part on the detected set of one or more glint locations and the estimated set of one or more glint locations.
20 . The method of claim 11 , further comprising:
in response to a determination that the total quantity of locations included in the detected set of one or more glint locations is greater than the particular value, determining a pose of the eye of the user based at least in part on the detected set of one or more glint locations.