Eye tracking based on optical flow
Eye motion is determined based on optical flow in images of an eye where the images are of reflections of light (coherent or incoherent) off of an eye. The reflections may be produced by selectively using one or more light sources and/or the images may be captured by selectively using one or more image sensors. For example, a subset of less than all of multiple available light sources and/or a subset of less than all of multiple available image sensors may be selected and used at a time based on the pose of the eye. One of the light sources and/or image sensors may be best suited for capturing images for eye tracking purposes when the user is looking left and while another light source and/or image sensor may be best suited when the user is looking right. The light sources and/or image sensors may be positioned to enable capturing images across the full range of eye poses.
1 . A method comprising:
selecting, at an electronic device having a processor, a light source from a plurality of light sources or an image sensor from a plurality of image sensors for optical-flow-based eye tracking, wherein the light source or image sensor is selected based on a pose of an eye;
producing, at the electronic device, light from the light source towards a portion of the eye, the light reflecting from the portion of the eye as reflected light;
capturing, at the electronic device, a sequence of images of the reflected light using the image sensor;
assessing, at the electronic device, image-to-image changes in the sequence of images, the image-to-image changes comprising an amount and a direction of positional change in at least one feature depicted in the images; and
determining, at the electronic device, a movement of the eye based on the image-to-image changes in the sequence of images.
2 . The method of claim 1 , wherein the method selects the light source from the plurality of light sources based on the pose of the eye.
3 . The method of claim 1 , wherein the method selects the image sensor from the plurality of image sensors based on the pose of the eye.
4 . The method of claim 1 , wherein the electronic device comprises a plurality of light source-to-image sensor pairs, wherein a respective pair of the pairs is selected to track the eye movement based on the pose of the eye.
5 . The method of claim 1 , wherein selecting the light source or the image sensor is based on specular coverage.
6 . The method of claim 1 , wherein the electronic device is configured to provide multiple optical paths from a plurality of light sources to a plurality of image sensors, wherein the multiple optical paths are configured to capture images of the portion of the eye while the eye is positioned in any position within an entire range of eye motion of the eye.
7 . The method of claim 6 further comprising toggling between the multiple optical paths based on tracking movement of the eye.
8 . The method of claim 1 , wherein the plurality of light sources comprises one or more lasers.
9 . The method of claim 1 , wherein the light source comprises one or more fixed position light emitting diodes (LEDs).
10 . The method of claim 1 , wherein the light source comprises one or more scanning beam light emitting diodes (LEDs).
11 . The method of claim 1 , wherein determining the movement of the eye comprises:
determining an optical flow; and
determining an amount and direction of eye movement based on the optical flow and the relative positioning between the one more image sensors and the eye.
12 . The method of claim 11 , wherein determining the optical flow is based on determining a velocity based on the image-to-image changes in the sequence of images.
13 . The method of claim 1 , wherein the portion of the eye is a cornea, iris, or sclera of the eye.
14 . The method of claim 1 , wherein a resolution of each of the one or more image sensors is less than 50 by 50 pixels.
15 . The method of claim 1 , wherein a frame rate of the one or more image sensors is greater than 30 frames per second.
16 . The method of claim 1 , wherein determining the movement of the eye comprises determining an angular velocity of the eye.
17 . The method of claim 1 , wherein determining the movement of the eye comprises determining an angular position of the eye.
18 . The method of claim 1 , wherein determining the movement of the eye comprises:
classifying and quantifying a user gaze behavior;
identifying a blink of the eye;
identifying a smooth pursuit of the eye;
identifying a saccade of the eye;
identifying a fixation of the eye; or
identifying a nystagmus of the eye.
19 . The method of claim 1 , wherein determining the movement of the eye comprises determining an eye characteristic based on:
the determining of the movement of the eye; and
a gaze direction determined from a second tracking system having a sampling rate that is less than a frame rate of the one or more sensors.
20 . The method of claim 1 further comprising calibrating motion tracking output to compensate for geometrical factors.
21 . The method of claim 1 , wherein the eye movement is determined based both coherent and incoherent light sources illuminating the eye.
22 . The method of claim 1 further comprising initiating a functionality on the electronic device based on assessing the image-to-image changes in the sequence of images.
23 . The method of claim 1 , wherein the determining the movement of the eye comprises determining a gaze direction of the eye relative to a viewable portion of a head-mounted device, wherein the head mounted device comprises frame portions surrounding viewable portions, wherein the frame portion comprises a plurality of light sources and a plurality of image sensors.
24 . The method of claim 1 , wherein the selecting comprises both selecting the light source from the plurality of light sources and selecting the image sensor from the plurality of image sensors based on the pose of the eye.
25 . A device comprising:
one or more light sources;
one or more image sensors; and
a processor configured to execute instructions stored in a non-transitory computer-readable medium to perform operations comprising:
selecting, at an electronic device having a processor, a light source from a plurality of light sources or an image sensor from a plurality of image sensors for optical-flow-based eye tracking, wherein the light source or image sensor is selected based on a pose of an eye;
producing light from the light source towards a portion of the eye, the light reflecting from the portion of the eye as reflected light;
capturing a sequence of images of the reflected light using the image sensor; assessing image-to-image changes in the sequence of images, the image-to-image changes comprising an amount and a direction of positional change in at least one feature depicted in the images; and
determining a movement of the eye based on the image-to-image changes in the sequence of images.
26 . A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
selecting, at an electronic device having a processor, a light source from a plurality of light sources or an image sensor from a plurality of image sensors for optical-flow-based eye tracking, wherein the light source or image sensor is selected based on a pose of an eye;
producing light from the light source towards a portion of the eye, the light reflecting from the portion of the eye as reflected light;
capturing a sequence of images of the reflected light using the image sensor; assessing image-to-image changes in the sequence of images, the image-to-image changes comprising an amount and a direction of positional change in at least one feature depicted in the images; and
determining a movement of the eye based on the image-to-image changes in the sequence of images.