Event camera for eye tracking
An eye tracking system includes an event camera. The event camera includes an event sensor and a controller. The event sensor includes a plurality of photodiodes that asynchronously output data values corresponding to relative intensity changes of light reflected from a user's eyes. The controller populates an event matrix based in part on data values asynchronously received from the event sensor and positions of photodiodes associated with the received data values over a first time period. The controller populates a change matrix based in part on a threshold intensity value and the photodiodes associated with the received data values over the first time period, and generates an image of the user's eyes for the first time period using the event matrix and the change matrix. The eye tracking system uses the image of the user's eyes to determine eye tracking information indicating positions, orientations and/or movement of the user's eyes.
1. An eye tracking system comprising:
an event camera configured to image an eye of a user, the event camera comprising:
an event sensor comprising a plurality of photodiodes, the event sensor configured to operate in a mode of a plurality of modes, the plurality of modes including an event mode that causes the plurality of photodiodes to asynchronously output data values, wherein a data value output by each respective photodiode, of the plurality of photodiodes, is based at least in part on a difference of a value previously output by the photodiode and an intensity value captured by the photodiode relative to a threshold intensity value; and
a camera controller configured to:
populate an event matrix based in part on data values asynchronously received from the event sensor and positions of photodiodes of the plurality of photodiodes associated with the received data values over a first time period,
populate a change matrix based in part on the threshold intensity value and the photodiodes associated with the received data values over the first time period, and
generate an image of the eye for the first time period using the event matrix and the change matrix; and
a controller configured to determine eye tracking information using the generated image.
2. The eye tracking system of claim 1 , wherein the eye tracking system is part of a head-mounted display (HMD).
3. The eye tracking system of claim 2 , wherein the HMD includes an electronic display, and the HMD is configured to display content on a first portion of the electronic display at a target resolution, the first portion corresponding to a foveal region of the user's eye-gaze, and display content on other portions of the electronic display at a resolution that is less than the target resolution.
4. The eye tracking system of claim 3 , wherein the target resolution corresponds to a foveal resolution of a human eye.
5. The eye tracking system of claim 1 , wherein the plurality of modes includes a conventional mode causes the plurality of photodiodes to each output an intensity value that is used to generate an image frame.
6. The eye tracking system of claim 5 , wherein the controller is further configured to:
generate imaging instructions that determine the mode the event sensor operates in, wherein the imaging instructions cause the event sensor to operate in the conventional mode for a first time period, and then operate in the event mode for a second time period that is subsequent to the first time period; and
provide the imaging instructions to the event sensor.
7. The eye tracking system of claim 1 , wherein each photodiode of the photodiodes is configured to output a data value after an absolute value of the difference between the data value and a value previously output by the each photodiode is greater than the threshold intensity value.
8. The eye tracking system of claim 7 , wherein the threshold intensity value is based on a statistical distribution of expected threshold intensities.
9. The eye tracking system of claim 1 , wherein the threshold intensity value is determined by the controller.
10. The eye tracking system of claim 1 , wherein populating an event matrix based in part on data values asynchronously received from the event sensor and the positions of the photodiodes associated with the received data values over a first time period, comprises:
for each of the received data values that indicate than an address event (AE) occurred over the first time period, adding a new row to the event matrix;
inserting a value of ‘0’ at locations within the new row of the event matrix that are associated with the positions of the photodiodes that are associated with received data values that indicate no AE occurred over the first time period;
inserting a value at locations within the new row of the event matrix that are associated with the positions of the photodiodes that are associated with the received data values that indicate that an AE occurred over a time period prior to the first time period; and
inserting a different value at locations within the new row of the event matrix that are associated with the positions of the photodiodes that are associated with received data values that indicate than an AE occurred over the first time period.
11. The eye tracking system of claim 1 , wherein the populating a change matrix based in part on the threshold intensity value and the photodiodes associated with the received data values over the first time period, comprises:
for each of the received data values that indicate that an address event (AE) occurred over the first time period, adding a new row to the change matrix; and
inserting a value that is a multiple of one of the received data values that indicates that the AE occurred in one or more positions in the new row.
12. The eye tracking system of claim 1 , wherein the controller is configured to generate the image for the first time period using the event matrix and the change matrix, based in part on solving for an intensity matrix using the equation:
Ax=y
where A is the event matrix,
y is the change matrix, and
x is the intensity matrix.
13. An eye tracking system comprising:
an event camera configured to image an eye of a user, the event camera comprising:
an event sensor comprising a plurality of photodiodes, the event sensor configured to operate in a mode of a plurality of modes, the plurality of modes including an event mode that causes the plurality of photodiodes to asynchronously output data values, wherein a data value output by each respective photodiode, of the plurality photodiodes, is based at least in part on a difference of a value previously output by the photodiode and an intensity value captured by the photodiode relative to a threshold intensity value;
an optical assembly configured to direct light from a local area in a field of view of the eye tracking system to the event sensor; and
a camera controller configured to:
populate an event matrix based in part on data values asynchronously received from the event sensor and positions of photodiodes of the plurality of photodiodes associated with the received data values over a first time period,
populate a change matrix based in part on the threshold intensity value and the photodiodes associated with the received data values over the first time period, and
generate an image of the eye for the first time period using the event matrix and the change matrix; and
a controller configured to determine eye tracking information using the generated image.
14. The eye tracking system of claim 13 , wherein the eye tracking system is part of a head-mounted display (HMD).
15. The eye tracking system of claim 14 , wherein the HMD includes an electronic display, and the HMD is configured to display content on a first portion of the electronic display at a target resolution, the first portion corresponding to a foveal region of the user's eye-gaze, and display content on other portions of the electronic display at a resolution that is less than the target resolution.
16. The eye tracking system of claim 15 , wherein the target resolution corresponds to a foveal resolution of a human eye.
17. The eye tracking system of claim 13 , wherein each photodiode of the photodiodes is configured to output a data value after an absolute value of the difference between the data value and a value previously output by the each photodiode is greater than the threshold intensity value.
18. The eye tracking system of claim 17 , wherein the threshold intensity value is based on a statistical distribution of expected threshold intensities.
19. The eye tracking system of claim 13 , wherein the controller is configured to generate the image for the first time period using the event matrix and the change matrix, based in part on solving for an intensity matrix using the equation:
Ax=y
where A is the event matrix,
y is the change matrix, and
x is the intensity matrix.
20. A non-transitory computer-readable storage medium storing executable computer program instructions, the instructions executable to perform steps comprising:
populating an event matrix based in part on:
data values asynchronously received from an event sensor comprising a plurality of photodiodes, the event sensor configured to operate in a mode of a plurality of modes, the plurality of modes including an event mode that causes the plurality of photodiodes to asynchronously output data values, wherein a data value output by each respective photodiode, of the plurality of photodiodes, is based at least in part on a difference of a value previously output by the photodiode and an intensity value captured by the photodiode relative to a threshold intensity value, and
positions of photodiodes of the plurality of photodiodes associated with the received data values over a first time period;
populating a change matrix based in part on the threshold intensity value and the photodiodes associated with the received data values over the first time period;
generating an image of an eye of a user for the first time period using the event matrix and the change matrix; and
determining eye tracking information using the generated image.