Predictive eyetracking using recurrent neural networks

1. A system comprising:

a head mounted display (HMD) comprising:

an electronic display that outputs gaze contingent content in accordance with display instructions;

a plurality of cameras configured to capture images of a first eye of a user of the HMD and a second eye of a user of the HMD; and

an optics block configured to direct the gaze contingent content to respective exit pupils of the HMD corresponding to locations of the first eye and the second eye of the user of the HMD;

an eye tracking controller configured to:

determine eye tracking information defining positions of a first eye and a second eye of a user using the captured images;

determine, based on the eye tracking information, velocities of the first eye and the second eye defining changes in the positions of the first eye and the second eye over time;

determine, based on the eye tracking information, a vergence depth of the first and second eyes; and

determine, based on supplying the eye tracking information, the velocities of the first eye and the second eye, and the vergence depth of the first and second eyes to a neural network model, predicted eye tracking information defining predicted positions of the first eye and the second eye, wherein the neural network model defines an algorithmic relationship between the predicted eye tracking information and (1) the eye tracking information, (2) the velocities of the first and second eyes, and (3) the vergence depth of the first and second eyes; and

a console configured to:

generate the display instructions using the predicted eye tracking information, the display instructions including content for presentation by the display; and

provide the display instructions to the HMD.

2. The system of claim 1 , wherein the neural network model includes a recurrent neural network model.

3. The system of claim 1 , wherein the neural network model defines an algorithmic relationship between the eye tracking information at first timesteps and the predicted eye tracking information at a second timestep subsequent to the first timesteps.

4. The system of claim 1 , wherein the eye tracking controller is further configured to determine an error value associated with the predicted eye tracking information.

5. The system of claim 1 , wherein the eye tracking controller is further configured to:

determine training data sets defining training eye tracking information including inputs and outputs of the neural network model;

determine, based on supplying the inputs of the training data sets to the neural network model, predicted training eye tracking information;

determine a cost function associated with the outputs of the training eye tracking information and the predicted training eye tracking information; and

update the neural network model based on minimizing the cost function.

6. The system of claim 5 , wherein the training eye tracking information of the training data sets include data loss intervals.

7. The system of claim 1 , wherein the eye tracking controller is further configured to:

determine, based on the eye tracking information, eye velocities over time;

determine a velocity threshold;

determine a motion time interval when the eye velocities over time satisfy the velocity threshold, the motion time interval defining an end period of a saccade;

determine a fixation time interval when the eye velocities over time fail to satisfy the velocity threshold, the fixation time interval defining an initial period prior to the end period;

determine a motion sample set including first eye tracking information and first predicted eye tracking information associated with the motion time interval;

determine a fixation sample set including second eye tracking information and second predicted eye tracking information associated with the fixation time interval;

determine first error values associated with the motion sample set using an error function;

determine second error values associated with the fixation sample set using the error function;

overweight the first error values relative to second error value; and

update the neural network model based on minimizing the error function using the overweighted first error values and the second error values.

8. The system of claim 7 , wherein the eye tracking controller is further configured to:

determine an earlier portion and a later portion of a saccade within the motion time interval; and

overweight a first portion of the first error values associated with the earlier portion of the saccade relative to a second portion of the first error values associated with the later portion of the saccade.

9. The system of claim 1 , wherein:

the eye tracking controller is further configured to determine the predicted eye tracking information based on supplying at least one of user head motion, user pupil size, or eye position estimation uncertainty to the neural network model; and

the neural network model defines an algorithmic relationship between the predicted eye tracking information and the least one of user head motion, user pupil size, or eye position estimation uncertainty.

10. The system of claim 1 , wherein the eye tracking controller is further configured to determine a data validity flag associated with the eye tracking information prior to applying the eye tracking information to the neural network model.

11. The system of claim 1 , wherein the eye tracking controller is further configured to store eye tracking information from a prior timestep and a hidden state of the neural network model from the prior timestep to facilitate determination of the predicted eye tracking information.

12. The system of claim 1 , wherein the eye tracking controller is further configured to:

determine a maximum acceleration;

determine a minimum vergence depth and a maximum vergence depth;

determine whether the predicted eye tracking information satisfies the maximum acceleration, the minimum vergence depth, and the maximum vergence depth;

in response to determining that the predicted eye tracking information fails to satisfy the maximum acceleration, the minimum vergence depth, or maximum vergence depth, update the predicted eye tracking information such that the maximum acceleration, the minimum vergence depth, and the maximum vergence depth are satisfied; and

provide the predicted eye tracking information to the console subsequent to updating the predicted eye tracking information.

13. The system of claim 1 , wherein:

the plurality of cameras includes a first camera configured to capture first image data frames of the first eye and a second camera configured to capture second image data frames of the second eye;

the first camera and the second camera operate using asynchronous threads; and

the eye tracking controller is further configured to synchronize the first image data frames and the second image data frames based on timestamps.

14. The system of claim 1 , wherein the eye tracking controller is located in a front rigid body of the HMD.

15. A system comprising:

a head mounted display (HMD) comprising:

an electronic display that outputs gaze contingent content in accordance with display instructions;

a plurality of cameras configured to capture images of a first eye of a user of the HMD and a second eye of a user of the HMD; and

an optics block configured to direct the gaze contingent content to respective exit pupils of the HMD corresponding to locations of the first eye and the second eye of the user of the HMD;

a console comprising:

an eye tracking controller configured to:

determine eye tracking information defining positions of a first eye and a second eye of a user using the captured images;

determine, based on the eye tracking information, velocities of the first eye and the second eye defining changes in the positions of the first eye and the second eye over time;

determine, based on the eye tracking information, a vergence depth of the first and second eyes; and

determine, based on supplying the eye tracking information, the velocities of the first eye and the second eye, and the vergence depth of the first and second eyes to a neural network model, predicted eye tracking information defining predicted positions of the first eye and the second eye, wherein the neural network model defines an algorithmic relationship between the predicted eye tracking information and (1) the eye tracking information, (2) the velocities of the first and second eyes, and (3) the vergence depth of the first and second eyes; and

a rendering module configured to:

generate the display instructions using the predicted eye tracking information, the display instructions including content for presentation by the display; and

provide the display instructions to the HMD.