Dynamic graphics rendering based on predicted saccade landing point

1. A method for predicting eye movement in a head mounted display (HMD), comprising:

tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points;

determining velocity of the movement based on the tracking;

determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity; and

predicting a landing point on a display of the HMD corresponding to a direction of the eye for the saccade;

rendering for display a first video frame having a foveal region centered about the landing point on the display; and

presenting the first video frame having the foveal region on the display of the HMD, wherein the eye is predicted to be oriented towards the landing point upon displaying the first video frame.

2. The method of claim 1 , further comprising:

terminating rendering of at least one video frame during the saccade that is to be rendered before the first video frame.

3. The method of claim 1 , further comprising:

rendering at low resolution at least one video frame during the saccade that is to be rendered before the first video frame.

4. The method of claim 1 , wherein the predicting the landing point includes:

at a first sample point, predicting a first landing point based on first eye orientation data of a first set of sample points including the first sample point and at least one previous sample point, wherein the first eye orientation data includes a plurality of directions of the eys with respect to the HMD; and

at a second sample point following the first sample point in the saccade, updating the landing point by predicting a second landing point based on second eye orientation data of a second set of sample points including the second sample point and at least one previous sample point.

5. The method of claim 1 , wherein the predicting the landing point includes:

collecting first eye orientation data when tracking the movement of the eye for a set of sample points, the first eye orientation data including a plurality of directions of the eye with respect to the HMD, wherein at least one sample point of the set of sample points occurs during the saccade;

providing as input the first eye orientation data for the set of sample points to a recurrent neural network trained on second eye orientation data of a plurality of saccades of test subjects;

comparing in the recurrent neural network a segment of an eye velocity graph built from the first eye orientation data for the set of sample points against a plurality of eye velocity graphs built from the plurality of saccades of test subjects; and

determining in the recurrent neural network the landing point.

6. The method of claim 5 , wherein the landing point occurs at an end of the saccade and corresponds to a fixation direction of the eye.

7. The method of claim 5 , wherein the landing point occurs at a mid-point of the saccade and corresponds to an intermediate direction of the eye.

8. The method of claim 5 , wherein the recurrent neural network includes at least one of:

a long-short-term-memory neural network; and

a fully connected multilayer perceptron network.

9. The method of claim 1 , wherein the determining velocity of the movement includes:

determining a velocity of the eye based on a first eye direction and a second eye direction from two sample points.

10. A computer system comprising:

a processor; and

memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for predicting eye movement in a head mounted display (HMD) comprising:

tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points;

determining velocity of the movement based on the tracking;

determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity; and

predicting a landing point on a display of the HMD corresponding to a direction of the eye for the saccade;

rendering for display a first video frame having a foveal region centered about the landing point on the display; and

presenting the first video frame having the foveal region on the display of the HMD, wherein the eye is predicted to be oriented towards the landing point upon displaying the first video frame.

11. The computer system of claim 10 , the method further comprising:

terminating rendering of at least one video frame during the saccade that is to be rendered before the first video frame.

12. The computer system of claim 10 , the method further comprising:

rendering at low resolution at least one video frame during the saccade that is to be rendered before the first video frame.

13. The computer system of claim 10 , wherein in the method the predicting the landing point includes:

at a first sample point, predicting a first landing point based on first eye orientation data of a first set of sample points including the first sample point and at least one previous sample point, wherein the first eye orientation data includes a plurality of directions of the eye with respect to the HMD; and

at a second sample point following the first sample point in the saccade, updating the landing point by predicting a second landing point based on second eye orientation data of a second set of sample points including the second sample point and at least one previous sample point.

14. The computer system of claim 10 , wherein in the method the predicting the landing point includes:

collecting first eye orientation data when tracking the movement of the eye for a set of sample points, the first eye orientation data including a plurality of directions of the eye with respect to the HMD, wherein at least one sample point of the set of sample points occurs during the saccade;

providing as input the first eye orientation data for the set of sample points to a recurrent neural network trained on second eye orientation data of a plurality of saccades of test subjects;

comparing in the recurrent neural network a segment of an eye velocity graph built from the first eye orientation data for the set of sample points against a plurality of eye velocity graphs built from the plurality of saccades of test subjects; and

determining in the recurrent neural network the landing point.

15. The computer system of claim 14 , wherein in the method the landing point occurs at an end of the saccade and corresponds to a fixation direction of the eye.

16. The computer system of claim 14 , wherein in the method the recurrent neural network includes at least one of:

a long-short-term-memory neural network; and

a fully connected multilayer perceptron network.

17. A non-transitory computer-readable medium storing a computer program for prediction, the computer-readable medium comprising:

program instructions for tracking movement of an eye of a user with a gaze tracking system disposed in a head mounted display (HMD) at a plurality of sample points;

program instructions for determining velocity of the movement based on the tracking;

program instructions for determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity; and

program instructions for predicting a landing point on a display of the HMD corresponding to a direction of the eye for the saccade;

program instructions for rendering for display a first video frame having a foveal region centered about the landing point on the display; and

program instructions for presenting the first video frame having the foveal region on the display of the HMD, wherein the eye is predicted to be oriented towards the landing point upon displaying the first video frame.

18. The computer-readable medium of claim 17 , further comprising:

terminating rendering of at least one video frame during the saccade that is to be rendered before the first video frame.

19. The computer-readable medium of claim 17 , further comprising:

rendering at low resolution at least one video frame during the saccade that is to be rendered before the first video frame.

20. The computer-readable medium of claim 17 , wherein the predicting the landing point includes:

at a first sample point, predicting a first landing point based on first eye orientation data of a first set of sample points including the first sample point and at least one previous sample point, wherein the first eye orientation data includes a plurality of directions of the eye with respect to the HMD; and

at a second sample point following the first sample point in the saccade, updating the landing point by predicting a second landing point based on second eye orientation data of a second set of sample points including the second sample point and at least one previous sample point.

21. The computer-readable medium of claim 17 , wherein the predicting the landing point includes:

collecting first eye orientation data when tracking the movement of the eye for a set of sample points, the first eye orientation data including a plurality of directions of the eye with respect to the HMD, wherein at least one sample point of the set of sample points occurs during the saccade;

providing as input the first eye orientation data for the set of sample points to a recurrent neural network trained on second eye orientation data of a plurality of saccades of test subjects;

comparing in the recurrent neural network a segment of an eye velocity graph built from the first eye orientation data for the set of sample points against a plurality of eye velocity graphs built from the plurality of saccades of test subjects; and

determining in the recurrent neural network the landing point.

22. The computer-readable medium of claim 21 , wherein the landing point occurs at an end of the saccade and corresponds to a fixation direction of the eye.

23. The computer-readable medium of claim 21 , wherein the recurrent neural network comprises:

a long-short-term-memory neural network; and

a fully connected multilayer perceptron network.

24. A method for predicting eye movement in a head mounted display (HMD), comprising:

tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points;

determining velocity of the movement based on the tracking;

determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity;

predicting a landing point on a display of the HMD corresponding to a direction of the eye for the saccade, wherein the predicting the landing point includes:

at a first sample point, predicting a first landing point based on first eye orientation data of a first set of sample points including the first sample point and at least one previous sample point, wherein the first eye orientation data includes a plurality of directions of the eye with respect to the HMD; and

at a second sample point following the first sample point in the saccade, updating the landing point by predicting a second landing point based on second eye orientation data of a second set of sample points including the second sample point and at least one previous sample point.

25. The method of claim 24 , further comprising:

terminating rendering of at least one video frame during the saccade that is to be rendered before a first video frame having a foveal region centered about the landing point on the display.

26. The method of claim 24 , further comprising:

rendering at low resolution at least one video frame during the saccade that is to be rendered before a first video frame having a foveal region centered about the landing point on the display.

27. A method for predicting eye movement in a head mounted display (HMD), comprising:

tracking movement of an eye of a user with a gaze tracking system disposed in the HMD at a plurality of sample points;

determining velocity of the movement based on the tracking;

determining that the eye of the user is in a saccade upon the velocity reaching a threshold velocity; and

predicting a landing point on a display of the HMD corresponding to a direction of the eye for the saccade, wherein the predicting the landing point includes:

collecting first eye orientation data when tracking the movement of the eye for a set of sample points, the first eye orientation data including a plurality of directions of the eye with respect to the HMD, wherein at least one sample point of the set of sample points occurs during the saccade;

providing as input the first eye orientation data for the set of sample points to a recurrent neural network trained on second eye orientation data of a plurality of saccades of test subjects;

comparing in the recurrent neural network a segment of an eye velocity graph built from the first eye orientation data for the set of sample points against a plurality of eye velocity graphs built from the plurality of saccades of test subjects; and

determining in the recurrent neural network the landing point.

28. The method of claim 27 , further comprising:

terminating rendering of at least one video frame during the saccade that is to be rendered before a first video frame having a foveal region centered about the landing point on the display.

29. The method of claim 27 , further comprising:

rendering at low resolution at least one video frame during the saccade that is to be rendered before the first video frame having a foveal region centered about the landing point on the display.

30. The method of claim 27 , wherein the landing point occurs at an end of the saccade and corresponds to a fixation direction of the eye.

31. The method of claim 27 , wherein the landing point occurs at a mid-point of the saccade and corresponds to an intermediate direction of the eye.