Pose prediction with recurrent neural networks

1. A computer-implemented method, the method comprising:

receiving a request for a head pose prediction for a virtual reality experience;

identifying data features including at least one positional indicator and at least one rotational indicator associated with the virtual reality experience;

identifying historical data features including at least one positional information and at least one rotational information associated with head poses collected over a historical time period;

providing the at least one positional indicator and the at least one rotational indicator to a Recurrent Neural Network (RNN) comprising a plurality of cells, the RNN including a plurality of recurrent steps that each include at least one of the plurality of cells and at least one fully connected (FC) layer, at least some of the plurality of cells being associated with a historical time period;

using the RNN and the historical data features to generate at least one pose prediction corresponding to head pose changes for the virtual reality experience for at least one upcoming time period; and

providing the at least one pose prediction responsive to the request and triggering display of virtual reality content in the virtual reality experience based on the at least one pose prediction.

2. The method of claim 1 , wherein the at least one positional indicator is a three-dimensional head position vector and the at least one rotational indicator is a four-dimensional quaternion.

3. The method of claim 1 , wherein the at least one rotational indicator comprises:

a yaw, a pitch, and a roll; or

a three-dimensional vector having a magnitude that represents an amount of rotation, and a direction that represents an axis of rotation.

4. The method of claim 1 , wherein the RNN is configured to:

generate additional candidate predictions based on the at least one pose prediction; and

determine a mean square error for each of the additional candidate predictions, the mean square error indicating whether to discard a respective additional candidate prediction.

5. The method of claim 1 , wherein the RNN is configured to recursively determine additional candidate predictions for head pose changes at further upcoming time periods.

6. The method of claim 1 , wherein:

receiving the request for a head pose prediction includes receipt of historical head pose data from a client device over a network; and

providing the at least one pose prediction for head pose changes includes providing rendered content in the virtual reality experience to the client device over the network based on the at least one pose prediction for head pose changes.

7. A computer implemented method for predicting poses in an augmented reality environment, the method comprising:

obtaining historical pose data corresponding to user movements in the augmented reality environment collected over a historical time period;

generating a historical vector sequence including pose features determined from the historical pose data;

normalizing the historical vector sequence;

determining, using the normalized historical vector sequence executing a Recurrent Neural Network (RNN), a pose prediction for an upcoming time period, the RNN including a plurality of long short-term memory (LSTM) cells and at least one fully connected neural network layer; and

recursively generating, using the RNN and the normalized historical vector, a plurality of additional pose predictions for subsequent time periods in response to receiving the pose prediction and a state of at least one of the plurality of LSTM cells.

8. The method of claim 7 , wherein the historical pose data corresponding to user movements in the augmented reality environment includes velocity or acceleration measurements associated with the user movements.

9. The method of claim 7 , further comprising:

determining locations in which to display augmented reality content based on the additional pose predictions at corresponding time periods beyond the upcoming time period; and

triggering rendering of the augmented reality content for display in the augmented reality environment.

10. The method of claim 7 , wherein:

the historical vector sequence represents input to the RNN that is normalized based at least in part on a calculated mean value and a calculated variance value; and

the pose prediction and the plurality of additional pose predictions are denormalized based on the calculated mean value and the calculated variance value before determining locations in which to display augmented reality content.

11. The method of claim 7 , wherein:

a state of each LSTM cell in the plurality of LSTM cells is provided as input to the RNN with a next sequential pose prediction; and

the plurality of additional pose predictions are based at least in part on the state of each LSTM cell in the plurality of LSTM cells.

12. The method of claim 7 , wherein:

the RNN is trained on a database of known pose sequences; and

the historical pose data is sequential pose data associated with a mobile device executing an augmented reality application.

13. The method of claim 7 , wherein the RNN is configured to determine an error function that defines a mean absolute error, a smoothness cost, and a pose change cost for each of the plurality of additional pose predictions.

14. The method of claim 13 , wherein:

the error function is determined based on detected angular error or detected eye location error associated with one or more of the plurality of additional pose predictions; or

the error function represents a weighted error based on a portion of the plurality of the additional pose predictions.

15. A computer program product tangibly embodied on a non-transitory computer-readable medium and comprising instructions that, when executed, are configured to cause at least one processor to:

receive a request for a pose prediction for an augmented reality experience;

identifying data features including at least one positional indicator and at least one rotational indicator associated with the augmented reality experience;

identifying historical data features including at least one positional information and at least one rotational information associated with head noses collected over a historical time period;

provide the at least one positional indicator and the at least one rotational indicator to a Recurrent Neural Network (RNN) comprising a plurality of cells, the RNN including a plurality of recurrent steps that each include at least one of the plurality of cells and at least one fully connected (FC) layer, at least some of the plurality of cells being associated with a historical time period;

use the RNN and the historical data features to generate at least one pose prediction corresponding to pose changes for the augmented reality experience for at least one upcoming time period; and

provide the at least one pose prediction responsive to the request and trigger display of augmented reality content in the augmented reality experience based on the at least one pose prediction.

16. The computer program product of claim 15 , wherein the RNN is configured to:

generate additional candidate predictions based on the at least one prediction; and

determine a mean square error for each of the additional candidate predictions, the mean square error indicating whether to discard a respective additional pose prediction.

17. The computer program product of claim 15 , wherein the at least one rotational indicator comprises a three-dimensional vector having a magnitude that represents an amount of rotation, and a direction that represents an axis of rotation.

18. The computer program product of claim 15 , wherein the at least one positional indicator is a three-dimensional position vector and the at least one rotational indicator is a four-dimensional quaternion.

19. The computer program product of claim 15 , wherein the plurality of cells are long short-term memory (LSTM) cells and the RNN is configured to recursively determine additional predictions for pose changes at further upcoming time periods.

20. The computer program product of claim 19 , wherein the RNN is configured to encode as input, and for each of a plurality of timesteps within the upcoming time period, a state for a respective LSTM cell, in the plurality of LSTM cells, corresponding to a respective timestep in the upcoming time period.