Deep predictor recurrent neural network for head pose prediction
Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.
1. A wearable system comprising:
a display configured to be disposed before an eye of a wearer of the wearable system, the display configured to display virtual content to the wearer of the wearable system;
non-transitory memory configured to store a head pose neural network; and
a hardware processor programmed to:
determine particular virtual content configured for display at a head pose of the wearer predicted for an upcoming time, the head pose being predicted based on sensor information associated with the display and the head pose prediction neural network; and
cause rendering of the particular virtual content, wherein rendering is configured to be completed when the display is associated with the head pose, and wherein the upcoming time is a time associated with rendering latency.
2. The system of claim 1 , wherein the sensor information is obtained via an inertial measurement unit.
3. The system of claim 1 , wherein the head pose prediction network is a neural network comprising a recurrent neural network.
4. The system of claim 3 , wherein the recurrent neural network is connected to a fully connected layer.
5. The system of claim 4 , wherein the fully connected layer outputs a 3 degree-of-freedom (DOF) pose or a 6 degree-of-freedom (DOF) pose.
6. The system of claim 3 , wherein the recurrent neural network comprises a stack of one or more long short term memory cells or a stack of one or more gated recurrent units.
7. The system of claim 1 , wherein the sensor data is filtered using an extended Kalman filter.
8. The system of claim 1 , wherein the head pose comprises an orientation of the head of the user, and wherein rendering is configured to be completed when the display is associated with the orientation.
9. A method implemented by a wearable system comprising a display configured to be disposed before an eye of a wearer of the wearable system, the display configured to display virtual content to the wearer of the wearable system, wherein the method comprises:
determine particular virtual content configured for display at a head pose of the wearer predicted for an upcoming time, the head pose being predicted based on sensor information associated with the display and the head pose prediction neural network; and
cause rendering of the particular virtual content, wherein rendering is configured to be completed when the display is associated with the head pose, and wherein the upcoming time is a time associated with rendering latency.
10. The method of claim 9 , wherein the sensor information is obtained via an inertial measurement unit.
11. The method of claim 9 , wherein the head pose prediction network is a neural network comprising a recurrent neural network.
12. The method of claim 11 , wherein the recurrent neural network is connected to a fully connected layer.
13. The method of claim 12 , wherein the fully connected layer outputs a 3 degree-of-freedom (DOF) pose or a 6 degree-of-freedom (DOF) pose.
14. The method of claim 11 , wherein the recurrent neural network comprises a stack of one or more long short term memory cells or a stack of one or more gated recurrent units.
15. The method of claim 9 , wherein the sensor data is filtered using an extended Kalman filter.
16. The method of claim 9 , wherein the head pose comprises an orientation of the head of the user, and wherein rendering is configured to be completed when the display is associated with the orientation.