Deep inertial prediction system and methods
A system configured to determine poses of a tracked device in a physical environment and to utilize the poses as an input to control or manipulate a virtual environment or mixed reality environment. In some cases, the system may include a fall back tracking system for when the main tracking system loses visual tracking of the tracked device.
1. A method comprising:
initializing, based at least in part on data associated with a prior pose of a tracked device, an Extended Kalman Filter of a deep inertial prediction system;
inputting a window of inertial measurement unit (IMU) measurements into a machine learned network, the IMU measurements associated with the tracked device;
receiving, by the Extended Kalman Filter and from the machine learned network, a linear velocity and uncertainty data associated with the linear velocity; and
determining, by the Extended Kalman Filter, a current pose of the tracked device based at least in part on the linear velocity and the uncertainty data.
2. The method as recited in any of claim 1 , further comprising:
responsive to determining that a pose output by a main tracking system of the extended reality system has a confidence score below a threshold confidence value, engaging the deep inertial prediction system.
3. The method as recited in claim 2 , wherein determining that the pose output by the main tracking system has a confidence score below the threshold confidence value further comprises determining that the tracked device is no longer represented in the image data being generated by an image device associated with the extended reality system.
4. The method as recited in claim 1 , wherein the data associated with a prior pose comprises a status of a main tracking system.
5. The method as recited in claim 1 , wherein the machine learned network is a deep neural network having a first set of fully connected layers to output the linear velocity and a second set of fully connected layers to output uncertainty data associated with the linear velocities.
6. The method as recited in claim 1 , further comprising:
determining, by the Extended Kalman Filter, bias data, gravity data, and yaw data associated with the tracked device;
generating, by an IMU data transformation engine, transformed IMU data based at least in part on the IMU data and one or more of the bias data, the gravity data, or the yaw data; and
wherein the window of IMU measurements input into the machine learned network includes the transformed IMU data.
7. The method as recited in claim 6 , wherein the bias data includes Extended Kalman Filter biases and the gravity data includes Extended Kalman Filter gravity rotation and normals.
8. A system comprising:
a display for presenting an extended reality scene to a user;
one or more image devices for capturing image data associated with a tracked device;
one or more communication interfaces for receiving inertial measurement unit (IMU) measurements from the tracked device;
one or more processors;
non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising:
initializing an Extended Kalman Filter based at least in part on data associated with a prior pose of the tracked device determined by a second tracking component of the system, the second tracking component different than the first tracking component;
inputting the IMU measurements into a machine learned network of a first tracking component;
receiving, from the machine learned network, a linear velocity associated with the tracked device and uncertainty data associated with the linear velocity; and
determining a current pose of the tracked device based at least in part on the linear velocity and the uncertainty data.
9. The system as recited in claim 8 , wherein the operations further comprise:
responsive to determining that a pose output by a second tracking component of the system has a confidence score below a threshold confidence value, engaging the first tracking system.
10. The system as recited in claim 8 , wherein the machine learned network is a deep neural network having a first set of fully connected layers to output the linear velocity and a second set of fully connected layers to output uncertainty data associated with the linear velocities.
11. The system as recited in claim 8 , wherein the operations further comprise:
determining bias data, gravity data, and yaw data associated with the tracked device;
generating transformed IMU measurements based at least in part on the IMU measurements and one or more of the bias data, the gravity data, or the yaw data; and
wherein inputting the IMU measurements into the machine learned network comprises inputting the transformed IMU data into the machine learned network.
12. The system as recited in claim 11 , wherein the bias data includes Extended Kalman Filter biases and the gravity data includes Extended Kalman Filter gravity rotation and normals.
13. The system as recited in claim 8 , wherein the machine learned model is at least partially trained using synthetic ground truth poses generated based at least in part on historical trajectories of tracked devices, the synthetic ground truth poses adjusted based at least in part by applying at least one of white Gaussian IMU noise, IMU biases, or IMU intrinsic.
14. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
responsive to determining that a pose output by a main tracking system of the extended reality system has a confidence score below a threshold confidence value:
initializing, based at least in part on data associated with a prior pose of a tracked device, an Extended Kalman Filter of a deep inertial prediction system;
inputting a window of inertial measurement unit (IMU) measurements into a machine learned network, the IMU measurements associated with the tracked device;
receiving, by the Extended Kalman Filter and from the machine learned network, a linear velocity and uncertainty data associated with the linear velocity; and
determining, by the Extended Kalman Filter, a current pose of the tracked device based at least in part on the linear velocity and the uncertainty data.
15. The one or more non-transitory computer-readable media of claim 14 , wherein determining that the pose output by the main tracking system has a confidence score below the threshold confidence value further comprises determining that the tracked device is no longer represented in the image data being generated by an image device associated with the extended reality system.
16. The one or more non-transitory computer-readable media of claim 14 , wherein the data associated with a prior pose comprises a status of a main tracking system.
17. The one or more non-transitory computer-readable media of claim 14 , wherein the machine learned network is a deep neural network having a first set of fully connected layers to output the linear velocity and a second set of fully connected layers to output uncertainty data associated with the linear velocities.
18. The one or more non-transitory computer-readable media of claim 14 , the operations further comprising:
determining, by the Extended Kalman Filter, bias data, gravity data, and yaw data associated with the tracked device;
generating, by an IMU data transformation engine, transformed IMU data based at least in part on the IMU data and one or more of the bias data, the gravity data, or the yaw data; and
wherein the window of IMU measurements input into the machine learned network includes the transformed IMU data.
19. The one or more non-transitory computer-readable media of claim 14 , wherein determining, by the Extended Kalman Filter, the current pose of the tracked device is based at least in part on a last valid pose from the main tracking system.
20. The one or more non-transitory computer-readable media of claim 14 , wherein the threshold confidence value is a threshold drift in the pose estimate.