Systems and methods for object tracking using fused data

1. A method comprising, by a computing system:

capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;

identifying one or more features from the first image to estimate a pose of the hand of the user;

estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;

receiving first inertial measurement unit (IMU) data of the controller;

estimating a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller;

receiving second IMU data of the controller;

calculating an IMU-predicted pose of the controller based on the second IMU data;

estimating an IMU-predicted pose of the hand by applying an inverse of the estimated grip to the IMU-predicted pose of the controller; and

determining, based on the estimated IMU-predicted pose of the hand, a search range of the hand when capturing a second image depicting at least a part of the hand of the user.

2. The method of claim 1 , wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.

3. The method of claim 1 , further comprising:

receiving IMU data of the wearable device to estimate a pose of the wearable device; and

updating the pose of the hand of the user based on the pose of the wearable device.

4. The method of claim 3 , wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.

5. The method of claim 1 , further comprising:

estimating a second pose of the hand based on the IMU-predicted pose of the hand.

6. The method of claim 5 , further comprising:

capturing the second image using the one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment;

identifying the one or more features from the second image; and

estimating a third pose of the hand based on the one or more features identified from the second image of the user.

7. The method of claim 6 , wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.

8. The method of claim 1 , wherein the wearable device comprises:

the one or more cameras configured to capture images;

a hand-tracking unit configured to estimate the pose of the hand of the user; and

a controller-tracking unit configured to estimate the second pose of the controller.

9. The method of claim 1 , wherein estimating the second pose of the controller comprises:

estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment;

adjusting the pose of the controller relative to the environment based on the IMU data of the controller;

estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and

estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;

identify one or more features from the first image to estimate a pose of the hand of the user;

estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;

receive first inertial measurement unit (IMU) data of the controller;

estimate a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller;

receive second IMU data of the controller;

calculate an IMU-predicted pose of the controller based on the second IMU data;

estimate an IMU-predicted pose of the hand by applying an inverse of the estimated grip to the IMU-predicted pose of the controller; and

determine, based on the estimated IMU-predicted pose of the hand, a search range of the hand when capturing a second image depicting at least a part of the hand of the user.

11. The media of claim 10 , wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.

12. The media of claim 10 , wherein the software is further operable when executed to:

receive IMU data of the wearable device to estimate a pose of the wearable device; and

update the pose of the hand of the user based on the pose of the wearable device.

13. The media of claim 12 , wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.

14. The media of claim 10 , wherein the software is further operable when executed to:

estimate a second pose of the hand based on the IMU-predicted pose of the hand.

15. The media of claim 14 , wherein the software is further operable when executed to:

capture the second image using the one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment;

identify the one or more features from the second image; and

estimate a third pose of the hand based on the one or more features identified from the second image of the user.

16. The media of claim 15 , wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.

17. The media of claim 10 , wherein the wearable device comprises:

the one or more cameras configured to capture images;

a hand-tracking unit configured to estimate the pose of the hand of the user; and

a controller-tracking unit configured to estimate the second pose of the controller.

18. The media of claim 10 , wherein estimating the second pose of the controller comprises:

estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment;

adjusting the pose of the controller relative to the environment based on the IMU data of the controller;

estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and

estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

19. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to:

capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;

identify one or more features from the first image to estimate a pose of the hand of the user;

estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;

receive first inertial measurement unit (IMU) data of the controller;

estimate a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller;

receive second IMU data of the controller;

calculate an IMU-predicted pose of the controller based on the second IMU data;

estimate an IMU-predicted pose of the hand by applying an inverse of the estimated grip to the IMU-predicted pose of the controller; and

determine, based on the estimated IMU-predicted pose of the hand, a search range of the hand when capturing a second image depicting at least a part of the hand of the user.

20. The system of claim 19 , wherein the instructions are further operable when executed to:

receive IMU data of the wearable device to estimate a pose of the wearable device; and

update the pose of the hand of the user based on the pose of the wearable device.