Visual tracking of peripheral devices

1. A method of performing localization of a handheld device with respect to a wearable device, the method comprising:

capturing, by a first imaging device mounted to the handheld device, a fiducial image containing a number of fiducials affixed to the wearable device;

capturing, by a second imaging device mounted to the handheld device, a world image containing one or more features surrounding the handheld device;

obtaining, by a sensor mounted to the handheld device, handheld data indicative of movement of the handheld device;

determining the number of fiducials contained in the fiducial image; and

updating a position and an orientation of the handheld device using at least one of the fiducial image or the world image and the handheld data.

2. The method of claim 1 wherein updating the position and the orientation of the handheld device is based at least in part on the number of fiducials contained in the fiducial image.

3. The method of claim 1 wherein the sensor comprises an inertial measurement unit (IMU).

4. The method of claim 1 wherein updating the position and orientation of the handheld device is in accordance with a first operating state or a second operating state.

5. The method of claim 4 , wherein updating the position and the orientation of the handheld device is based at least in part on the number of fiducials contained in the fiducial image, the method further comprising:

in response to determining that the number of fiducials is equal to or greater than three, updating at least one of the position and the orientation of the handheld device based on the fiducial image in accordance with the first operating state; and

in response to determining that the number of fiducials is equal to one or two, updating at least one of the position and the orientation of the handheld device based on the fiducial image, the world image, and the handheld data in accordance with the second operating state.

6. The method of claim 5 , further comprising in response to determining that the number of fiducials is equal to zero, updating at least one of the position and the orientation of the handheld device based on the world image and the handheld data in accordance with a third operating state.

7. The method of claim 6 wherein:

at least one of the position and the orientation of the handheld device is updated based solely on the fiducial image in accordance with the first operating state; and

at least one of the position and the orientation of the handheld device is updated based solely on the handheld data in accordance with the third operating state.

8. The method of claim 1 wherein the fiducials comprise light-emitting diodes (LEDs).

9. A system for performing localization of a handheld device with respect to a wearable device, the system comprising:

the wearable device;

the handheld device; and

one or more processors communicatively coupled to the wearable device and the handheld device, wherein the one or more processors are configured to perform operations including:

capturing, by a first imaging device mounted to the handheld device, a fiducial image containing a number of fiducials affixed to the wearable device;

capturing, by a second imaging device mounted to the handheld device, a world image containing one or more features surrounding the handheld device;

obtaining, by a sensor mounted to the handheld device, handheld data indicative of movement of the handheld device;

determining the number of fiducials contained in the fiducial image; and

updating a position and an orientation of the handheld device using at least one of the fiducial image or the world image and the handheld data.

10. The system of claim 9 , wherein updating the position and the orientation of the handheld device is based at least in part on the number of fiducials contained in the fiducial image.

11. The system of claim 9 , wherein the sensor comprises an inertial measurement unit (IMU).

12. The system of claim 9 , wherein updating the position and orientation of the handheld device is in accordance with a first operating state or a second operating state.

13. The system of claim 12 , wherein updating the position and the orientation of the handheld device is based at least on part on the number of fiducials contained in the fiducial image, and wherein the operations further comprise:

in response to determining that the number of fiducials is equal to or greater than three, updating at least one of the position and the orientation of the handheld device based on the fiducial image in accordance with the first operating state; and

in response to determining that the number of fiducials is equal to one or two, updating at least one of the position and the orientation of the handheld device based on the fiducial image, the world image, and the handheld data in accordance with the second operating state.

14. The system of claim 12 , further comprising in response to determining that the number of fiducials is equal to zero, updating at least one of the position and the orientation of the handheld device based on the world image and the handheld data in accordance with a third operating state.

15. The system of claim 14 , wherein

at least one of the position and the orientation of the handheld device is updated based solely on the fiducial image in accordance with the first operating state; and

at least one of the position and the orientation of the handheld device is updated based solely on the handheld data in accordance with the third operating state.

16. The system of claim 9 , wherein the fiducials comprise light-emitting diodes (LEDs).