VISUAL TRACKING OF PERIPHERAL DEVICES

1 . An augmented reality (AR) system, the system comprising:

a wearable device comprising:

a display inside the wearable device and operable to display virtual content; and

an imaging device mounted to the wearable device;

a handheld device comprising handheld fiducials affixed to the handheld device; and

a computing apparatus configured to perform localization of the handheld device with respect to the wearable device.

2 . The AR system of claim 1 wherein the wearable device further comprises a wearable sensor mounted on the wearable device, and wherein the wearable sensor is operable to capture wearable device data indicative of movement of the wearable device.

3 . The AR system of claim 2 wherein the wearable sensor comprises an inertial measurement unit (IMU).

4 . The AR system of claim 1 wherein the handheld device further comprises a handheld sensor, and wherein the handheld sensor is operable to capture handheld data.

5 . The AR system of claim 4 wherein the handheld sensor comprises an inertial measurement unit (IMU).

6 . The AR system of claim 1 wherein the handheld device comprises a second imaging device.

7 . The AR system of claim 6 wherein the second imaging device is operable to capture a world image containing one or more surrounding features.

8 . The AR system of claim 1 wherein the imaging device is operable to capture a fiducial image containing a number of the handheld fiducials.

9 . The AR system of claim 1 further comprising a belt pack, wherein the computing apparatus is disposed in the belt pack.

10 . The AR system of claim 1 wherein the computing apparatus is disposed in the wearable device or the handheld device.

11 . A method of performing localization of a handheld device with respect to a wearable device, the method comprising:

obtaining, by a wearable imaging device mounted on the wearable device, fiducial data indicative of movement of the handheld device; and

updating a position and an orientation of the handheld device using the fiducial data.

12 . The method of claim 11 wherein obtaining the fiducial data includes capturing, by the wearable imaging device, a fiducial image containing a number of fiducials affixed to the handheld device.

13 . The method of claim 12 wherein the number of fiducials affixed to the handheld device comprise a number of light-emitting fiducials affixed to the handheld device and the number of light-emitting fiducials is sufficient to determine the position and the orientation of the handheld device in full six degrees of freedom based solely on the fiducial data.

14 . The method of claim 13 wherein the number of light-emitting fiducials comprises a number of light-emitted diodes (LEDs).

15 . The method of claim 11 further comprising:

obtaining, by a sensor on the handheld device, handheld data; and

updating the position and the orientation of the handheld device using the handheld data.

16 . The method of claim 15 wherein the sensor on the handheld device comprises an inertial measurement unit (IMU).

17 . The method of claim 15 wherein:

the sensor on the handheld device comprises a second imaging device; and

obtaining the handheld data includes capturing, by the second imaging device, a world image containing one or more features surrounding the handheld device.

18 . The method of claim 11 further comprising:

capturing, by a sensor on the wearable device, wearable device data indicative of movement of the wearable device; and

updating the position and the orientation of the handheld device using the wearable device data.

19 . The method of claim 18 wherein the sensor on the wearable device comprises an inertial measurement unit (IMU).

20 . The method of claim 11 wherein updating the position and the orientation of the handheld device is in accordance with a first operating state or a second operating state.