Visual tracking of peripheral devices
Techniques are disclosed for performing localization of a handheld device with respect to a wearable device. At least one sensor mounted to the handheld device, such as an inertial measurement unit (IMU), may obtain handheld data indicative of movement of the handheld device with respect to the world. An imaging device mounted to either the handheld device or the wearable device may capture a fiducial image containing a number of fiducials affixed to the other device. The number of fiducials contained in the image are determined. Based on the number of fiducials, at least one of a position and an orientation of the handheld device with respect to the wearable device are updated based on the image and the handheld data in accordance with a first operating state, a second operating state, or a third operating state.
1. A method of performing localization of a handheld device with respect to a wearable device, the method comprising:
obtaining, by a wearable imaging device mounted on the wearable device, fiducial data indicative of movement of the handheld device, wherein obtaining the fiducial data includes capturing, by the wearable imaging device, a fiducial image containing a number of light-emitting fiducials affixed to the handheld device, wherein the number of light-emitting fiducials is sufficient to determine a position and an orientation of the handheld device in full six degrees of freedom based solely on the fiducial data;
capturing, by a sensor on the handheld device, handheld data;
capturing, by a sensor on the wearable device, wearable device data indicative of movement of the wearable device; and
updating the position and the orientation of the handheld device using the wearable device data, the fiducial data, and the handheld data.
2. The method of claim 1 wherein the sensor on the handheld device comprises an inertial measurement unit (IMU).
3. The method of claim 1 wherein updating the position and the orientation of the handheld device is in accordance with a first operating state or a second operating state.
4. The method of claim 3 , further comprising:
determining the number of light-emitting fiducials contained in the fiducial image;
in response to determining that the number of light-emitting fiducials is equal to or greater than three, updating at least one of the position and the orientation of the handheld device based on the fiducial data in accordance with the first operating state; and
in response to determining that the number of light-emitting fiducials is equal to one or two, updating at least one of the position and the orientation of the handheld device based on the fiducial data and the handheld data in accordance with the second operating state.
5. The method of claim 4 , further comprising in response to determining that the number of light-emitting fiducials is equal to zero, updating at least one of the position and the orientation of the handheld device based on the handheld data in accordance with a third operating state.
6. The method of claim 5 wherein:
at least one of the position and the orientation of the handheld device is updated based solely on the fiducial data in accordance with the first operating state; and
at least one of the position and the orientation of the handheld device is updated based solely on the handheld data in accordance with the third operating state.
7. The method of claim 1 wherein the number of light-emitting fiducials comprises a number of light-emitted diodes (LEDs).
8. The method of claim 1 wherein:
the sensor on the handheld device comprises a second imaging device; and
obtaining the handheld data includes capturing, by the second imaging device, a world image containing one or more features surrounding the handheld device.
9. An augmented reality (AR) system, the system comprising:
a wearable device comprising an imaging device and a wearable sensor mounted on the wearable device, wherein the wearable sensor is operable to capture wearable device data indicative of movement of the wearable device;
a handheld device comprising a handheld sensor and handheld light-emitting fiducials affixed to the handheld device, wherein:
the imaging device is operable to capture a fiducial image containing a number of the handheld light-emitting fiducials, wherein the number of light-emitting fiducials is sufficient to determine a position and an orientation of the handheld device in full six degrees of freedom based solely on the fiducial data, wherein the handheld sensor is operable to capture handheld data; and
a computing apparatus configured to perform localization of the handheld device with respect to the wearable device using the wearable device data, the fiducial image, and the handheld data.
10. The AR system of claim 9 wherein the wearable sensor comprises an inertial measurement unit (IMU).
11. The AR system of claim 9 wherein the handheld sensor comprises an inertial measurement unit (IMU).
12. The AR system of claim 9 wherein the handheld sensor comprises a second imaging device.
13. The AR system of claim 12 , wherein the second imaging device is operable to capture a world image containing one or more surrounding features.
14. The AR system of claim 9 , wherein the handheld device further comprises a second handheld sensor, wherein the handheld sensor comprises an inertial measurement unit (IMU) and the second handheld sensor comprises a second imaging device.
15. The AR system of claim 9 , further comprising a belt pack, wherein the computing apparatus is disposed in the belt pack.
16. The AR system of claim 9 , wherein the computing apparatus is disposed in the wearable device or the handheld device.
17. The AR system of claim 9 wherein the handheld light-emitting fiducials comprises a number of light-emitting diodes (LEDs).