JOINT INFRARED AND VISIBLE LIGHT VISUAL-INERTIAL OBJECT TRACKING
In one embodiment, a method for tracking includes receiving motion data captured by a motion sensor of a wearable device, generating a pose of the wearable device based on the motion data, capturing a first frame of the wearable device by a camera using a first exposure time, identifying, in the first frame, a pattern of lights disposed on the wearable device, capturing a second frame of the wearable device by the camera using a second exposure time, identifying, in the second frame, predetermined features of the wearable device, and adjusting the pose of the wearable device in the environment based on the identified pattern of light in the first frame or the identified predetermined features in the second frame. The method utilizes the predetermined features for tracking the wearable device in a visible-light frame under specific light conditions to improve the accuracy of the pose of controller.
1 . A method comprising, by a computing system:
receiving motion data captured by one or more motion sensors of a wearable device;
generating a pose of the wearable device based on the motion data;
capturing a first frame of the wearable device by a camera using a first exposure time;
identifying, in the first frame, a pattern of lights disposed on the wearable device;
capturing a second frame of the wearable device by the camera using a second exposure time;
identifying, in the second frame, predetermined features of the wearable device; and
adjusting the pose of the wearable device in an environment based on at least one of (1) the identified pattern of lights in the first frame or (2) the identified predetermined features in the second frame.
2 . The method of claim 1 , wherein:
capturing the first frame of the wearable device using the first exposure time when the environment has a first light condition; and
capturing the second frame of the wearable device using the second exposure time when the environment has a second light condition.
3 . The method of claim 2 , wherein the second light condition comprises one or more of:
an environment having bright light;
an environment having a light source to interfere the pattern of lights of the wearable device; and
the camera not being able to capture the pattern of lights.
4 . The method of claim 1 , wherein:
the wearable device is equipped with one or more inertial measurement units (IMUs) and one or more infrared (IR) light emitting diodes (LEDs);
the first frame is an IR image; and
the second frame is a visible-light image.
5 . The method of claim 1 , wherein the second exposure time is longer than the first exposure time.
6 . The method of claim 1 , wherein the pose of the wearable device is generated at a faster frequency than a frequency that the first frame and the second frame are captured.
7 . The method of claim 1 , further comprising:
capturing a third frame of the wearable device by the camera using the second exposure time;
identifying, in the third frame, one or more features corresponding to the predetermined features of the wearable device;
determining correspondence data between the predetermined features and the one or more features; and
tracking the wearable device in the environment based on the correspondence data.
8 . The method of claim 1 , wherein the computing system comprises:
the camera configured to capture the first frame and the second frame of the wearable device;
an identifying unit configured to identify the pattern of lights and the predetermined features of the wearable device; and
a filter unit configured to adjust the pose of the wearable device.
9 . The method of claim 1 ,
wherein the camera is located within a head-mounted device; and
wherein the wearable device is a controller separated from the head-mounted device.
10 . The method of claim 9 , wherein the head-mounted device comprises one or more processors, wherein the one or more processors are configured to implement the camera, the identifying unit, and the filter unit.
11 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive motion data captured by one or more motion sensors of a wearable device;
generate a pose of the wearable device based on the motion data;
capture a first frame of the wearable device by a camera using a first exposure time;
identify, in the first frame, a pattern of lights disposed on the wearable device;
capture a second frame of the wearable device by the camera using a second exposure time;
identify, in the second frame, predetermined features of the wearable device; and
adjust the pose of the wearable device in an environment based on at least one of (1) the identified pattern of lights in the first frame or (2) the identified predetermined features in the second frame.
12 . The media of claim 11 , wherein the software is further operable when executed to:
capture the first frame of the wearable device using the first exposure time when the environment has a first light condition; and
capture the second frame of the wearable device using the second exposure time when the environment has a second light condition.
13 . The media of claim 12 , wherein the second light condition comprises one or more of:
an environment having bright light;
an environment having a light source to interfere the pattern of lights of the wearable device; and
the camera not being able to capture the pattern of lights.
14 . The media of claim 11 , wherein:
the wearable device is equipped with one or more inertial measurement units (IMUs) and one or more infrared (IR) light emitting diodes (LEDs);
the first frame is an IR image; and
the second frame is a visible-light image.
15 . The media of claim 11 , wherein the second exposure time is longer than the first exposure time.
16 . The media of claim 11 , wherein the pose of wearable device is generated at a faster frequency than a frequency that the first frame and the second frame are captured.
17 . The media of claim 11 , wherein the software is further operable when executed to:
capture a third frame of the wearable device by the camera using the second exposure time;
identify, in the third frame, one or more features corresponding to the predetermined features of the wearable device;
determine correspondence data between the predetermined features and the one or more features; and
track the wearable device in the environment based on the adjusted pose and the correspondence data.
18 . The media of claim 11 ,
wherein the camera is located within a head-mounted device; and
wherein the wearable device is a remote controller separated from the head-mounted device.
19 . A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to:
receive motion data captured by one or more motion sensors of a wearable device;
generate a pose of the wearable device based on the motion data;
capture a first frame of the wearable device by a camera using a first exposure time;
identify, in the first frame, a pattern of lights disposed on the wearable device;
capture a second frame of the wearable device by the camera using a second exposure time;
identify, in the second frame, predetermined features of the wearable device; and
adjust the pose of the wearable device in an environment based on at least one of (1) the identified pattern of lights in the first frame or (2) the identified predetermined features in the second frame.
20 . The system according to claim 19 , wherein the instructions further cause the system to:
capture the first frame of the wearable device using the first exposure time when the environment has a first light condition; and
capture the second frame of the wearable device using the second exposure time when the environment has a second light condition.