WEARABLE SYSTEM WITH HEADSET AND CONTROLLER INSIDE-OUT TRACKING
Wearable systems and method for operation thereof incorporating headset and controller inside-out tracking are disclosed. A wearable system may include a headset and a controller. The wearable system may cause fiducials of the controller to flash. The wearable system may track a pose of the controller by capturing headset images using a headset camera, identifying the fiducials in the headset images, and tracking the pose of the controller based on the identified fiducials in the headset images and based on a pose of the headset. While tracking the pose of the controller, the wearable system may capture controller images using a controller camera. The wearable system may identify two-dimensional feature points in each controller image and determine three-dimensional map points based on the two-dimensional feature points and the pose of the controller.
1 . A method of operating a wearable system comprising a headset and a controller, the method comprising:
causing a set of fiducials of the controller to flash, the set of fiducials being arranged in a known geometry;
tracking a pose of the controller by:
capturing a set of headset images using a headset camera of the headset;
identifying the set of fiducials in the set of headset images; and
tracking the pose of the controller based on the identified set of fiducials in the set of headset images and based on a pose of the headset;
while tracking the pose of the controller, capturing a set of controller images using a controller camera of the controller;
identifying a set of two-dimensional feature points in each controller image of the set of controller images; and
determining a set of three-dimensional map points based on the set of two-dimensional feature points and the pose of the controller.
2 . The method of claim 1 , wherein the wearable system comprises:
the headset comprising:
the headset camera; and
a headset inertial measurement unit; and
the controller comprising:
the set of fiducials arranged in the known geometry;
the controller camera; and
a controller inertial measurement unit;
wherein the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the controller camera, the headset inertial measurement unit, or the controller inertial measurement unit.
3 . The method of claim 1 , further comprising:
generating a map using the set of three-dimensional map points, wherein each three-dimensional map point of the set of three-dimensional map points corresponds to a particular feature in an environment of the wearable system.
4 . The method of claim 3 , wherein the map comprises a map point identifier and a feature identifier associated with each three-dimensional map point of the set of three-dimensional map points.
5 . The method of claim 3 , wherein the map comprises a list of keyframe poses and a corresponding list of the set of two-dimensional feature points for each three-dimensional map point of the set of three-dimensional map points.
6 . The method of claim 1 , further comprising:
generating a set of transforms representing a position and an orientation of the controller relative to the headset at a set of time instances corresponding to when each of the set of controller images were captured; and
associating the set of transforms with the set of controller images.
7 . The method of claim 6 , further comprising:
determining a location of a three-dimensional map point of the set of three-dimensional map points by triangulating a two-dimensional feature point of the set of two-dimensional feature points in each of the set of controller images using the set of transforms.
8 . A wearable system comprising:
a headset;
a controller having a set of fiducials; and
one or more processors configured to perform operations comprising:
causing the set of fiducials of the controller to flash, the set of fiducials 5 being arranged in a known geometry;
tracking a pose of the controller by:
capturing a set of headset images using a headset camera of the headset;
identifying the set of fiducials in the set of headset images; and
tracking the pose of the controller based on the identified set of fiducials in the set of headset images and based on a pose of the headset;
while tracking the pose of the controller, capturing a set of controller images using a controller camera of the controller;
identifying a set of two-dimensional feature points in each controller image of the set of controller images; and
determining a set of three-dimensional map points based on the set of two-dimensional feature points and the pose of the controller.
9 . The wearable system of claim 8 , wherein:
the headset comprises:
the headset camera; and
a headset inertial measurement unit; and
the controller comprises:
the set of fiducials arranged in the known geometry;
the controller camera; and
a controller inertial measurement unit; and
the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the controller camera, the headset inertial measurement unit, or the controller inertial measurement unit.
10 . The wearable system of claim 8 , wherein the operations further comprise:
generating a map using the set of three-dimensional map points, wherein each three-dimensional map point of the set of three-dimensional map points corresponds to a particular feature in an environment of the wearable system.
11 . The wearable system of claim 10 , wherein the map comprises a map point identifier and a feature identifier associated with each three-dimensional map point of the set of three-dimensional map points.
12 . The wearable system of claim 10 , wherein the map comprises a list of keyframe poses and a corresponding list of the set of two-dimensional feature points for each three-dimensional map point of the set of three-dimensional map points.
13 . The wearable system of claim 8 , wherein the operations further comprise:
generating a set of transforms representing a position and an orientation of the controller relative to the headset at a set of time instances corresponding to when each of the set of controller images were captured; and
associating the set of transforms with the set of controller images.
14 . The wearable system of claim 13 , wherein the operations further comprise:
determining a location of a three-dimensional map point of the set of three-dimensional map points by triangulating a two-dimensional feature point of the set of two-dimensional feature points in each of the set of controller images using the set of transforms.
15 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for operating a wearable system comprising a headset and a controller, the operations comprising:
causing a set of fiducials of the controller to flash, the set of fiducials being arranged in a known geometry;
tracking a pose of the controller by:
capturing a set of headset images using a headset camera of the headset;
identifying the set of fiducials in the set of headset images; and
tracking the pose of the controller based on the identified set of fiducials in the set of headset images and based on a pose of the headset;
while tracking the pose of the controller, capturing a set of controller images using a controller camera of the controller;
identifying a set of two-dimensional feature points in each controller image of the set of controller images; and
determining a set of three-dimensional map points based on the set of two-dimensional feature points and the pose of the controller.
16 . The non-transitory computer-readable medium of claim 15 , wherein the wearable system comprises:
the headset comprising:
the headset camera; and
a headset inertial measurement unit; and
the controller comprising:
the set of fiducials arranged in the known geometry;
the controller camera; and
a controller inertial measurement unit;
wherein the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the controller camera, the headset inertial measurement unit, or the controller inertial measurement unit.
17 . The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:
generating a map using the set of three-dimensional map points, wherein each three-dimensional map point of the set of three-dimensional map points corresponds to a particular feature in an environment of the wearable system.
18 . The non-transitory computer-readable medium of claim 17 , wherein the map comprises a map point identifier and a feature identifier associated with each three-dimensional map point of the set of three-dimensional map points.
19 . The non-transitory computer-readable medium of claim 17 , wherein the map comprises a list of keyframe poses and a corresponding list of the set of two-dimensional feature points for each three-dimensional map point of the set of three-dimensional map points.
20 . The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:
generating a set of transforms representing a position and an orientation of the controller relative to the headset at a set of time instances corresponding to when each of the set of controller images were captured; and
associating the set of transforms with the set of controller images.