Wearable system with controller localization using headset cameras and controller fiducials
Wearable systems and method for operation thereof incorporating headset and controller localization using headset cameras and controller fiducials are disclosed. A wearable system may include a headset and a controller. The wearable system may alternate between performing headset tracking and performing controller tracking by repeatedly capturing images using a headset camera of the headset during headset tracking frames and controller tracking frames. The wearable system may cause the headset camera to capture a first exposure image an exposure above a threshold and cause the headset camera to capture a second exposure image having an exposure below the threshold. The wearable system may determine a fiducial interval during which fiducials of the controller are to flash at a fiducial frequency and a fiducial period. The wearable system may cause the fiducials to flash during the fiducial interval in accordance with the fiducial frequency and the fiducial period.
1 . A method of operating a wearable system having a headset and a controller, the method comprising:
alternating between performing headset tracking and performing controller tracking by repeatedly capturing images using a headset camera of the headset during headset tracking frames and controller tracking frames, respectively;
during each of the headset tracking frames:
causing the headset camera to capture a first exposure image of the images having an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration;
during each of the controller tracking frames:
causing the headset camera to capture a second exposure image of the images having an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration;
determining a fiducial interval defined by a fiducial start time and a fiducial end time during which a set of fiducials of the controller are to flash multiple times at a fiducial frequency and a fiducial period, wherein the fiducial interval is determined such that the second exposure interval at least partially overlaps with the fiducial interval; and
causing the set of fiducials to flash multiple times during the fiducial interval in accordance with the fiducial frequency and the fiducial period.
2 . The method of claim 1 , wherein the wearable system comprises:
the headset comprising:
the headset camera; and
a headset inertial measurement unit; and
the controller comprising:
the set of fiducials arranged in a known geometry;
one or more controller cameras; and
a controller inertial measurement unit;
wherein the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the one or more controller cameras, the headset inertial measurement unit, or the controller inertial measurement unit.
3 . The method of claim 2 , wherein operating the wearable system comprises:
determining a first pose of the headset with respect to a reference frame based on data captured by the headset camera of the headset or the headset inertial measurement unit of the headset;
causing the set of fiducials of the controller to flash;
determining a second pose of the controller with respect to the headset by:
capturing a headset image using the headset camera;
identifying the set of fiducials in the headset image; and
determining the second pose of the controller with respect to the headset based on the set of fiducials identified in the headset image and the known geometry.
4 . The method of claim 1 , wherein the fiducial interval is determined such that the second exposure interval is centered with the fiducial interval.
5 . The method of claim 1 , wherein a first length of time of a headset tracking frame of the headset tracking frames is equal to a second length of time of a controller tracking frame of the controller tracking frames.
6 . The method of claim 1 , wherein the first exposure duration comprises at least one millisecond.
7 . A wearable system comprising:
a headset;
a controller having a set of fiducials; and
one or more processors configured to perform operations comprising:
alternating between performing headset tracking and performing controller tracking by repeatedly capturing images using a headset camera of the headset during headset tracking frames and controller tracking frames, respectively;
during each of the headset tracking frames:
causing the headset camera to capture a first exposure image of the images having an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration;
during each of the controller tracking frames:
causing the headset camera to capture a second exposure image of the images having an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration;
determining a fiducial interval defined by a fiducial start time and a fiducial end time during which the set of fiducials of the controller are to flash multiple times at a fiducial frequency and a fiducial period, wherein the fiducial interval is determined such that the second exposure interval at least partially overlaps with the fiducial interval; and
causing the set of fiducials to flash multiple times during the fiducial interval in accordance with the fiducial frequency and the fiducial period.
8 . The wearable system of claim 7 , wherein:
the headset comprises:
the headset camera; and
a headset inertial measurement unit; and
the controller comprises:
the set of fiducials arranged in a known geometry;
one or more controller cameras; and
a controller inertial measurement unit; and
the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the one or more controller cameras, the headset inertial measurement unit, or the controller inertial measurement unit.
9 . The wearable system of claim 8 , wherein the operations further comprise:
determining a first pose of the headset with respect to a reference frame based on data captured by the headset camera of the headset or the headset inertial measurement unit of the headset;
causing the set of fiducials of the controller to flash;
determining a second pose of the controller with respect to the headset by:
capturing a headset image using the headset camera;
identifying the set of fiducials in the headset image; and
determining the second pose of the controller with respect to the headset based on the set of fiducials identified in the headset image and the known geometry.
10 . The wearable system of claim 7 , wherein the fiducial interval is determined such that the second exposure interval is centered with the fiducial interval.
11 . The wearable system of claim 7 , wherein a first length of time of a headset tracking frame of the headset tracking frames is equal to a second length of time of a controller tracking frame of the controller tracking frames.
12 . The wearable system of claim 7 , wherein the first exposure duration comprises at least one millisecond.
13 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for operating a wearable system having a headset and a controller, the operations comprising:
alternating between performing headset tracking and performing controller tracking by repeatedly capturing images using a headset camera of the headset during headset tracking frames and controller tracking frames, respectively;
during each of the headset tracking frames:
causing the headset camera to capture a first exposure image of the images having an exposure above a threshold, wherein the first exposure image is associated with a first exposure interval defined by a first exposure start time, a first exposure end time, and a first exposure duration;
during each of the controller tracking frames:
causing the headset camera to capture a second exposure image of the images having an exposure below the threshold, wherein the second exposure image is associated with a second exposure interval defined by a second exposure start time, a second exposure end time, and a second exposure duration, wherein the second exposure duration is less than the first exposure duration;
determining a fiducial interval defined by a fiducial start time and a fiducial end time during which a set of fiducials of the controller are to flash multiple times at a fiducial frequency and a fiducial period, wherein the fiducial interval is determined such that the second exposure interval at least partially overlaps with the fiducial interval; and
causing the set of fiducials to flash multiple times during the fiducial interval in accordance with the fiducial frequency and the fiducial period.
14 . The non-transitory computer-readable medium of claim 13 , wherein the wearable system comprises:
the headset comprising:
the headset camera; and
a headset inertial measurement unit; and
the controller comprising:
the set of fiducials arranged in a known geometry;
one or more controller cameras; and
a controller inertial measurement unit;
wherein the wearable system is configured to determine a position or orientation of the headset or the controller based on data captured by the headset camera, the one or more controller cameras, the headset inertial measurement unit, or the controller inertial measurement unit.
15 . The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:
determining a first pose of the headset with respect to a reference frame based on data captured by the headset camera of the headset or the headset inertial measurement unit of the headset;
causing the set of fiducials of the controller to flash;
determining a second pose of the controller with respect to the headset by:
capturing a headset image using the headset camera;
identifying the set of fiducials in the headset image; and
determining the second pose of the controller with respect to the headset based on the set of fiducials identified in the headset image and the known geometry.
16 . The non-transitory computer-readable medium of claim 13 , wherein the fiducial interval is determined such that the second exposure interval is centered with the fiducial interval.
17 . The non-transitory computer-readable medium of claim 13 , wherein a first length of time of a headset tracking frame of the headset tracking frames is equal to a second length of time of a controller tracking frame of the controller tracking frames.
18 . The non-transitory computer-readable medium of claim 13 , wherein the first exposure duration comprises at least one millisecond.