Reducing power consumption of extended reality devices
Examples describe a method performed by an extended reality (XR) device that implements a multi-camera object tracking system. The XR device accesses object tracking data associated with an object in a real-world environment. Based on the object tracking data, the XR device activates a low-power mode of the multi-camera object tracking system. In the low-power mode, a state of the object in the real-world environment is determined by using the multi-camera object tracking system.
1 . A method performed by an extended reality (XR) device that implements a multi-camera object tracking system, the method comprising:
accessing object tracking data associated with an object in a real-world environment;
activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system;
in the low-power mode, activating a dynamic switching function that causes the XR device to switch between different subsets of cameras of the multi-camera object tracking system;
cycling through the different subsets of cameras of the multi-camera object tracking system according to a sampling sequence that is selected based on the object tracking data, the sampling sequence sampling more frequently from one or more cameras than from one or more other cameras of the multi-camera object tracking system; and
determining a state of the object in the real-world environment using the multi-camera object tracking system.
2 . The method of claim 1 , wherein the object is a hand of a user of the XR device.
3 . The method of claim 2 , wherein the object tracking data comprises handedness data that identifies the hand of the user.
4 . The method of claim 1 , wherein activating the low-power mode comprises dynamically selecting a subset of cameras of the multi-camera object tracking system for determining the state of the object.
5 . The method of claim 1 , wherein activating the dynamic switching function causes, at a first point in time, a first subset of the cameras to be selected to determine the state of the object, and at a second point in time, a second subset of the cameras to be selected to determine the state of the object.
6 . The method of claim 1 , wherein the state of the object comprises at least a location associated with the object, and wherein the object tracking data comprises a predicted location of the object within a three-dimensional reference coordinate system of the XR device.
7 . The method of claim 6 , wherein the predicted location of the object is generated based on at least one of: historic tracking data; or a predicted pose of the XR device.
8 . The method of claim 6 , wherein activating the low-power mode comprises selecting, based on the predicted location of the object relative to a field of view of each respective camera of the multi-camera object tracking system, a subset of cameras of the multi-camera object tracking system for determining the state of the object in the real-world environment.
9 . The method of claim 6 , wherein determining the state of the object comprises:
projecting, for each camera of the multi-camera object tracking system and based on a predetermined calibration of the camera, the predicted location of the object onto a two-dimensional camera view image to obtain a two-dimensional projected location;
selecting, based on the projected locations, a subset of cameras of the multi-camera object tracking system; and
using only the subset of cameras to determine the state of the object in the real-world environment.
10 . The method of claim 9 , wherein each camera in the subset of cameras is selected based on the projected location relative to at least one reference location within the respective camera view image.
11 . The method of claim 10 , wherein, for each camera in the subset of cameras, the camera is selected based on the projected location meeting a predefined condition, wherein the predefined condition is one of: a distance between the projected location and the at least one reference location is less than a threshold value;
a difference between the distance between the projected location and the at least one reference location for one or more other cameras of the multi-camera object tracking system and the distance between the projected location and the at least one reference location for the camera exceeds a threshold value;
the projected location is within a predetermined zone in the respective camera view image relative to the at least one reference location; or
the projected location is outside of a predetermined zone in the respective camera view image relative to the at least one reference location.
12 . The method of claim 11 , wherein the camera is selected based on the predefined condition being met with respect to multiple consecutive frames.
13 . The method of claim 10 , wherein projecting the predicted location of the object comprises projecting at least one point of interest associated with the object onto the camera view image and determining a bounding box associated with the at least one point of interest, the method further comprising, for each camera of the multi-camera object tracking system, determining a distance between the bounding box and the at least one reference location within the respective camera view image, each camera in the subset of cameras being selected based on the determined distance.
14 . The method of claim 1 , wherein activating the low-power mode comprises reducing a sampling rate of at least a subset of cameras of the multi-camera object tracking system.
15 . The method of claim 1 , wherein activating the low-power mode comprises activating a reduced image processing setting for at least a subset of cameras of the multi-camera object tracking system.
16 . The method of claim 1 , wherein activating the low-power mode comprises:
selecting, based on the object tracking data, a subset of cameras of the multi-camera object tracking system for determining the state of the object; and
switching off each camera of the multi-camera object tracking system that is excluded from the selected subset of cameras.
17 . The method of claim 1 , wherein the object tracking data comprises at least one of:
detection data indicating whether the object has been detected;
a predicted location of the object within a three-dimensional reference coordinate system of the XR device;
a predicted appearance zone of the object;
predicted motion of the object;
handedness data of a user of the XR device;
historic power consumption data for at least one camera of the multi-camera object tracking system; or
expected inference performance for at least one camera of the multi-camera object tracking system with respect to the object.
18 . The method of claim 1 , wherein cycling through the different subsets of cameras of the multi-camera object tracking system causes a reduction in power consumption associated with the one or more other cameras.
19 . An extended reality (XR) device comprising:
a multi-camera object tracking system;
a memory that stores instructions; and
at least one processor configured by the instructions to perform operations comprising:
accessing object tracking data associated with an object in a real-world environment;
activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system;
in the low-power mode, activating a dynamic switching function that causes the XR device to switch between different subsets of cameras of the multi-camera object tracking system;
cycling through the different subsets of cameras of the multi-camera object tracking system according to a sampling sequence that is selected based on the object tracking data, the sampling sequence sampling more frequently from one or more cameras than from one or more other cameras of the multi-camera object tracking system; and
determining a state of the object in the real-world environment using the multi-camera object tracking system.
20 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of an extended reality (XR) device that implements a multi-camera object tracking system, cause the at least one processor to perform operations comprising:
accessing object tracking data associated with an object in a real-world environment;
activating, based on the object tracking data, a low-power mode of the multi-camera object tracking system;
in the low-power mode, activating a dynamic switching function that causes the XR device to switch between different subsets of cameras of the multi-camera object tracking system;
cycling through the different subsets of cameras of the multi-camera object tracking system according to a sampling sequence that is selected based on the object tracking data, the sampling sequence sampling more frequently from one or more cameras than from one or more other cameras of the multi-camera object tracking system; and
determining a state of the object in the real-world environment using the multi-camera object tracking system.