Power-efficient hand tracking with time-of-flight sensor
Techniques are disclosed for operating a time-of-flight (TOF) sensor. The TOF may be operated in a low power mode by repeatedly performing a low power mode sequence, which may include performing a depth frame by emitting light pulses, detecting reflected light pulses, and computing a depth map based on the detected reflected light pulses. Performing the low power mode sequence may also include performing an amplitude frame at least one time by emitting a light pulse, detecting a reflected light pulse, and computing an amplitude map based on the detected reflected light pulse. In response to determining that an activation condition is satisfied, the TOF may be switched to operate in a high accuracy mode by repeatedly performing a high accuracy mode sequence, which may include performing the depth frame multiple times.
1. A method of operating a time-of-flight (TOF) sensor, the method comprising:
operating the TOF sensor in a low power mode by repeatedly performing a low power mode sequence, wherein performing the low power mode sequence includes:
performing a depth frame, wherein performing the depth frame includes emitting light pulses, detecting reflected light pulses, and computing a depth map based on the detected reflected light pulses; and
performing an amplitude frame at least one time, wherein performing the amplitude frame includes emitting a light pulse, detecting a reflected light pulse, and computing an amplitude map based on the detected reflected light pulse;
determining that an activation condition is satisfied; and
in response to determining that the activation condition is satisfied, switching from operating the TOF sensor in the low power mode to operating the TOF sensor in a high accuracy mode by repeatedly performing a high accuracy mode sequence, wherein performing the high accuracy mode sequence includes:
performing the depth frame multiple times.
2. The method of claim 1 , wherein performing the depth frame further includes:
computing three-dimensional (3D) positions of a plurality of keypoints along a target object based on the depth map.
3. The method of claim 2 , wherein the target object is a user's hand.
4. The method of claim 3 , wherein determining that the activation condition is satisfied includes:
determining that the user's hand is currently interacting or is about to interact in a Z dimension.
5. The method of claim 2 , wherein performing the amplitude frame further includes:
computing two-dimensional (2D) positions of the plurality of keypoints along the target object based on the amplitude map.
6. The method of claim 5 , wherein performing the amplitude frame further includes:
estimating the 3D positions of the plurality of keypoints along the target object based on the 2D positions of the plurality of keypoints.
7. The method of claim 6 , wherein the 3D positions of the plurality of keypoints are estimated further based on the depth map.
8. A system for operating a time-of-flight (TOF) sensor, the system comprising:
one or more processors; and
a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
operating the TOF sensor in a low power mode by repeatedly performing a low power mode sequence, wherein performing the low power mode sequence includes:
performing a depth frame, wherein performing the depth frame includes emitting light pulses, detecting reflected light pulses, and computing a depth map based on the detected reflected light pulses; and
performing an amplitude frame at least one time, wherein performing the amplitude frame includes emitting a light pulse, detecting a reflected light pulse, and computing an amplitude map based on the detected reflected light pulse;
determining that an activation condition is satisfied; and
in response to determining that the activation condition is satisfied, switching from operating the TOF sensor in the low power mode to operating the TOF sensor in a high accuracy mode by repeatedly performing a high accuracy mode sequence, wherein performing the high accuracy mode sequence includes:
performing the depth frame multiple times.
9. The system of claim 8 , wherein performing the depth frame further includes:
computing three-dimensional (3D) positions of a plurality of keypoints along a target object based on the depth map.
10. The system of claim 9 , wherein the target object is a user's hand.
11. The system of claim 10 , wherein determining that the activation condition is satisfied includes:
determining that the user's hand is currently interacting or is about to interact in a Z dimension.
12. The system of claim 9 , wherein performing the amplitude frame further includes:
computing two-dimensional (2D) positions of the plurality of keypoints along the target object based on the amplitude map.
13. The system of claim 12 , wherein performing the amplitude frame further includes:
estimating the 3D positions of the plurality of keypoints along the target object based on the 2D positions of the plurality of keypoints.
14. The system of claim 13 , wherein the 3D positions of the plurality of keypoints are estimated further based on the depth map.
15. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for operating a time-of-flight (TOF) sensor, the operations comprising:
operating a TOF sensor in a low power mode by repeatedly performing a low power mode sequence, wherein performing the low power mode sequence includes:
performing a depth frame, wherein performing the depth frame includes emitting light pulses, detecting reflected light pulses, and computing a depth map based on the detected reflected light pulses; and
performing an amplitude frame at least one time, wherein performing the amplitude frame includes emitting a light pulse, detecting a reflected light pulse, and computing an amplitude map based on the detected reflected light pulse;
determining that an activation condition is satisfied; and
in response to determining that the activation condition is satisfied, switching from operating the TOF sensor in the low power mode to operating the TOF sensor in a high accuracy mode by repeatedly performing a high accuracy mode sequence, wherein performing the high accuracy mode sequence includes:
performing the depth frame multiple times.
16. The non-transitory computer-readable medium of claim 15 , wherein performing the depth frame further includes:
computing three-dimensional (3D) positions of a plurality of keypoints along a target object based on the depth map.
17. The non-transitory computer-readable medium of claim 16 , wherein the target object is a user's hand.
18. The non-transitory computer-readable medium of claim 17 , wherein determining that the activation condition is satisfied includes:
determining that the user's hand is currently interacting or is about to interact in a Z dimension.
19. The non-transitory computer-readable medium of claim 16 , wherein performing the amplitude frame further includes:
computing two-dimensional (2D) positions of the plurality of keypoints along the target object based on the amplitude map.
20. The non-transitory computer-readable medium of claim 19 , wherein performing the amplitude frame further includes:
estimating the 3D positions of the plurality of keypoints along the target object based on the 2D positions of the plurality of keypoints.