Steerable camera for AR hand tracking
A system for hand tracking for an Augmented Reality (AR) system. The AR system uses a camera of the AR system to capture tracking video frame data of a hand of a user of the AR system. The AR system generates a skeletal model based on the tracking video frame data and determines a location of the hand of the user based on the skeletal model. The AR system causes a steerable camera of the AR system to focus on the hand of the user.
1. A computer-implemented method comprising:
capturing, by one or more processors, using a first camera of an Augmented Reality (AR) system, tracking video frame data of a hand of a user of the AR system;
determining, by the one or more processors, a location of the hand of the user using the tracking video frame data; and
focusing, by the one or more processors, a steerable camera of the AR system on the hand of the user using the location;
wherein the first camera is the steerable camera, the method further comprising:
scanning, by the one or more processors, using the steerable camera, an AR Field Of View (FOV) of the AR system to initially locate the hand of the user;
using, by the one or more processors, a look-ahead process to predict a next location of the hand of the user based on a current location of the hand; and
focusing, by the one or more processors, the steerable camera on the next location.
2. The method of claim 1 , wherein the first camera comprises a non-steerable camera having a camera FOV equal to an AR FOV of the AR system.
3. The method of claim 1 , wherein the next location is predicted further based on a language model.
4. The method of claim 3 , wherein the language model is for American Sign Language.
5. The method of claim 1 , further wherein the steerable camera comprises:
a camera having sensor and a lens assembly; and
one or more actuators linked to the camera.
6. The method of claim 1 , wherein the AR system comprises a head-worn device.
7. A computing apparatus comprising:
one or more processors; and
a memory storing instructions that, when executed by one or more processors, cause the computing apparatus to perform operations comprising:
capturing, by one or more processors, using a first camera of an Augmented Reality (AR) system, tracking video frame data of a hand of a user of the AR system;
determining, by the one or more processors, a location of the hand of the user using the tracking video frame data; and
focusing, by the one or more processors, a steerable camera of the AR system on the hand of the user using the locations;
wherein the first camera is the steerable camera, and
wherein the instructions, when executed by the one or more processors, further cause the computing apparatus to perform operations comprising:
scanning, using the steerable camera, an AR FOV of the AR system to initially locate the hand of the user;
using a look-ahead process to predict a next location of the hand of the user based on a current location of the hand; and
focusing the steerable camera on the next location.
8. The computing apparatus of claim 7 , wherein the first camera comprises a non-steerable camera having a camera FOV equal to an AR FOV of the AR system.
9. The computing apparatus of claim 7 , wherein the next location is predicted further based on a language model.
10. The computing apparatus of claim 9 , wherein the language model is for American Sign Language.
11. The computing apparatus of claim 7 , wherein the steerable camera comprises:
a camera having sensor and a lens assembly; and
one or more actuators linked to the camera.
12. The computing apparatus of claim 7 , wherein the AR system comprises a head-worn device.
13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computing apparatus, cause the computing apparatus to perform operations comprising:
capturing, by one or more processors, using a first camera of an Augmented Reality (AR) system, tracking video frame data of a hand of a user of the AR system;
determining, by the one or more processors, a location of the hand of the user using the tracking video frame data; and
focusing, by the one or more processors, a steerable camera of the AR system on the hand of the user using the location;
wherein the first camera is the steerable camera, and
wherein the instructions, when executed by the computing apparatus, further cause the computing apparatus to perform operations comprising:
scanning, using the steerable camera, an AR FOV of the AR system to initially locate the hand of the user;
using a look-ahead process to predict a next location of the hand of the user based on a current location of the hand; and
focusing the steerable camera on the next location.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the first camera comprises a non-steerable camera having a camera FOV equal to an AR FOV of the AR system.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the next location is predicted further based on a language model.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the language model is for American Sign Language.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the AR system comprises a head-worn device.