Steerable camera for AR hand tracking
A system for hand tracking for an Augmented Reality (AR) system. The AR system uses a camera of the AR system to capture tracking video frame data of a hand of a user of the AR system. The AR system generates a skeletal model based on the tracking video frame data and determines a location of the hand of the user based on the skeletal model. The AR system causes a steerable camera of the AR system to focus on the hand of the user.
1. A machine-implemented method, comprising:
capturing, by an Augmented Reality (AR) camera of an AR system, video frame data of a hand of a user;
generating coordinate transformation data based on the video frame data, wherein the coordinate transformation data represents a location of the hand of the user in a 3D spherical coordinate system relative to a viewpoint of the user;
generating camera steering command data based on the coordinate transformation data; and
adjusting an optical axis of a steerable camera based on the camera steering command data to align the optical axis with the location of a skeletal model of the user's hand;
wherein generating the camera steering command data comprises:
determining a centroid of the skeletal model;
projecting a ray from a viewpoint of the user to the centroid; and
generating pitch and yaw adjustment commands based on azimuth and inclination angles of the ray.
2. The machine-implemented method of claim 1 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more actuators linked to the camera.
3. The machine-implemented method of claim 1 , wherein the steerable camera comprises:
a stationary camera;
a mirror; and
one or more actuators linked to the mirror.
4. The machine-implemented method of claim 1 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more configurable optical elements comprising Spatial Light Modulators (SLMs) that spatially modulate their phase.
5. The machine-implemented method of claim 1 , wherein the AR camera and the steerable camera are the same camera.
6. The machine-implemented method of claim 1 , wherein the AR system comprises a head-worn device.
7. A machine comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:
capturing, by an Augmented Reality (AR) camera of an AR system, video frame data of a hand of a user;
generating coordinate transformation data based on the video frame data, wherein the coordinate transformation data represents a location of the hand of the user in a 3D spherical coordinate system relative to a viewpoint of the user;
generating camera steering command data based on the coordinate transformation data; and
adjusting an optical axis of a steerable camera based on the camera steering command data to align the optical axis with the location of a skeletal model of the user's hand;
wherein generating the camera steering command data comprises:
determining a centroid of the skeletal model;
projecting a ray from a viewpoint of the user to the centroid; and
generating pitch and yaw adjustment commands based on azimuth and inclination angles of the ray.
8. The machine of claim 7 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more actuators linked to the camera.
9. The machine of claim 7 , wherein the steerable camera comprises:
a stationary camera;
a mirror; and
one or more actuators linked to the mirror.
10. The machine of claim 7 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more configurable optical elements comprising Spatial Light Modulators (SLMs) that spatially modulate their phase.
11. The machine of claim 7 , wherein the AR camera and the steerable camera are the same camera.
12. The machine of claim 7 , wherein the AR system comprises a head-worn device.
13. A non-transitory computer-readable medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
capturing, by an Augmented Reality (AR) camera of an AR system, video frame data of a hand of a user;
generating coordinate transformation data based on the video frame data, wherein the coordinate transformation data represents a location of the hand of the user in a 3D spherical coordinate system relative to a viewpoint of the user;
generating camera steering command data based on the coordinate transformation data; and
adjusting an optical axis of a steerable camera based on the camera steering command data to align the optical axis with the location of a skeletal model of the user's hand;
wherein generating the camera steering command data comprises:
determining a centroid of the skeletal model;
projecting a ray from a viewpoint of the user to the centroid; and
generating pitch and yaw adjustment commands based on azimuth and inclination angles of the ray.
14. The non-transitory computer-readable medium of claim 13 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more actuators linked to the camera.
15. The non-transitory computer-readable medium of claim 13 , wherein the steerable camera comprises:
a stationary camera;
a mirror; and
one or more actuators linked to the mirror.
16. The non-transitory computer-readable medium of claim 13 , wherein the steerable camera comprises:
a camera having a sensor and a lens assembly; and
one or more configurable optical elements comprising Spatial Light Modulators (SLMs) that spatially modulate their phase.
17. The non-transitory computer-readable medium of claim 13 , wherein the AR camera and the steerable camera are the same camera.