IP Library › Granted Patent US 12710828
Granted Patent B2
US 12710828 · App. 19/294,509 · Granted Aug 18, 2026

Steerable camera for AR hand tracking

Inventors: Daniel Colascione (Melbourne Beach, FL); Patrick Timothy McSweeney Simons (Downey, CA); Weston Welge (Boulder, CO); Ramzi Zahreddine (Denver, CO)
Assignee: Snap Inc.
G06F3/017G02B27/017G06F3/011G06T19/006G02B2027/0138G02B2027/0178
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12710828
App. No.
19/294,509
Granted
Aug 18, 2026
Kind
B2
Abstract

A system for hand tracking for an Augmented Reality (AR) system. The AR system uses a camera of the AR system to capture tracking video frame data of a hand of a user of the AR system. The AR system generates a skeletal model based on the tracking video frame data and determines a location of the hand of the user based on the skeletal model. The AR system causes a steerable camera of the AR system to focus on the hand of the user.

Claims (34)

1 . A computer-implemented method comprising:

capturing, by a non-steerable camera of an Augmented Reality (AR) system, video frame data of a real-world scene environment from a perspective of a user of the AR system, the video frame data including portions of one or more hands of the user, and the non-steerable camera having a field of view equal to a field of view of the AR system;

determining a location of one or more hands of the user based on the video frame data captured by the non-steerable camera;

generating steering command data based on the location of the one or more hands of the user; and

steering an optical axis of a steerable camera of the AR system based on the steering command data to focus the steerable camera on the one or more hands of the user.

2 . The computer-implemented method of claim 1 , wherein the steerable camera has a narrower field of view than the non-steerable camera and a higher resolution than the non-steerable camera.

3 . The computer-implemented method of claim 1 , wherein the non-steerable camera has a wider field of view than the steerable camera and a lower resolution than the steerable camera.

4 . The computer-implemented method of claim 1 , wherein determining the location comprises determining a centroid of a skeletal model of the one or more hands of the user and projecting a ray extending from a viewpoint of the user to the centroid of the skeletal model.

5 . The computer-implemented method of claim 4 , wherein the steering command data includes a pitch adjustment command instructing a camera to set a pitch angle of the steerable camera to an azimuth angle of the ray projected from the viewpoint of the user to the centroid of the skeletal model.

6 . The computer-implemented method of claim 5 , wherein the steering command data further includes a yaw adjustment command instructing the camera system to set a yaw angle of the steerable camera to an inclination angle of the ray projected from the viewpoint of the user to the centroid of the skeletal model.

7 . The computer-implemented method of claim 1 , wherein the AR system comprises a head-worn device.

8 . A machine comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:

capturing, by a non-steerable camera of an Augmented Reality (AR) system, video frame data of a real-world scene environment from a perspective of a user of the AR system, the video frame data including portions of one or more hands of the user, and the non-steerable camera having a field of view equal to a field of view of the AR system;

determining a location of one or more hands of the user based on the video frame data captured by the non-steerable camera;

generating steering command data based on the location of the one or more hands of the user; and

steering an optical axis of a steerable camera of the AR system based on the steering command data to focus the steerable camera on the one or more hands of the user.

9 . The machine of claim 8 , wherein the steerable camera has a narrower field of view than the non-steerable camera and a higher resolution than the non-steerable camera.

10 . The machine of claim 8 , wherein the non-steerable camera has a wider field of view than the steerable camera and a lower resolution than the steerable camera.

11 . The machine of claim 8 , wherein determining the location comprises determining a centroid of a skeletal model of the one or more hands of the user and projecting a ray extending from a viewpoint of the user to the centroid of the skeletal model.

12 . The machine of claim 11 , wherein the steering command data includes a pitch adjustment command a camera to set a pitch angle of the steerable camera to an azimuth angle of the ray projected from the viewpoint of the user to the centroid of the skeletal model.

13 . The machine of claim 12 , wherein the steering command data further includes a yaw adjustment command the camera system to set a yaw angle of the steerable camera to an inclination angle of the ray projected from the viewpoint of the user to the centroid of the skeletal model.

14 . The machine of claim 8 , wherein the AR system comprises a head-worn device.

15 . A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

capturing, by a non-steerable camera of an Augmented Reality (AR) system, video frame data of a real-world scene environment from a perspective of a user of the AR system, the video frame data including portions of one or more hands of the user, and the non-steerable camera having a field of view equal to a field of view of the AR system;

determining a location of one or more hands of the user based on the video frame data captured by the non-steerable camera;

generating steering command data based on the location of the one or more hands of the user; and

steering an optical axis of a steerable camera of the AR system based on the steering command data to focus the steerable camera on the one or more hands of the user.

16 . The non-transitory machine-readable storage medium of claim 15 , wherein the steerable camera has a narrower field of view than the non-steerable camera and a higher resolution than the non-steerable camera.

17 . The non-transitory machine-readable storage medium of claim 15 , wherein the non-steerable camera has a wider field of view than the steerable camera and a lower resolution than the steerable camera.

18 . The non-transitory machine-readable storage medium of claim 15 , wherein determining the location comprises determining a centroid of a skeletal model of the one or more hands of the user and projecting a ray extending from a viewpoint of the user to the centroid of the skeletal model.

19 . The non-transitory machine-readable storage medium of claim 18 , wherein the steering command data includes a pitch adjustment command a camera to set a pitch angle of the steerable camera to an azimuth angle of the ray projected from the viewpoint of the user to the centroid of the skeletal model.

20 . The non-transitory machine-readable storage medium of claim 15 , wherein the AR system comprises a head-worn device.