Granted Patent B2

US 12,572,216 · App. 18/741,375 · Granted Mar 10, 2026

Inventors: Markus Diem (Vienna, AT); Thomas Muttenthaler (Vienna, AT); Daniel Wolf (Mödling, AT)

Assignee: Snap Inc.

G06F3/017G06F3/011

Abstract

Claims (52)

1 . A method performed by an extended reality (XR) device, the method comprising:

accessing tracking data captured via one or more sensors of a plurality of sensors associated with the XR device;

processing the tracking data to track at least one body part;

dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part;

using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and

controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part,

wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and selection of the subset is based on at least one of:

a camera of the plurality of cameras predicted to have the at least one body part closest to a center of a field of view of the camera;

a camera of the plurality of cameras predicted to have a least occluded view of the at least one body part; or

a camera of the plurality of cameras predicted to have a clearest view of a predetermined portion of the at least one body part.

2 . The method of claim 1 , wherein the XR device is a head-wearable XR device, and the at least one body part comprises at least part of a hand of a user of the XR device.

3 . The method of claim 2 , wherein the tracking data is processed to track positions of a plurality of landmarks comprising a plurality of joints of the hand, and the kinematic model is applied by the XR device to describe joint positions and joint angles.

4 . The method of claim 3 , wherein dynamically updating the kinematic state tracking data comprises tracking, over time, at least one of linear velocity of one or more of the plurality of joints, angular velocity of one or more of the plurality of joints, linear acceleration of one or more of the plurality of joints, angular acceleration of one or more of the plurality of joints, linear jerk of one or more of the plurality of joints, or angular jerk of one or more of the plurality of joints.

5 . The method of claim 1 , wherein the kinematic state tracking data tracks a kinematic state of the at least one body part over time, and the kinematic state is defined using the kinematic model.

6 . The method of claim 5 , wherein the predicted future kinematic state is generated based on the kinematic state tracking data and motion constraints defined by the kinematic model.

7 . The method of claim 5 , wherein the predicted future kinematic state is generated at a first point in time to predict the kinematic state of the at least one body part at a second point in time, and the second point in time is less than 1 second from the first point in time.

8 . The method of claim 5 , wherein the predicted future kinematic state is generated at a first point in time to predict the kinematic state of the at least one body part at a second point in time, and the second point in time is less than 500 ms from the first point in time.

9 . The method of claim 5 , wherein the kinematic state comprises a six degrees-of-freedom (6DoF) pose of the at least one body part in a real-world environment.

10 . The method of claim 1 , wherein at least some of the kinematic state tracking data is generated or updated, using inverse kinematics, by fitting positions of a plurality of landmarks obtained from the tracking data to the kinematic model.

11 . The method of claim 1 , wherein controlling the operation of the XR device based on the predicted future kinematic state further comprises:

identifying, based on the predicted future kinematic state, a predicted user action;

determining a device action corresponding to the predicted user action; and

synchronizing the device action with occurrence of the predicted user action.

12 . The method of claim 11 , wherein the predicted user action comprises a user of the XR device performing a detectable gesture.

13 . The method of claim 12 , wherein the device action comprises a response to the detectable gesture.

14 . The method of claim 11 , wherein the device action comprises triggering rendering of virtual content for presentation to a user via the XR device.

15 . The method of claim 1 , wherein the predicted future kinematic state comprises a predicted pose of the at least one body part, and the subset is further selected based at least partially on the predicted pose in relation to the field of view of each respective camera of the plurality of cameras.

16 . The method of claim 1 , wherein processing of the tracking data comprises executing a machine learning model that is trained to perform object tracking.

17 . An extended reality (XR) device comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the XR device to perform operations comprising:

accessing tracking data captured via one or more sensors of a plurality of sensors associated with the XR device;

processing the tracking data to track at least one body part;

dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part;

using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and

controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part,

wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and selection of the subset is based on at least one of:

a camera of the plurality of cameras predicted to have the at least one body part closest to a center of a field of view of the camera;

a camera of the plurality of cameras predicted to have a least occluded view of the at least one body part; or

a camera of the plurality of cameras predicted to have a clearest view of a predetermined portion of the at least one body part.

18 . The XR device of claim 17 , wherein the XR device is a head-wearable XR device, and the at least one body part comprises at least part of a hand of a user of the XR device.

19 . A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:

accessing tracking data captured via one or more sensors of a plurality of sensors associated with an extended reality (XR) device;

processing the tracking data to track at least one body part;

dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part;

using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and

controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part,

wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and selection of the subset is based on at least one of:

a camera of the plurality of cameras predicted to have the at least one body part closest to a center of a field of view of the camera;

a camera of the plurality of cameras predicted to have a least occluded view of the at least one body part; or

a camera of the plurality of cameras predicted to have a clearest view of a predetermined portion of the at least one body part.

20 . The non-transitory computer-readable storage medium of claim 19 , wherein the XR device is a head-wearable XR device, and the at least one body part comprises at least part of a hand of a user of the XR device.

Assignments (3)

Continuity (1)

References Cited (90)