Granted Patent B2

US 12,437,491 · App. 18/065,201 · Granted Oct 7, 2025

Inventor: Mason Spong (Provo, UT)

Assignee: Snap Inc.

G06T19/20G06F3/014G06F3/017G06T19/006G06T2219/2016

Abstract

Claims (77)

1. A machine comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:

providing to a user an extended Reality (XR) user interface of an XR system, the XR user interface comprising a virtual object displayed to the user;

determining a pinch location of a pinch hand pose being made by the user using one hand of the user;

scaling the virtual object based on the pinch hand pose of the one hand using the pinch location and a virtual object center point of the virtual object by performing operations comprising:

generating a virtual object vector based on a center point of the virtual object and the pinch location;

generating a pinch vector based on the center point of the virtual object and a virtual object pinch collider of the virtual object;

determining a scaling vector based on a projection of the pinch vector onto the virtual object vector; and

adjusting the scale of the virtual object based on a ratio of a length of the scaling vector and a length of the virtual object vector; and

redisplaying the scaled virtual object to the user in the XR user interface.

2. The machine of claim 1 , wherein determining a pinch location comprises:

capturing using one or more cameras of the XR system, tracking video frame data;

determining hand-tracking data based on the tracking video frame data; and

determining the pinch location based on the hand-tracking data.

3. The machine of claim 2 , wherein determining the pinch location based on the hand-tracking data comprises:

determining a pinch location collider based on skeletal model data of the hand-tracking data; and

determining the pinch location based on the pinch location collider.

4. The machine of claim 1 , wherein the operations further comprise:

detecting that the pinch hand pose is still being held by the user after redisplaying the scaled object; and

in response to detecting that the pinch hand pose is still being held, continuing to scale the virtual object based on the scaling vector.

5. The machine of claim 1 , wherein the XR system comprises a head-wearable apparatus.

6. The machine of claim 1 , wherein determining the pinch location comprises:

detecting an intersection between a pinch location collider and a virtual object pinch collider; and

determining three-dimensional coordinates of the pinch location based on the detected intersection between the pinch location collider and the virtual object pinch collider.

7. The machine of claim 1 , wherein determining that the pinch hand pose is no longer being held comprises:

detecting a sequence of proto hand poses as the user moves a fingertip of a forefinger away from a fingertip of a thumb of the one hand making the pinch hand pose; and

determining the pinch hand pose is released based on the detected sequence of proto hand poses.

8. A computer-implemented method comprising:

providing, by one or more processors, to a user an XR user interface of an XR system, the XR user interface comprising a virtual object displayed to the user;

determining, by the one or more processors, a pinch location of a pinch hand pose being made by the user using one hand of the user;

scaling, by the one or more processors, the virtual object based on the pinch hand pose of the one hand using the pinch location and a virtual object center point of the virtual object by performing operations comprising:

generating a virtual object vector based on a center point of the virtual object and the pinch location;

generating a pinch vector based on the center point of the virtual object and a virtual object pinch collider of the virtual object;

determining a scaling vector based on a projection of the pinch vector onto the virtual object vector; and

adjusting the scale of the virtual object based on a ratio of a length of the scaling vector and a length of the virtual object vector; and

redisplaying, by the one or more processors, the scaled virtual object to the user in the XR user interface.

9. The computer-implemented method of claim 8 , wherein determining a pinch location comprises:

capturing, by the one or more processors, using one or more cameras of the XR system, tracking video frame data;

determining, by the one or more processors, hand-tracking data based on the tracking video frame data; and

determining, by the one or more processors, the pinch location based on the hand-tracking data.

10. The computer-implemented method of claim 9 , wherein determining the pinch location based on the hand-tracking data comprises:

determining, by the one or more processors, a pinch location collider based on skeletal model data of the hand-tracking data; and

determining, by the one or more processors, the pinch location based on the pinch location collider.

11. The computer-implemented method of claim 8 , further comprising:

detecting, by the one or more processors, that the pinch hand pose is still being held by the user after redisplaying the scaled virtual object; and

in response to detecting that the pinch hand pose is still being held, continuing, by the one or more processors, to scale the virtual object based on the scaling vector.

12. The computer-implemented method of claim 8 , wherein the XR system comprises a head-wearable apparatus.

13. The computer-implemented method of claim 8 , wherein determining the pinch location comprises:

detecting an intersection between a pinch location collider and a virtual object pinch collider; and

determining three-dimensional coordinates of the pinch location based on the detected intersection between the pinch location collider and the virtual object pinch collider.

14. The computer-implemented method of claim 8 , wherein determining that the pinch hand pose is no longer being held comprises:

detecting a sequence of proto hand poses as the user moves a fingertip of a forefinger away from a fingertip of a thumb of the one hand making the pinch hand pose; and

determining the pinch hand pose is released based on the detected sequence of proto hand poses.

15. A non-transitory machine-readable storage medium, the machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

providing, to a user, an XR user interface of an XR system, the XR user interface comprising a virtual object displayed to the user;

determining a pinch location of a pinch hand pose being made by the user using one hand of the user;

scaling the virtual object based on the pinch hand pose of the one hand using the pinch location and a virtual object center point of the virtual object by performing operations comprising:

generating a virtual object vector based on a center point of the virtual object and the pinch location;

generating a pinch vector based on the center point of the virtual object and a virtual object pinch collider of the virtual object;

determining a scaling vector based on a projection of the pinch vector onto the virtual object vector; and

adjusting the scale of the virtual object based on a ratio of a length of the scaling vector and a length of the virtual object vector; and

redisplaying the scaled virtual object to the user in the XR user interface.

16. The non-transitory machine-readable storage medium of claim 15 , wherein determining a pinch location comprises:

capturing, using one or more cameras of the XR system, tracking video frame data;

determining hand-tracking data based on the tracking video frame data; and

determining the pinch location based on the hand-tracking data.

17. The non-transitory machine-readable storage medium of claim 16 , wherein determining the pinch location based on the hand-tracking data comprises:

determining a pinch location collider based on skeletal model data of the hand-tracking data; and

determining the pinch location based on the pinch location collider.

18. The non-transitory machine-readable storage medium of claim 15 , wherein the XR system comprises a head-wearable apparatus.

19. The non-transitory machine-readable storage medium of claim 15 , wherein determining the pinch location comprises:

detecting an intersection between a pinch location collider and a virtual object pinch collider; and

determining three-dimensional coordinates of the pinch location based on the detected intersection between the pinch location collider and the virtual object pinch collider.

20. The non-transitory machine-readable storage medium of claim 15 , wherein determining that the pinch hand pose is no longer being held comprises:

detecting a sequence of proto hand poses as the user moves a fingertip of a forefinger away from a fingertip of a thumb of the one hand making the pinch hand pose; and

determining the pinch hand pose is released based on the detected sequence of proto hand poses.

Assignments (1)

Continuity (1)

References Cited (102)