Granted Patent B2

US 12,373,096 · App. 17/804,818 · Granted Jul 29, 2025

Inventors: Sharon Moll (Lachen, CH); Dawei Zhang (Seattle, WA)

Assignee: Snap Inc.

G06F3/04886G06F3/011G06F3/04842

Abstract

Claims (80)

1. A computer-implemented method comprising:

while continuously capturing, by one or more processors of an Augmented Realty (AR) system using one or more cameras of the AR system, video frame data of one or more hands of a user using the AR system, performing operations comprising:

detecting, by the one or more processors based on the video frame data, a start text entry gesture made by a user of the AR system;

generating, by the one or more processors, in response to detecting the start text entry gesture, a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys;

providing to the user, by the one or more processors, the virtual keyboard user interface using a display of the AR system;

detecting, by the one or more processors based on the video frame data, an enter text gesture made by the user using a free hand of the one or more hands;

detecting, by the one or more processors based on the video frame data, a hold of the enter text gesture by the user;

in response to detecting the hold of the enter text gesture by the user, performing operations comprising:

determining, by the one or more processors based on the video frame data, a continuous motion gesture by the user passing through one or more virtual keys of the plurality of virtual keys, the continuous motion gesture being made by the user using a text entry hand of the one or more hands separate from the free hand while the free hand holds the enter text gesture;

detecting, by the one or more processors based on the video frame data, a release of the enter text gesture being held by the free hand; and

in response to detecting the release of the enter text gesture by the free hand, performing operations comprising:

generating, by the one or more processors, entered text data using the continuous motion gesture; and

providing, by the one or more processors, the entered text data to the user using the display of the AR system;

detecting, by the one or more processors using the video frame data, an end text entry gesture made by the user; and

in response to detecting the end text entry gesture, closing, by the one or more processors, the virtual keyboard user interface.

2. The computer-implemented method of claim 1 , wherein the detecting, by the one or more processors based on the video frame data, the start text entry gesture made by the user of the AR system further comprises:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data; and

determining the start text entry gesture on a basis of categorizing the one or more skeletal models using artificial intelligence methodologies and a gesture model previously generated using machine learning methodologies.

3. The computer-implemented method of claim 1 , wherein the detecting, by the one or more processors based on the video frame data, the start text entry gesture made by the user of the AR system further comprises:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data;

comparing the one or more skeletal models to previously generated gesture skeletal models; and

determining the start text entry gesture based on the comparing of the one or more skeletal models with the gesture skeletal models.

4. The computer-implemented method of claim 1 , wherein the AR system comprises a head-worn device.

5. The computer-implemented method of claim 1 , further comprising: dynamically adjusting a depth location of the virtual keyboard user interface to maintain a fixed distance from one or more eyes of the user as determined by tracking data generated by the one or more processors based on the video frame data.

6. A computing apparatus of an AR system, comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the computing apparatus to perform operations comprising:

while continuously capturing, using one or more cameras of an AR system, video frame data of one or more hands of a user using the AR system, performing operations comprising:

detecting, based on the video frame data, a start text entry gesture made by a user of the AR system;

generating, in response to detecting the start text entry gesture, a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys;

providing to the user, the virtual keyboard user interface using a display of the AR system;

detecting, based on the video frame data, an enter text gesture made by the user using a free hand of the one or more hands;

detecting, based on the video frame data, a hold of the enter text gesture by the user;

in response to detecting the hold of the enter text gesture by the user, performing operations comprising:

determining, based on the video frame data, a continuous motion gesture by the user passing through one or more virtual keys of the plurality of virtual keys, the continuous motion gesture being made by the user using a text entry hand of the one or more hands separate from the free hand while the free hand holds the enter text gesture;

detecting, based on the video frame data, a release of the enter text gesture being held by the free hand; and

in response to detecting the release of the enter text gesture by the free hand, performing operations comprising:

generating entered text data using the continuous motion gesture; and

providing the entered text data to the user using the display of the AR system;

detecting, based on the video frame data, an end text entry gesture made by the user; and

in response to detecting the end text gesture, closing the virtual keyboard user interface.

7. The computing apparatus of the AR system of claim 6 , wherein the detecting, based on the video frame data, the start text entry gesture made by the user of the AR system, further comprises:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data; and

determining the start text entry gesture on a basis of categorizing the one or more skeletal models using artificial intelligence methodologies and a gesture model previously generated using machine learning methodologies.

8. The computing apparatus of the AR system of claim 6 , wherein the detecting, based on the video frame data, the start text entry gesture made by the user of the AR system further comprises:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data;

comparing the one or more skeletal models to previously generated gesture skeletal models; and

determining the start text entry gesture based on the comparing of the one or more skeletal models with the gesture skeletal models.

9. The computing apparatus of the AR system of claim 6 , wherein the AR system comprises a head-worn device.

10. The computing apparatus of the AR system of claim 6 , wherein the instructions further cause the computing apparatus to perform operations comprising:

dynamically adjusting a depth location of the virtual keyboard user interface to maintain a fixed distance from one or more eyes of the user as determined by tracking data generated by the one or more processors based on the video frame data.

11. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer of an AR system, cause the computer to perform operations comprising:

while continuously capturing, using one or more cameras of an AR system, video frame data of one or more hands of a user using the AR system, performing operations comprising:

detecting, based on the video frame data, a start text entry gesture made by a user of the AR system;

generating, in response to detecting the start text entry gesture, a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys;

providing to the user, the virtual keyboard user interface using a display of the AR system;

detecting, based on the video frame data, an enter text gesture made by the user using a free hand of the one or more hands;

detecting, based on the video frame data, a hold of the enter text gesture made by the user using the free hand;

in response to detecting the hold of the enter text gesture by the user, performing operations comprising:

determining, based on the video frame data, a continuous motion gesture by the user passing through one or more virtual keys of the plurality of virtual keys, the continuous motion gesture being made by the user using a text entry hand of the one or more hands separate from the free hand while the free hand holds the enter text gesture;

detecting, based on the video frame data, a release of the enter text gesture being held by the free hand; and

in response to detecting the release of the enter text gesture by the free hand, performing operations comprising:

generating entered text data using the continuous motion gesture; and

providing the entered text data to the user using the display of the AR system;

detecting, based on the video frame data, an end text entry gesture made by the user; and

in response to detecting the end text gesture, closing the virtual keyboard user interface.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions causing the computer to perform the operation of detecting, based on the video frame data, the start text entry gesture made by the user of the AR system, further cause the computer to perform operations of:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data; and

determining the start text entry gesture on a basis of categorizing the one or more skeletal models using artificial intelligence methodologies and a gesture model previously generated using machine learning methodologies.

13. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions causing the computer to perform the operation of detecting, based on the video frame data, the start text entry gesture made by the user of the AR system, further cause the computer to perform operations of:

generating tracking data based on the video frame data;

generating one or more skeletal models based on the tracking data;

comparing the one or more skeletal models to previously generated gesture skeletal models; and

determining the start text entry gesture based on the comparing of the one or more skeletal models with the gesture skeletal models.

14. The non-transitory computer-readable storage medium of claim 11 , wherein the AR system comprises a head-worn device.

Assignments (1)

Continuity (1)

References Cited (240)