Granted Patent B1

US 12,572,219 · App. 18/828,952 · Granted Mar 10, 2026

Inventors: Daniel Voros (Budapest, HU); David Retek (Budapest, HU); Gabor Nemeth (Gyor, HU); Levente Torok (Vienna, AT)

Assignee: SNAP INC.

G06F3/017G06V10/26G06V10/40G06V10/764G06V10/806G06V10/82G06V40/28

Abstract

Claims (76)

1 . A method, comprising:

receiving a set of inputs from multiple input sources, the receiving the set of inputs from the multiple input sources comprising:

receiving a set of images;

receiving a set of hand skeleton models; and

receiving a set of occlusion information;

determining a set of input features based on the set of inputs from the multiple input sources;

performing a time window-based aggregation on the set of input features to generate a set of aggregated features;

performing feature extraction, using a set of modular components of a modular classifier network, on the set of aggregated features to generate a set of extracted features;

generating, using a pinch detection head, a probability score indicating a likelihood of an occurrence of a pinch gesture based on the set of extracted features;

determining, using triggering logic, whether a pinch gesture has occurred based at least in part on the probability score; and

providing a pinch detection output based at least in part on the determining.

2 . The method of claim 1 , wherein determining the set of input features based on the set of inputs from the multiple input sources comprises:

determining a set of image features based on the set of images;

determining a set of derived features per frame based on the set of hand skeleton models; and

determining a set of occlusion features based on the set of occlusion information.

3 . The method of claim 1 , wherein performing the time window-based aggregation on the set of input features comprises:

applying polynomial fitting on the set of input features;

determining a set of averages on the set of input features; and

determining a set of statistical descriptors on the set of input features.

4 . The method of claim 1 , wherein performing feature extraction, using the set of modular components of the modular classifier network, on the set of aggregated features comprises:

performing, using a first modular component of the modular classifier network, a first feature extraction process to determine a first set of extracted features;

performing, using a second modular component of the modular classifier network, a second feature extraction process to determine a second set of extracted features;

performing, using a third modular component of the modular classifier network, a third feature extraction process to determine a third set of extracted features; and

performing, using a fourth modular component of the modular classifier network, a feature aggregation process to combine at least some of the first set of extracted features, the second set of extracted features, and the third set of extracted features into a combined set of extracted features.

5 . The method of claim 1 , wherein generating, using the pinch detection head, the probability score indicating the likelihood of the occurrence of the pinch gesture comprises:

combining the set of extracted features, the combined set of extracted features having a reduction in noise and errors compared to the set of extracted features prior to the combining.

6 . The method of claim 1 , wherein determining, using the triggering logic, whether the pinch gesture has occurred comprises:

applying a set of additional rules to the set of extracted features; and

generating information for the pinch detection output based at least in part on the applied set of additional rules and the probability score.

7 . The method of claim 6 , wherein the set of additional rules comprises hand orientation-based feature weighting and velocity-based false positive filtering.

8 . The method of claim 7 , wherein the pinch detection output is provided to an augmented reality application.

9 . The method of claim 8 , wherein the augmented reality application utilizes the pinch detection output to perform an object selection, object manipulation, interface navigation, or zoom control.

10 . A system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

receiving a set of inputs from multiple input sources, the receiving the set of inputs from the multiple input sources comprising:

receiving a set of images;

receiving a set of hand skeleton models; and

receiving a set of occlusion information;

determining a set of input features based on the set of inputs from the multiple input sources;

performing a time window-based aggregation on the set of input features to generate a set of aggregated features;

performing feature extraction, using a set of modular components of a modular classifier network, on the set of aggregated features to generate a set of extracted features;

generating, using a pinch detection head, a probability score indicating a likelihood of an occurrence of a pinch gesture based on the set of extracted features;

determining, using triggering logic, whether a pinch gesture has occurred based at least in part on the probability score; and

providing a pinch detection output based at least in part on the determining.

11 . The system of claim 10 , wherein determining the set of input features based on the set of inputs from the multiple input sources comprises:

determining a set of image features based on the set of images;

determining a set of derived features per frame based on the set of hand skeleton models; and

determining a set of occlusion features based on the set of occlusion information.

12 . The system of claim 10 , wherein performing the time window-based aggregation on the set of input features comprises:

applying polynomial fitting on the set of input features;

determining a set of averages on the set of input features; and

determining a set of statistical descriptors on the set of input features.

13 . The system of claim 10 , wherein performing feature extraction, using the set of modular components of the modular classifier network, on the set of aggregated features comprises:

performing, using a first modular component of the modular classifier network, a first feature extraction process to determine a first set of extracted features;

performing, using a second modular component of the modular classifier network, a second feature extraction process to determine a second set of extracted features;

performing, using a third modular component of the modular classifier network, a third feature extraction process to determine a third set of extracted features; and

performing, using a fourth modular component of the modular classifier network, a feature aggregation process to combine at least some of the first set of extracted features, the second set of extracted features, and the third set of extracted features into a combined set of extracted features.

14 . The system of claim 10 , wherein generating, using the pinch detection head, the probability score indicating the likelihood of the occurrence of the pinch gesture comprises:

combining the set of extracted features, the combined set of extracted features having a reduction in noise and errors compared to the set of extracted features prior to the combining.

15 . The system of claim 10 , wherein determining, using the triggering logic, whether the pinch gesture has occurred comprises:

applying a set of additional rules to the set of extracted features; and

generating information for the pinch detection output based at least in part on the applied set of additional rules and the probability score.

16 . The system of claim 15 , wherein the set of additional rules comprises hand orientation-based feature weighting and velocity-based false positive filtering.

17 . The system of claim 16 , wherein the pinch detection output is provided to an augmented reality application.

18 . A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

receiving a set of inputs from multiple input sources, the receiving the set of inputs from the multiple input sources comprising:

receiving a set of images;

receiving a set of hand skeleton models; and

receiving a set of occlusion information;

determining a set of input features based on the set of inputs from the multiple input sources;

performing a time window-based aggregation on the set of input features to generate a set of aggregated features;

performing feature extraction, using a set of modular components of a modular classifier network, on the set of aggregated features to generate a set of extracted features;

generating, using a pinch detection head, a probability score indicating a likelihood of an occurrence of a pinch gesture based on the set of extracted features;

determining, using triggering logic, whether a pinch gesture has occurred based at least in part on the probability score; and

providing a pinch detection output based at least in part on the determining.

Assignments (3)

References Cited (10)

Cited By (1)

US 12,734,449