Granted Patent B1

US 11,531,402 · App. 17/589,065 · Granted Dec 20, 2022

Inventor: Karen Stolzenberg (Venice, CA)

Assignee: Snap Inc.

G06F3/017G02B27/0101G02B27/017G06F3/04815G06F3/04847G06T19/006G06V20/46G06V40/10G06V40/28G02B2027/014G02B2027/0138G02B2027/0178

Abstract

Claims (85)

1. A method of controlling a graphical element in response to bimanual gestures detected with an eyewear device, the eyewear device comprising a camera system, an image processing system, a localization system, and a display, the method comprising:

presenting on the display one or more graphical elements at a first position relative to a display plane;

capturing frames of video data with the camera system;

detecting a first hand in the captured frames of video data with the image processing system;

defining, with the image processing system, a first input plane associated with a surface of the detected first hand, wherein the first input plane is defined relative to the display plane;

determining, with the localization system, a current eyewear location relative to the defined first input plane;

calculating a correlation between the defined first input plane and the display plane, in accordance with the current eyewear location;

detecting, with the image processing system, a series of bimanual hand shapes comprising a series of current fingertip locations associated with a fingertip of a second hand, relative to the defined first input plane;

presenting on the display a first movable element at a current first element position, in accordance with both a current fingertip location and the calculated correlation;

determining, with the image processing system, whether the detected series of bimanual hand shapes matches a predefined hand gesture selected from a plurality of predefined hand gestures; and

executing a selecting action in accordance with both the matching predefined hand gesture and the current first element position.

2. The method of claim 1 , further comprising:

detecting a subsequent series of bimanual hand shapes;

presenting on the display a subsequent movable element in accordance with the detected subsequent series of bimanual hand shapes and the calculated correlation;

determining whether the detected subsequent series of bimanual hand shapes matches a subsequent predefined hand gesture selected from the plurality of predefined hand gestures; and

executing a subsequent selecting action in accordance with the matching subsequent predefined hand gesture and the current first element position.

3. The method of claim 1 , wherein the surface of the detected first hand is a region selected from the group consisting of at least a portion of: a dorsal surface, a palmar surface, a hypothenar eminence, and a thenar eminence.

4. The method of claim 1 , wherein the first movable element comprises a cursor presented on the display at the current first element position, and

wherein the step of executing a selecting action comprises selecting a first selectable graphical element nearest the current first element position.

5. The method of claim 1 , wherein the first movable element comprises a slider presented on the display at the current first element position, and

wherein the step of executing a selecting action comprises releasably setting the slider near the current first element position.

6. The method of claim 1 , wherein the step of executing a selecting action further comprises:

establishing, relative to the defined first input plane, a map comprising a plurality of keys;

selecting a first key in accordance with the current fingertip location relative to the map; and

presenting on the display a first icon associated with the selected first key.

7. The method of claim 1 , wherein the plurality of predefined hand gestures comprises a gesture selected from the group consisting of:

a pressing gesture characterized by the identified current fingertip location moving, relative to the defined first input plane, downward from an initial depth to within a predefined pressing depth for at least a minimum pressing duration, and then upward to at least the initial depth; and

a tapping gesture characterized by the identified current fingertip location moving, relative to the defined first input plane, upward from an initial depth to within a predefined lifted depth, downward to at least the initial depth, and then upward to within a predefined return depth.

8. The method of claim 1 , wherein the step of presenting on the display a first movable element at a current first element position, in accordance with a current fingertip location further comprises:

defining the current fingertip location to include a depth orthogonal to the defined first input plane;

determining whether the depth exceeds a predefined depth threshold above or below the defined first input plane;

in response to determining that the depth exceeds the predefined depth threshold, persistently presenting the first movable element at a recent current first element position.

9. A bimanual control system, comprising:

an eyewear device comprising a processor, a memory, a camera system, an image processing system, a localization system, and a display;

programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:

present on the display one or more graphical elements at a first position relative to a display plane;

capture frames of video data with the camera system;

detect a first hand in the captured frames of video data with the image processing system;

define, with the image processing system, a first input plane associated with a surface of the detected first hand, wherein the first input plane is defined relative to the display plane;

determine, with the localization system, a current eyewear location relative to the defined first input plane;

calculate a correlation between the defined first input plane and the display plane, in accordance with the current eyewear location;

detect, with the image processing system, a series of bimanual hand shapes comprising a series of current fingertip locations associated with a fingertip of a second hand, relative to the defined first input plane;

present on the display a first movable element at a current first element position, in accordance with both a current fingertip location and the calculated correlation;

determine, with the image processing system, whether the detected series of bimanual hand shapes matches a predefined hand gesture selected from a plurality of predefined hand gestures; and

execute a selecting action in accordance with both the matching predefined hand gesture and the current first element position.

10. The bimanual control system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

detect a subsequent series of bimanual hand shapes;

present on the display a subsequent movable element in accordance with the detected subsequent series of bimanual hand shapes and the calculated correlation;

determine whether the detected subsequent series of bimanual hand shapes matches a subsequent predefined hand gesture selected from the plurality of predefined hand gestures; and

execute a subsequent selecting action in accordance with the matching subsequent predefined hand gesture and the current first element position.

11. The bimanual control system of claim 9 , wherein the surface of the detected first hand is a region selected from the group consisting of at least a portion of: a dorsal surface, a palmar surface, a hypothenar eminence, and a thenar eminence.

12. The bimanual control system of claim 9 , wherein the first movable element comprises a cursor presented on the display at the current first element position, and

wherein the selecting action comprises selecting a first selectable graphical element nearest the current first element position.

13. The bimanual control system of claim 9 , wherein the first movable element comprises a slider presented on the display at the current first element position, and

wherein the selecting action comprises releasably setting the slider near the current first element position.

14. The bimanual control system of claim 9 , wherein the function to execute a selecting action further comprises functions to:

establish, relative to the defined first input plane, a map comprising a plurality of keys;

select a first key in accordance with the current fingertip location relative to the map; and

present on the display a first icon associated with the selected first key.

15. The bimanual control system of claim 9 , wherein the plurality of predefined hand gestures comprises a gesture selected from the group consisting of:

a pressing gesture characterized by the identified current fingertip location moving, relative to the defined first input plane, downward from an initial depth to within a predefined pressing depth for at least a minimum pressing duration, and then upward to at least the initial depth; and

a tapping gesture characterized by the identified current fingertip location moving, relative to the defined first input plane, upward from an initial depth to within a predefined lifted depth, downward to at least the initial depth, and then upward to within a predefined return depth.

16. The bimanual control system of claim 9 , wherein the function to present on the display a first movable element at a current first element position, in accordance with a current fingertip location further comprises functions to:

define the current fingertip location to include a depth orthogonal to the defined first input plane;

determine whether the depth exceeds a predefined depth threshold above or below the defined first input plane;

in response to determining that the depth exceeds the predefined depth threshold, persistently present the first movable element at a recent current first element position.

17. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

presenting on a display one or more graphical elements at a first position relative to a display plane, wherein the display is coupled to an eyewear device comprising a processor, a memory, a camera system, a localization system, and an image processing system;

capturing frames of video data with the camera system;

detecting a first hand in the captured frames of video data with the image processing system;

defining, with the image processing system, a first input plane associated with a surface of the detected first hand, wherein the first input plane is defined relative to the display plane;

determining, with the localization system, a current eyewear location relative to the defined first input plane;

calculating a correlation between the defined first input plane and the display plane, in accordance with the current eyewear location;

detecting, with the image processing system, a series of bimanual hand shapes comprising a series of current fingertip locations associated with a fingertip of a second hand, relative to the defined first input plane;

presenting on the display a first movable element at a current first element position, in accordance with both a current fingertip location and the calculated correlation;

determining, with the image processing system, whether the detected series of bimanual hand shapes matches a predefined hand gesture selected from a plurality of predefined hand gestures; and

executing a selecting action in accordance with both the matching predefined hand gesture and the current first element position.

18. The non-transitory computer-readable medium storing program code of claim 17 , wherein the first movable element comprises a cursor presented on the display at the current first element position, and

wherein the step of executing a selecting action comprises selecting a first selectable graphical element nearest the current first element position.

19. The non-transitory computer-readable medium storing program code of claim 17 , wherein the first movable element comprises a slider presented on the display at the current first element position, and

wherein the step of executing a selecting action comprises releasably setting the slider near the current first element position.

20. The non-transitory computer-readable medium storing program code of claim 17 , wherein the step of executing a selecting action further comprises:

establishing, relative to the defined first input plane, a map comprising a plurality of keys;

selecting a first key in accordance with the current fingertip location relative to the map; and

presenting on the display a first icon associated with the selected first key.

Assignments (1)

Continuity (1)

Cited By (144)

US 12,200,399 US 12,204,105 US 12,204,693 US 12,204,807 US 12,210,691 US 12,210,736 US 12,212,536 US 12,216,823 US 12,216,827 US 12,223,402 US 12,229,342 US 12,231,437 US 12,236,148 US 12,236,512 US 12,236,515 US 12,236,546 US 12,242,706 US 12,265,663 US 12,265,664 US 12,266,057 US 12,271,517 US 12,271,532 US 12,271,586 US 12,282,607 US 12,283,020 US 12,287,921 US 12,288,285 US 12,288,298 US 12,299,798 US 12,307,287 US 12,307,580 US 12,314,472 US 12,314,485 US 12,326,980 US 12,327,302 US 12,332,438 US 12,333,083 US 12,333,658 US 12,348,855 US 12,353,628 US 12,353,632 US 12,353,840 US 12,357,911 US 12,360,663 US 12,361,648 US 12,361,661 US 12,361,664 US 12,367,616 US 12,373,096 US 12,380,618 US 12,382,188 US 12,393,613 US 12,401,772 US 12,405,672 US 12,405,675 US 12,411,555 US 12,412,347 US 12,423,910 US 12,429,953 US 12,432,441 US 12,437,491 US 12,437,494 US 12,443,270 US 12,443,273 US 12,443,325 US 12,443,335 US 12,450,851 US 12,452,384 US 12,456,271 US 12,461,641 US 12,468,395 US 12,468,396 US 12,469,090 US 12,469,219 US 12,475,635 US 12,482,171 US 12,494,078 US 12,498,783 US 12,498,793 US 12,501,150 US 12,511,847 US 12,517,521 US 12,518,490 US 12,524,142 US 12,524,146 US 12,524,956 US 12,524,977 US 12,525,139 US 12,530,086 US 12,530,088 US 12,531,959 US 12,535,931 US 12,536,762 US 12,541,280 US 12,547,295 US 12,547,309 US 12,566,532 US 12,572,216 US 12,572,261 US 12,573,077 US 12,573,100 US 12,596,472 US 12,608,115 US 12,608,126 US 12,608,877 US 12,608,890 US 12,614,567 US 12,620,187 US 12,625,608 US 12,626,455 US 12,632,120 US 12,632,157 US 12,632,170 US 12,632,984 US 12,633,044 US 12,638,951 US 12,641,190 US 12,650,732 US 12,656,875 US 12,656,876 US 12,657,753 US 12,663,907 US 12,663,958 US 12,664,667 US 12,670,643 US 12,670,674 US 12,675,168 US 12,675,169 US 12,675,958 US 12,681,587 US 12,682,489 US 12,687,922 US 12,688,660 US 12,693,770 US 12,700,187 US 12,705,009 US 12,710,807 US 12,712,835 US 12,726,454 US 12,730,524 US 12,731,341 US 12,731,342 US 12,737,053 US 12,737,093