IP Library › Granted Patent US 11,340,756
Granted Patent B2
US 11,340,756 · App. 17/030,200 · Granted May 24, 2022

Devices, methods, and graphical user interfaces for interacting with three-dimensional environments

Inventors: Jeffrey M. Faulkner (San Francisco, CA); Israel Pastrana Vicente (Santa Clara, CA); Pol Pla I. Conesa (Belmont, CA); Stephen O. Lemay (Palo Alto, CA); Robert T. Tilton (Los Angeles, CA); William A. Sorrentino, III (Mill Valley, CA); Kristi E. S. Bauerly (Los Gatos, CA)
Assignee: APPLE INC.
G06F3/04815G06F3/017G06F3/04817G06T7/70G06T15/00G06T2200/24
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,340,756
App. No.
17/030,200
Filed
Sep 23, 2020
Granted
May 24, 2022
Kind
B2
Art Unit
2613
USPC
345/419
Abstract

While displaying a three-dimensional environment, a computer system detects a hand at a first position that corresponds to a portion of the three-dimensional environment. In response to detecting the hand at the first position: in accordance with a determination that the hand is being held in a first predefined configuration, the computer system displays a visual indication of a first operation context for gesture input using hand gestures in the three-dimensional environment; and in accordance with a determination that the hand is not being held in the first predefined configuration, the computer system forgoes display of the visual indication.

Claims (89)

1. A method, comprising:

at a computing system including a display generation component and one or more input devices:

displaying a view of a three-dimensional environment;

while displaying the three-dimensional environment, detecting a hand at a first position that corresponds to a portion of the three-dimensional environment;

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment:

in accordance with a determination that the hand is being held in a first predefined configuration;

displaying a visual indication of a first operation context for gesture input using hand gestures in the three-dimensional environment;

while displaying the visual indication, detecting a change in configuration of the hand from the first predefined configuration to a configuration that does not meet input criteria;

in response to detecting the change in the configuration of the hand from the first predefined configuration to the configuration that does not meet the input criteria, ceasing to display the visual indication;

after ceasing to display the visual indication, detecting a change in the configuration of the hand to the first predefined configuration; and

in response to detecting the change in the configuration of the hand to the first predefined configuration, redisplaying the visual indication; and

in accordance with a determination that the hand is not being held in the first predefined configuration, forgoing display of the visual indication of the first operation context for gesture input using hand gestures in the three-dimensional environment.

2. The method of claim 1 , wherein the visual indication of the first operation context for gesture input using hand gestures is displayed at a location in the portion of the three-dimensional environment that corresponds to the first position.

3. The method of claim 2 , including:

while displaying the visual indication in the portion of the three-dimensional environment, detecting a change in position of the hand from the first position to a second position; and

in response to detecting the change in position of the hand from the first position to the second position, changing the displayed location of the visual indication in accordance with the detected change in position of the hand.

4. The method of claim 1 , wherein the visual indication includes one or more user interface objects.

5. The method of claim 4 , wherein the one or more user interface objects include a plurality of application launch icons, wherein activation of a respective one of the application launch icons causes an operation associated with a corresponding application to be performed.

6. The method of claim 1 , including:

while displaying the visual indication, detecting a change in a configuration of the hand from the first predefined configuration to a second predefined configuration;

in response to detecting the change in the configuration of the detected hand from the first predefined configuration to the second predefined configuration, displaying a first collection of user interface objects, wherein activation of a respective user interface object of the first collection of user interface objects causes an operation associated with the respective user interface object to be performed.

7. The method of claim 6 , including:

while displaying the visual indication, determining, during a time window whether movement of the hand meets interaction criteria; and

in accordance with the determination that movement of the hand does not meet the interaction criteria during the time window, ceasing to display the visual indication.

8. The method of claim 1 , including:

while displaying the visual indication, detecting a change in the configuration of the hand from the first predefined configuration to a second predefined configuration that meets the input criteria; and,

in response to detecting the change in the configuration of the hand from the first predefined configuration to the second predefined configuration that meets the input criteria, adjusting the visual indication.

9. The method of claim 1 , including:

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment, in accordance with a determination that the hand is not being held in the first predefined configuration, performing an operation distinct from displaying the visual indication of the first operation context for gesture input using hand gestures.

10. A computing system, comprising:

a display generation component;

one or more input devices;

one or more processors; and

memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for:

displaying a view of a three-dimensional environment;

while displaying the three-dimensional environment, detecting a hand at a first position that corresponds to a portion of the three-dimensional environment;

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment:

in accordance with a determination that the hand is being held in a first predefined configuration;

displaying a visual indication of a first operation context for gesture input using hand gestures in the three-dimensional environment;

while displaying the visual indication, detecting a change in configuration of the hand from the first predefined configuration to a configuration that does not meet input criteria;

in response to detecting the change in the configuration of the hand from the first predefined configuration to the configuration that does not meet the input criteria, ceasing to display the visual indication;

after ceasing to display the visual indication, detecting a change in the configuration of the hand to the first predefined configuration; and

in response to detecting the change in the configuration of the hand to the first predefined configuration, redisplaying the visual indication; and

in accordance with a determination that the hand is not being held in the first predefined configuration, forgoing display of the visual indication of the first operation context for gesture input using hand gestures in the three-dimensional environment.

11. The computing system of claim 10 , wherein the visual indication of the first operation context for gesture input using hand gestures is displayed at a location in the portion of the three-dimensional environment that corresponds to the first position.

12. The computing system of claim 11 , wherein the one or more programs includes instructions for:

while displaying the visual indication in the portion of the three-dimensional environment, detecting a change in position of the hand from the first position to a second position; and

in response to detecting the change in position of the hand from the first position to the second position, changing the displayed location of the visual indication in accordance with the detected change in position of the hand.

13. The computing system of claim 10 , wherein the visual indication includes one or more user interface objects.

14. The computing system of claim 13 , wherein the one or more user interface objects include a plurality of application launch icons, wherein activation of a respective one of the application launch icons causes an operation associated with a corresponding application to be performed.

15. The computing system of claim 10 , wherein the one or more programs includes instructions for:

while displaying the visual indication, detecting a change in a configuration of the hand from the first predefined configuration to a second predefined configuration;

in response to detecting the change in the configuration of the detected hand from the first predefined configuration to the second predefined configuration, displaying a first collection of user interface objects, wherein activation of a respective user interface object of the first collection of user interface objects causes an operation associated with the respective user interface object to be performed.

16. The computing system of claim 15 , wherein the one or more programs includes instructions for:

while displaying the visual indication, determining, during a time window whether movement of the hand meets interaction criteria; and

in accordance with the determination that movement of the hand does not meet the interaction criteria during the time window, ceasing to display the visual indication.

17. The computing system of claim 10 , wherein the one or more programs includes instructions for:

while displaying the visual indication, detecting a change in the configuration of the hand from the first predefined configuration to a second predefined configuration that meets the input criteria; and,

in response to detecting the change in the configuration of the hand from the first predefined configuration to the second predefined configuration that meets the input criteria, adjusting the visual indication.

18. The computing system of claim 10 , wherein the one or more programs include instructions for:

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment, in accordance with a determination that the hand is not being held in the first predefined configuration, performing an operation distinct from displaying the visual indication of the first operation context for gesture input using hand gestures.

19. A non-transitory computer readable storage medium storing executable instructions that, when executed by a computer system with one or more processors and a display generation component, cause the computer system to:

display a view of a three-dimensional environment;

while displaying the three-dimensional environment, detect a hand at a first position that corresponds to a portion of the three-dimensional environment; and

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment:

in accordance with a determination that the hand is being held in a first predefined configuration;

display a visual indication of a first operation context for gesture input using hand gestures in the three-dimensional environment;

while displaying the visual indication, detect a change in configuration of the hand from the first predefined configuration to a configuration that does not meet input criteria;

in response to detecting the change in the configuration of the hand from the first predefined configuration to the configuration that does not meet the input criteria, cease to display the visual indication;

after ceasing to display the visual indication, detect a change in the configuration of the hand to the first predefined configuration; and

in response to detecting the change in the configuration of the hand to the first predefined configuration, redisplay the visual indication; and

in accordance with a determination that the hand is not being held in the first predefined configuration, forgo display of the visual indication of the first operation context for gesture input using hand gestures in the three-dimensional environment.

20. The non-transitory computer readable storage medium of claim 19 , wherein the visual indication of the first operation context for gesture input using hand gestures is displayed at a location in the portion of the three-dimensional environment that corresponds to the first position.

21. The non-transitory computer readable storage medium of claim 20 , including instructions that, when executed by the computer system, cause the computer system to:

while displaying the visual indication in the portion of the three-dimensional environment, detect a change in position of the hand from the first position to a second position; and

in response to detecting the change in position of the hand from the first position to the second position, change the displayed location of the visual indication in accordance with the detected change in position of the hand.

22. The non-transitory computer readable storage medium of claim 19 , wherein the visual indication includes one or more user interface objects.

23. The non-transitory computer readable storage medium of claim 22 , wherein the one or more user interface objects include a plurality of application launch icons, wherein activation of a respective one of the application launch icons causes an operation associated with a corresponding application to be performed.

24. The non-transitory computer readable storage medium of claim 19 , including instructions that, when executed by the computer system, cause the computer system to:

while displaying the visual indication, detect a change in a configuration of the hand from the first predefined configuration to a second predefined configuration;

in response to detecting the change in the configuration of the detected hand from the first predefined configuration to the second predefined configuration, display a first collection of user interface objects, wherein activation of a respective user interface object of the first collection of user interface objects causes an operation associated with the respective user interface object to be performed.

25. The non-transitory computer readable storage medium of claim 24 , including instructions that, when executed by the computer system, cause the computer system to:

while displaying the visual indication, determine, during a time window whether movement of the hand meets interaction criteria; and

in accordance with the determination that movement of the hand does not meet the interaction criteria during the time window, cease to display the visual indication.

26. The non-transitory computer readable storage medium of claim 19 , including instructions that, when executed by the computer system, cause the computer system to:

while displaying the visual indication, detect a change in the configuration of the hand from the first predefined configuration to a second predefined configuration that meets the input criteria; and,

in response to detecting the change in the configuration of the hand from the first predefined configuration to the second predefined configuration that meets the input criteria, adjust the visual indication.

27. The non-transitory computer readable storage medium of claim 19 , including instructions that, when executed by the computer system, cause the computer system to:

in response to detecting the hand at the first position that corresponds to the portion of the three-dimensional environment, in accordance with a determination that the hand is not being held in the first predefined configuration, perform an operation distinct from displaying the visual indication of the first operation context for gesture input using hand gestures.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 18, 2025
From: ANTON, PETER D.
To: APPLE INC.
Reel/Frame 070882/0298 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 21, 2022
From: TILTON, ROBERT T.; SORRENTINO, WILLIAM A., III; BAUERLY, KRISTI E.S.
To: APPLE INC.
Reel/Frame 058723/0730 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2021
From: FAULKNER, JEFFREY M.; PASTRANA VICENTE, ISRAEL; CONESA, POL PLA I; LEMAY, STEPHEN O.
To: APPLE INC.
Reel/Frame 055200/0652 →
Continuity (2)
Provisional Application 62907480 · Sep 27, 2019
Related Publication 20210096726A1 · Apr 1, 2021
Cited By (54)
US 12,229,342 US 12,236,512 US 12,236,515 US 12,236,546 US 12,242,706 US 12,272,005 US 12,283,020 US 12,287,921 US 12,288,285 US 12,299,251 US 12,300,060 US 12,307,580 US 12,315,091 US 12,321,563 US 12,321,666 US 12,333,083 US 12,353,672 US 12,394,167 US 12,405,704 US 12,437,494 US 12,443,273 US 12,443,286 US 12,443,335 US 12,456,271 US 12,461,641 US 12,475,635 US 12,511,009 US 12,511,847 US 12,524,142 US 12,524,977 US 12,535,931 US 12,536,762 US 12,541,280 US 12,541,281 US 12,566,532 US 12,572,261 US 12,596,472 US 12,608,115 US 12,608,877 US 12,608,890 US 12,620,187 US 12,625,608 US 12,632,170 US 12,657,958 US 12,663,853 US 12,663,875 US 12,663,907 US 12,670,674 US 12,675,958 US 12,681,587 US 12,687,922 US 12,688,660 US 12,737,093 US 12,749,272