IP Library Granted Patent US 10,475,253
Granted Patent B2
US 10,475,253 · App. 16/145,045 · Granted Nov 12, 2019

Devices, methods, and graphical user interfaces for system-wide behavior for 3D models

Inventors: Giancarlo Yerkes (Menlo Park, CA); Stephen O. Lemay (San Francisco, CA); Jeffrey M. Faulkner (San Francisco, CA); Grant R. Paul (San Francisco, CA); Lisa K. Forssell (Palo Alto, CA); Bradley W. Griffin (Berkeley, CA)
Assignee: APPLE INC.
G06T19/20G06F3/04815G06F3/04845G06F3/04883G06T7/70G06T7/80G06T19/003G06T19/006H04M1/72522G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,475,253
App. No.
16/145,045
Filed
Sep 27, 2018
Granted
Nov 12, 2019
Kind
B2
Examiner
SHENG, XIN
Art Unit
2611
USPC
345/633
Abstract

A computer system having a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors receives a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras. In response to receiving the request, in accordance with a determination that calibration criteria are not met, a calibration user interface object is displayed. In response to detecting a change in attitude of the one or more cameras in the physical environment, at least one display parameter of the calibration user interface object is adjusted in accordance with the detected change. In response to detecting that the calibration criteria are met, the calibration user interface object ceases to be displayed.

Claims (109)

1. A method, comprising:

at a device having a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors for detecting changes in attitude of the device including the one or more cameras:

receiving a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detecting that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, ceasing to display the calibration user interface object.

2. The method of claim 1 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

3. The method of claim 2 , including:

displaying the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

4. The method of claim 2 , including:

displaying the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

5. The method of claim 1 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

6. The method of claim 1 , including:

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

7. The method of claim 1 , including:

displaying a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

8. The method of claim 1 , including:

in response to detecting that the calibration criteria are met, displaying a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

9. The method of claim 1 , including:

in response to receiving the request to display the augmented reality view of the physical environment:

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, displaying an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

10. The method of claim 1 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

11. The method of claim 1 , including:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgoing moving the calibration user interface object based on the second type of movement.

12. The method of claim 1 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

13. The method of claim 1 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

14. The method of claim 1 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.

15. A computer system, comprising:

a display generation component;

one or more input devices;

one or more cameras;

one or more attitude sensors;

one or more processors; and

memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detecting that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, ceasing to display the calibration user interface object.

16. The computer system of claim 15 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

17. The computer system of claim 16 , wherein the one or more programs include instructions for:

displaying the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

18. The computer system of claim 16 , wherein the one or more programs include instructions for:

displaying the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

19. The computer system of claim 15 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

20. The computer system of claim 15 , wherein the one or more programs include instructions for:

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

21. The computer system of claim 15 , wherein the one or more programs include instructions for:

displaying a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

22. The computer system of claim 15 , wherein the one or more programs include instructions for:

in response to detecting that the calibration criteria are met, displaying a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

23. The computer system of claim 15 , wherein the one or more programs include instructions for:

in response to receiving the request to display the augmented reality view of the physical environment:

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, displaying an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

24. The computer system of claim 15 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

25. The computer system of claim 15 , wherein the one or more programs include instructions for:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgoing moving the calibration user interface object based on the second type of movement.

26. The computer system of claim 15 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

27. The computer system of claim 15 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

28. The computer system of claim 15 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.

29. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computer system with a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors, cause the computer system to:

receive a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, display the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detect that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, cease to display the calibration user interface object.

30. The non-transitory computer readable storage medium of claim 29 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

31. The non-transitory computer readable storage medium of claim 30 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

32. The non-transitory computer readable storage medium of claim 30 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

33. The non-transitory computer readable storage medium of claim 29 , wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

34. The non-transitory computer readable storage medium of claim 29 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to receiving the request to display the augmented reality view of the physical environment, display the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

35. The non-transitory computer readable storage medium of claim 29 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

36. The non-transitory computer readable storage medium of claim 29 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to detecting that the calibration criteria are met, display a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

37. The non-transitory computer readable storage medium of claim 29 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to receiving the request to display the augmented reality view of the physical environment:

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, display an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

38. The non-transitory computer readable storage medium of claim 29 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

39. The non-transitory computer readable storage medium of claim 29 , wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

while displaying the calibration user interface object, detect, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgo moving the calibration user interface object based on the second type of movement.

40. The non-transitory computer readable storage medium of claim 29 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

41. The non-transitory computer readable storage medium of claim 29 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

42. The non-transitory computer readable storage medium of claim 29 , wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2019
From: YERKES, GIANCARLO; LEMAY, STEPHEN O.; FAULKNER, JEFFREY M.; PAUL, GRANT R.; FORSSELL, LISA K.; GRIFFIN, BRADLEY W.
To: APPLE INC.
Reel/Frame 047961/0606 →
Continuity (3)
Provisional Application 62679951 · Jun 3, 2018
Provisional Application 62621529 · Jan 24, 2018
Related Publication 20190228582A1 · Jul 25, 2019
Cited By (3)
US 12,608,115 US 12,633,044 US 12,711,724