IP Library › Granted Patent US 11,818,455

Granted Patent B2

US 11,818,455 · App. 18/107,381 · Granted Nov 14, 2023

Devices, methods, and graphical user interfaces for depth-based annotation

Inventors: Joseph A. Malia (Isle of Wight, GB); Mark K. Hauenstein (London, GB); Praveen Sharma (Brooklyn, NY); Matan Stauber (San Francisco, CA); Julian K. Missig (Burlingame, CA); Jeffrey T. Bernstein (San Francisco, CA); Lukas Robert Tom Girling (Bristol, GB); Matthaeus Krenn (Sunnyvale, CA)

Assignee: APPLE INC.

H04N23/631 G06T11/00 H04N23/633 G06F3/0488 G06T2200/24

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,818,455
App. No.: 18/107,381
Granted: Nov 14, 2023
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A first device sends a request to a second device to initiate a shared annotation session. In response to receiving acceptance of the request, a first prompt to move the first device toward the second device is displayed. In accordance with a determination that connection criteria for the first device and the second device are met, a representation of a field of view of the camera(s) of the first device is displayed in the shared annotation session with the second device. During the shared annotation session, one or more annotations are displayed via the first display generation component and one or more second virtual annotations corresponding to annotation input directed to the respective location in the physical environment by the second device is displayed via the first display generation component, provided that the respective location is included in the field of view of the first set of cameras.

Claims (103)

1. A method, comprising:

at a first device having a first display generation component, a first set of one or more input devices, and a first set of one or more cameras:

sending a request to a remote device to initiate a shared annotation session with a second device that includes a second display generation component, a second set of one or more input devices, and a second set of one or more cameras;

in response to sending the request to initiate the shared annotation session with the second device, receiving an indication of acceptance of the request to initiate the shared annotation session;

in response to receiving the indication of acceptance of the request to initiate the shared annotation session, displaying, via the first display generation component, a first prompt to move the first device toward the second device; and

after displaying the first prompt, in accordance with a determination that connection criteria for the first device and the second device are met, displaying a representation of a field of view of the first set of cameras in the shared annotation session with the second device, wherein:

the connection criteria require that at least a portion of the field of view of the first device and a portion of a field of view of the second device correspond to a same portion of physical environment surrounding the first and second devices, and

during the shared annotation session, one or more first virtual annotations corresponding to annotation input directed to a respective location in the physical environment by the first device is displayed via the first display generation component and one or more second virtual annotations corresponding to annotation input directed to the respective location in the physical environment by the second device is displayed via the first display generation component, provided that the respective location is included in the field of view of the first set of cameras.

2. The method of claim 1 , wherein:

during the shared annotation session, the one or more first virtual annotations corresponding to the annotation input directed to the respective location in the physical environment by the first device is displayed via the second display generation component and the one or more second virtual annotations corresponding to the annotation input directed to the respective location in the physical environment by the second device is displayed via the second display generation component, provided that the respective location is included in the field of view of the second set of cameras.

3. The method of claim 2 , wherein:

during the shared annotation session, the one or more first virtual annotations corresponding to the annotation input directed to the respective location in the physical environment by the first device is displayed via the second display generation component and not via the first display generation component, provided that the respective location is included in the field of view of the second set of cameras and not included in the field of view of the first set of cameras.

4. The method of claim 1 , including:

after displaying the first prompt, in accordance with a determination that the connection criteria for the first device and the second device are met, displaying a notification in the representation of the field of view of the first set of cameras that the shared annotation session with the second device has been established.

5. The method of claim 1 , wherein the first prompt includes an animation illustrating representations of the first and second devices moving toward each other, and the method includes:

after displaying the first prompt, in accordance with a determination that the connection criteria for the first device and the second device are met, replacing display of the first prompt with the representation of the field of view of the first set of cameras.

6. The method of claim 1 , including:

during the shared annotation session:

while displaying, via the first display generation component, the representation of the field of view of the first set of cameras, in accordance with a determination that a first annotation input directed to a portion of the physical environment captured in the field of view of the first set of cameras is received from the second device, displaying an avatar of the second device at a location in the representation of the field of view of the first set of cameras that corresponds to a location to which the first annotation input is directed.

7. The method of claim 1 , including:

during the shared annotation session:

while displaying, via the first display generation component, the representation of the field of view of the first set of cameras, displaying a virtual object in the representation of the field of view of the first set of cameras with a location that corresponds to a current location of the second device, including:

detecting movement of the second device from a first location to a second location in the physical environment; and

displaying movement of the virtual object in the representation of the field of view of the first set of cameras that corresponds to the movement of the second device from the first location to the second location in the physical environment, provided that the first location and the second location are within the field of view of the first set of cameras during the movement of the second device from the first location to the second location in the physical environment.

8. The method of claim 1 , wherein:

during the shared annotation session, the first device displays the representation of the field of view of the first set of cameras via the first display generation component and the second device displays the representation of the field of view of the first set of cameras via the second display generation component.

9. The method of claim 1 , wherein:

during the shared annotation session, the first device displays the representation of the field of view of the first set of cameras via the first display generation component and the second device displays the representation of the field of view of the second set of cameras via the second display generation component.

10. The method of claim 1 , including:

during the shared annotation session, in accordance with a determination that the connection criteria are no longer met, terminating the shared annotation session.

11. The method of claim 1 , including:

during the shared annotation session, in accordance with a determination that an amount of overlap between the field of view of the first set of cameras and the field of view of the second set of cameras is below a first threshold amount, generating, via the first display generation component, a second prompt to move the first device toward the second device.

12. The method of claim 1 , including:

during the shared annotation session, in accordance with a determination that an amount of overlap between the field of view of the first set of cameras and the field of view of the second set of cameras is below a first threshold amount, displaying, via the first display generation component, an indication of a current relative spatial relationship between the first device and the second device.

13. A first device, comprising:

a first display generation component;

one or more input devices;

one or more cameras;

one or more processors; and

memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:

in response to sending the request to initiate the shared annotation session with the second device, receiving an indication of acceptance of the request to initiate the shared annotation session;

14. The first device of claim 13 , wherein:

15. The first device of claim 14 , wherein:

during the shared annotation session, the one or more first virtual annotations corresponding to the annotation input directed to the respective location in the physical environment by the first device is displayed via the second display generation component and not via the first display generation component, provided that the respective location is included in the field of view of the second set of cameras and not included in the field of view of the first set of cameras.

16. The first device of claim 13 , wherein the one or more programs include instructions for:

17. The first device of claim 13 , wherein the first prompt includes an animation illustrating representations of the first and second devices moving toward each other, and the one or more programs include instructions for:

18. The first device of claim 13 , wherein the one or more programs include instructions for:

during the shared annotation session, while displaying, via the first display generation component, the representation of the field of view of the first set of cameras, in accordance with a determination that a first annotation input directed to a portion of the physical environment captured in the field of view of the first set of cameras is received from the second device, displaying an avatar of the second device at a location in the representation of the field of view of the first set of cameras that corresponds to a location to which the first annotation input is directed.

19. The first device of claim 13 , wherein the one or more programs include instructions for:

during the shared annotation session:

detecting movement of the second device from a first location to a second location in the physical environment; and

20. The first device of claim 13 , wherein:

21. The first device of claim 13 , wherein:

during the shared annotation session, the first device displays the representation of the field of view of the first set of cameras via the first display generation component and the second device displays the representation of the field of view of the second set of cameras via the second display generation component.

22. The first device of claim 13 , wherein the one or more programs include instructions for:

during the shared annotation session, in accordance with a determination that the connection criteria are no longer met, terminating the shared annotation session.

23. The first device of claim 13 , wherein the one or more programs include instructions for:

24. The first device of claim 13 , wherein the one or more programs include instructions for:

during the shared annotation session, in accordance with a determination that an amount of overlap between the field of view of the first set of cameras and the field of view of the second set of cameras is below a first threshold amount, displaying, via the first display generation component, an indication of a current relative spatial relationship between the first device and the second device.

25. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a first device with a first display generation component, one or more input devices, and one or more cameras, cause the first device to:

send a request to a remote device to initiate a shared annotation session with a second device that includes a second display generation component, a second set of one or more input devices, and a second set of one or more cameras;

in response to sending the request to initiate the shared annotation session with the second device, receiving an indication of acceptance of the request to initiate the shared annotation session;

in response to receiving the indication of acceptance of the request to initiate the shared annotation session, display, via the first display generation component, a first prompt to move the first device toward the second device; and

after displaying the first prompt, in accordance with a determination that connection criteria for the first device and the second device are met, display a representation of a field of view of the first set of cameras in the shared annotation session with the second device, wherein:

26. The non-transitory computer readable storage medium of claim 25 , wherein:

27. The non-transitory computer readable storage medium of claim 26 , wherein:

during the shared annotation session, the one or more first virtual annotations corresponding to the annotation input directed to the respective location in the physical environment by the first device is displayed via the second display generation component and not via the first display generation component, provided that the respective location is included in the field of view of the second set of cameras and not included in the field of view of the first set of cameras.

28. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

29. The non-transitory computer readable storage medium of claim 25 , wherein the first prompt includes an animation illustrating representations of the first and second devices moving toward each other, and the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

30. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

31. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

during the shared annotation session:

detecting movement of the second device from a first location to a second location in the physical environment; and

32. The non-transitory computer readable storage medium of claim 25 , wherein:

33. The non-transitory computer readable storage medium of claim 25 , wherein:

during the shared annotation session, the first device displays the representation of the field of view of the first set of cameras via the first display generation component and the second device displays the representation of the field of view of the second set of cameras via the second display generation component.

34. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

during the shared annotation session, in accordance with a determination that the connection criteria are no longer met, terminating the shared annotation session.

35. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

36. The non-transitory computer readable storage medium of claim 25 , wherein the one or more programs include instructions that, when executed by the first device, cause the first device to perform operations including:

during the shared annotation session, in accordance with a determination that an amount of overlap between the field of view of the first set of cameras and the field of view of the second set of cameras is below a first threshold amount, displaying, via the first display generation component, an indication of a current relative spatial relationship between the first device and the second device.

Continuity (5)

Continuation 17716984 · Apr 8, 2022

Continuation 16997860 · Aug 19, 2020

Continuation 16574029 · Sep 17, 2019

Provisional Application 62739178 · Sep 29, 2018

Related Publication 20230199296A1 · Jun 22, 2023

Cited By (1)

US 12,198,447