Granted Patent B2

US 12,749,272 · App. 18/907,301 · Granted Sep 29, 2026

Inventors: Brad K. Herman (Culver City, CA); Shiraz Akmal (Playa Vista, CA)

Assignee: Apple Inc.

G06T19/006G06F3/013G06F3/167G06T3/40G06T3/60G06T13/20G10L15/22G10L2015/228

Abstract

Claims (112)

1 . A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive, from a user, a first input corresponding to a request for assistance with a drawing;

in accordance with receiving the first input, select a physical surface in a physical environment;

display, over the selected physical surface, a first instance of a graphical overlay representing the drawing, wherein displaying the first instance of the graphical overlay over the selected physical surface includes anchoring the first instance of the graphical overlay to the selected physical surface;

while displaying the first instance of the graphical overlay over the selected physical surface, receive, from the user, a second input corresponding to a request to save a layout corresponding to the drawing;

in accordance with receiving the second input, generate the layout corresponding to the drawing, wherein the layout indicates an anchoring of the first instance of the graphical overlay to the selected physical surface; and

transmit the layout to an external electronic device, wherein after the external electronic device receives the layout, the external electronic device:

detects the selected physical surface based on the layout; and

displays, over the selected physical surface, a second instance of the graphical overlay, wherein the external electronic device anchors the second instance of the graphical overlay to the selected physical surface based on the layout.

2 . The non-transitory computer-readable storage medium of claim 1 , wherein the second input includes a speech input corresponding to the request to save the layout corresponding to the drawing.

3 . The non-transitory computer-readable storage medium of claim 1 , wherein the second input corresponds to a selection of an icon displayed concurrently with the first instance of the graphical overlay.

4 . The non-transitory computer-readable storage medium of claim 1 , wherein transmitting the layout to the external electronic device includes transmitting the layout using at least one of:

an email message;

a text message; and

Bluetooth communication between the electronic device and the external electronic device.

5 . The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

while displaying the first instance of the graphical overlay over the selected physical surface:

detect a marking produced by the user on the physical surface, the marking corresponding to the drawing.

6 . The non-transitory computer-readable storage medium of claim 5 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

determine that the detected marking corresponds to a completed portion of the drawing; and

wherein displaying, over the selected physical surface, the first instance of the graphical overlay includes:

in accordance with a determination that the detected marking corresponds to the completed portion of the drawing, modifying a first display manner of a first portion of the first instance of the graphical overlay, the first portion corresponding to the completed portion of the drawing.

7 . The non-transitory computer-readable storage medium of claim 6 , wherein modifying the first display manner includes modifying a color of the first portion.

8 . The non-transitory computer-readable storage medium of claim 6 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

in accordance with modifying the first display manner:

update the layout to indicate that the first display manner has been modified, wherein updating the layout causes the external electronic device to:

while displaying the second instance of the graphical overlay over the selected physical surface, modify a second display manner of a second portion of the second instance of the graphical overlay according to the first display manner that has been modified, wherein the first portion and the second portion correspond to a same portion of the drawing.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein updating the layout to indicate that the first display manner has been modified is performed in accordance with a determination that a modification to the first display manner is a predetermined type of modification.

10 . The non-transitory computer-readable storage medium of claim 1 , wherein the first input corresponds to a selection of an image corresponding to the drawing or to a selection of a 3-dimensional model corresponding to the drawing.

11 . The non-transitory computer-readable storage medium of claim 1 , wherein the first input includes a second speech input specifying the drawing.

12 . The non-transitory computer-readable storage medium of claim 1 , wherein selecting the physical surface includes:

receiving, from the user, a third input corresponding to a selection of the physical surface, wherein displaying the first instance of the graphical overlay over the selected physical surface is performed in accordance with receiving the third input.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the third input includes a user gaze input, wherein the user gaze input indicates that a user gaze is directed at the physical surface.

14 . The non-transitory computer-readable storage medium of claim 12 , wherein the third input includes a third speech input including a reference to the physical surface, and wherein selecting the physical surface further includes:

processing the third speech input using context information associated with the third speech input to determine that the reference refers to the physical surface.

15 . The non-transitory computer-readable storage medium of claim 12 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

in accordance with receiving the first input corresponding to the request for assistance with the drawing, and prior to receiving the third input, display the first instance of the graphical overlay over the physical environment at a first location, and wherein:

the third input corresponds to displacing the first instance of the graphical overlay from the first location to a second location on the physical surface.

16 . The non-transitory computer-readable storage medium of claim 1 , wherein anchoring the first instance of the graphical overlay to the selected physical surface includes:

displaying the first instance of the graphical overlay at a third location on the selected physical surface;

detecting movement of the electronic device; and

in response to detecting the movement, continuing to display the first instance of the graphical overlay at the third location.

17 . An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving, from a user, a first input corresponding to a request for assistance with a drawing;

in accordance with receiving the first input, selecting a physical surface in a physical environment;

displaying, over the selected physical surface, a first instance of a graphical overlay representing the drawing, wherein displaying the first instance of the graphical overlay over the selected physical surface includes anchoring the first instance of the graphical overlay to the selected physical surface;

while displaying the first instance of the graphical overlay over the selected physical surface, receiving, from the user, a second input corresponding to a request to save a layout corresponding to the drawing;

in accordance with receiving the second input, generating the layout corresponding to the drawing, wherein the layout indicates an anchoring of the first instance of the graphical overlay to the selected physical surface; and

transmitting the layout to an external electronic device, wherein after the external electronic device receives the layout, the external electronic device:

detects the selected physical surface based on the layout; and

displays, over the selected physical surface, a second instance of the graphical overlay, wherein the external electronic device anchors the second instance of the graphical overlay to the selected physical surface based on the layout.

18 . The electronic device of claim 17 , wherein transmitting the layout to the external electronic device includes transmitting the layout using at least one of:

an email message;

a text message; and

Bluetooth communication between the electronic device and the external electronic device.

19 . The electronic device of claim 17 , wherein the one or more programs further include instructions for:

while displaying the first instance of the graphical overlay over the selected physical surface:

detecting a marking produced by the user on the physical surface, the marking corresponding to the drawing.

20 . The electronic device of claim 19 , wherein the one or more programs further include instructions for:

determining that the detected marking corresponds to a completed portion of the drawing; and

wherein displaying, over the selected physical surface, the first instance of the graphical overlay includes:

in accordance with a determination that the detected marking corresponds to the completed portion of the drawing, modifying a first display manner of a first portion of the first instance of the graphical overlay, the first portion corresponding to the completed portion of the drawing.

21 . The electronic device of claim 20 , wherein the one or more programs further include instructions for:

in accordance with modifying the first display manner:

updating the layout to indicate that the first display manner has been modified, wherein updating the layout causes the external electronic device to:

while displaying the second instance of the graphical overlay over the selected physical surface, modify a second display manner of a second portion of the second instance of the graphical overlay according to the first display manner that has been modified, wherein the first portion and the second portion correspond to a same portion of the drawing.

22 . The electronic device of claim 17 , wherein the first input corresponds to a selection of an image corresponding to the drawing or to a selection of a 3-dimensional model corresponding to the drawing.

23 . The electronic device of claim 17 , wherein selecting the physical surface includes:

receiving, from the user, a third input corresponding to a selection of the physical surface, wherein displaying the first instance of the graphical overlay over the selected physical surface is performed in accordance with receiving the third input.

24 . The electronic device of claim 23 , wherein the third input includes a third speech input including a reference to the physical surface, and wherein selecting the physical surface further includes:

processing the third speech input using context information associated with the third speech input to determine that the reference refers to the physical surface.

25 . The electronic device of claim 17 , wherein anchoring the first instance of the graphical overlay to the selected physical surface includes:

displaying the first instance of the graphical overlay at a third location on the selected physical surface;

detecting movement of the electronic device; and

in response to detecting the movement, continuing to display the first instance of the graphical overlay at the third location.

26 . A method, comprising:

at an electronic device with one or more processors and memory:

receiving, from a user, a first input corresponding to a request for assistance with a drawing;

in accordance with receiving the first input, selecting a physical surface in a physical environment;

displaying, over the selected physical surface, a first instance of a graphical overlay representing the drawing, wherein displaying the first instance of the graphical overlay over the selected physical surface includes anchoring the first instance of the graphical overlay to the selected physical surface;

while displaying the first instance of the graphical overlay over the selected physical surface, receiving, from the user, a second input corresponding to a request to save a layout corresponding to the drawing;

in accordance with receiving the second input, generating the layout corresponding to the drawing, wherein the layout indicates an anchoring of the first instance of the graphical overlay to the selected physical surface; and

transmitting the layout to an external electronic device, wherein after the external electronic device receives the layout, the external electronic device:

detects the selected physical surface based on the layout; and

displays, over the selected physical surface, a second instance of the graphical overlay, wherein the external electronic device anchors the second instance of the graphical overlay to the selected physical surface based on the layout.

27 . The method of claim 26 , wherein transmitting the layout to the external electronic device includes transmitting the layout using at least one of:

an email message;

a text message; and

Bluetooth communication between the electronic device and the external electronic device.

28 . The method of claim 26 , further comprising:

while displaying the first instance of the graphical overlay over the selected physical surface:

detecting a marking produced by the user on the physical surface, the marking corresponding to the drawing.

29 . The method of claim 28 , further comprising:

determining that the detected marking corresponds to a completed portion of the drawing; and

wherein displaying, over the selected physical surface, the first instance of the graphical overlay includes:

in accordance with a determination that the detected marking corresponds to the completed portion of the drawing, modifying a first display manner of a first portion of the first instance of the graphical overlay, the first portion corresponding to the completed portion of the drawing.

30 . The method of claim 29 , further comprising:

in accordance with modifying the first display manner:

updating the layout to indicate that the first display manner has been modified, wherein updating the layout causes the external electronic device to:

while displaying the second instance of the graphical overlay over the selected physical surface, modify a second display manner of a second portion of the second instance of the graphical overlay according to the first display manner that has been modified, wherein the first portion and the second portion correspond to a same portion of the drawing.

31 . The method of claim 26 , wherein the first input corresponds to a selection of an image corresponding to the drawing or to a selection of a 3-dimensional model corresponding to the drawing.

32 . The method of claim 26 , wherein selecting the physical surface includes:

receiving, from the user, a third input corresponding to a selection of the physical surface, wherein displaying the first instance of the graphical overlay over the selected physical surface is performed in accordance with receiving the third input.

33 . The method of claim 32 , wherein the third input includes a third speech input including reference to the physical surface, and wherein selecting the physical surface further includes:

processing the third speech input using context information associated with the third speech input to determine that the reference refers to the physical surface.

34 . The method of claim 26 , wherein anchoring the first instance of the graphical overlay to the selected physical surface includes:

displaying the first instance of the graphical overlay at a third location on the selected physical surface;

detecting movement of the electronic device; and

in response to detecting the movement, continuing to display the first instance of the graphical overlay at the third location.

Continuity (3)

References Cited (400)