Granted Patent B2

US 12,423,917 · App. 18/202,849 · Granted Sep 23, 2025

Inventors: Lynn I. Streja (San Francisco, CA); Keith P. Avery (Seattle, WA); Karan M. Daryanani (San Francisco, CA); Stephen O. Lemay (Palo Alto, CA); Myra C. Lukens (San Francisco, CA); Sreeneel K. Maddika (San Ramon, CA); Chaitanya Mannemala (San Ramon, CA); Aswath Manoharan (Tucson, AZ); Pedro Mari (Santa Cruz, CA); Jay Moon (San Francisco, CA); Abhishek Rawat (Fremont, CA)

Assignee: Apple Inc.

G06T19/003G06F3/013G06F3/017G06T13/40G10L15/22G06T2219/2012G06T2219/2016G06T2219/2021G10L2015/223

Abstract

Claims (145)

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with one or more sensors, cause the electronic device to:

while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:

detect, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;

in response to detecting the user gaze at the first object, expand the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;

detect, with the one or more sensors, a user gaze at the second object;

in accordance with detecting the user gaze at the second object, display a first animation of the second object indicating that a first digital assistant session is initiated;

receive a first audio input from the user of the electronic device; and

display a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

2. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes displaying a change in a shape, a size, or a color of the second object.

3. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes moving the second object away from the list of objects.

4. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes:

ceasing to display of the list of objects.

5. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes determining, based on the user gaze at the second object, that the first audio input is intended for the digital assistant.

6. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

while displaying the list of objects, receive a hand gesture from the user, the hand gesture corresponding to a selection of the second object; and

in response to receiving the hand gesture, display an animation of the second object, wherein the animation of the second object indicates that a second digital assistant session is initiated.

7. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

receive a second audio input including a spoken trigger for initiating a second digital assistant session; and

in response to receiving the second audio input, initiate the second digital assistant session, including displaying an animation of the second object.

8. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the second animation of the second object includes expanding and shrinking a size of the second object responsive to the first audio input.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

display a virtual object in response to receiving the first audio input, the virtual object corresponding to a response, by the digital assistant, to the first audio input.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object and the second object are persistent in the current field of view of the XR environment.

11. The non-transitory computer-readable storage medium of claim 9 , wherein the second object is persistent in the current field of view of the XR environment while the virtual object is positioned at a fixed location in the XR environment.

12. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object and the second object are positioned at respective fixed locations in the XR environment.

13. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object is displayed below the second object.

14. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object is positioned within a predetermined distance from the second object in the XR environment.

15. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

receive, from the user, a request to interact with the virtual object; and

in response to receiving the request to interact with the virtual object, expand the virtual object into a user interface of an application corresponding to the virtual object.

16. The non-transitory computer-readable storage medium of claim 15 , wherein:

the request to interact with the virtual object corresponds to moving the virtual object from an initial location to a destination location; and

expanding the virtual object into the user interface includes displaying the user interface at the destination location.

17. The non-transitory computer-readable storage medium of claim 15 , wherein:

the second object is displayed while the user interface is displayed; and

the user can control the application using the second object by gazing at the second object or by speaking a trigger phrase.

18. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

receive a request to integrate the virtual object into a second application; and

in response to receiving the request to integrate the virtual object into the second application, integrate the virtual object into the second application, including:

displaying a content of the virtual object within a user interface of the second application.

19. The non-transitory computer-readable storage medium of claim 18 , wherein:

while the content of the virtual object is displayed within the user interface of the second application:

the second object is displayed outside of the user interface of the second application.

20. The non-transitory computer-readable storage medium of claim 18 , wherein:

while the content of the virtual object is displayed within the user interface of the second application:

the second object is displayed inside of the user interface of the second application.

21. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

in accordance with displaying the second object:

cease to display the second object a predetermined duration after displaying the virtual object; and

display the first object.

22. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

determine whether one or more digital assistant dismissal criteria are satisfied; and

in accordance with a determination that the one or more digital assistant dismissal criteria are satisfied, cease to display the second object, wherein satisfying the digital assistant dismissal criteria includes at least one of:

determining that the user's attention is not directed to the virtual object;

determining that the user's attention is not directed to the virtual object for at least a second predetermined duration; and

determining that the user has used a gesture to interact with a user interface element displayed in the XR environment, wherein the user interface element does not correspond to the digital assistant.

23. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:

in accordance with displaying the second object:

cease to display the second object a third predetermined duration after receiving the first audio input from the user; and

display the first object.

24. The non-transitory computer-readable storage medium of claim 1 , wherein expanding the first object into the list of objects includes:

displaying a third animation of the second object.

25. An electronic device, comprising:

one or more sensors;

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:

detecting, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;

in response to detecting the user gaze at the first object, expanding the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;

detecting, with the one or more sensors, a user gaze at the second object;

in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a first digital assistant session is initiated;

receiving a first audio input from the user of the electronic device; and

displaying a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

26. A method, comprising:

at an electronic device with one or more processors, memory, and one or more sensors:

while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:

detecting, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;

in response to detecting the user gaze at the first object, expanding the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;

detecting, with the one or more sensors, a user gaze at the second object;

in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a first digital assistant session is initiated;

receiving a first audio input from the user of the electronic device; and

displaying a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

27. The electronic device of claim 25 , wherein displaying the first animation of the second object includes displaying a change in a shape, a size, or a color of the second object.

28. The electronic device of claim 25 , wherein displaying the first animation of the second object includes moving the second object away from the list of objects.

29. The electronic device of claim 25 , wherein displaying the first animation of the second object includes:

ceasing to display of the list of objects.

30. The electronic device of claim 25 , wherein displaying the first animation of the second object includes determining, based on the user gaze at the second object, that the first audio input is intended for the digital assistant.

31. The electronic device of claim 25 , wherein the one or more programs further include instructions for:

while displaying the list of objects, receiving a hand gesture from the user, the hand gesture corresponding to a selection of the second object; and

in response to receiving the hand gesture, displaying an animation of the second object, wherein the animation of the second object indicates that a second digital assistant session is initiated.

32. The electronic device of claim 25 , wherein displaying the second animation of the second object includes expanding and shrinking a size of the second object responsive to the first audio input.

33. The electronic device of claim 25 , wherein the one or more programs further include instructions for:

displaying a virtual object in response to receiving the first audio input, the virtual object corresponding to a response, by the digital assistant, to the first audio input.

34. The electronic device of claim 33 , wherein the virtual object and the second object are persistent in the current field of view of the XR environment.

35. The electronic device of claim 33 , wherein the second object is persistent in the current field of view of the XR environment while the virtual object is positioned at a fixed location in the XR environment.

36. The electronic device of claim 33 , wherein the virtual object and the second object are positioned at respective fixed locations in the XR environment.

37. The electronic device of claim 33 , wherein the virtual object is displayed below the second object.

38. The electronic device of claim 33 , wherein the virtual object is positioned within a predetermined distance from the second object in the XR environment.

39. The electronic device of claim 33 , wherein the one or more programs further include instructions for:

receiving, from the user, a request to interact with the virtual object; and

in response to receiving the request to interact with the virtual object, expanding the virtual object into a user interface of an application corresponding to the virtual object.

40. The electronic device of claim 33 , wherein the one or more programs further include instructions for:

in accordance with displaying the second object:

ceasing to display the second object a predetermined duration after displaying the virtual object; and

displaying the first object.

41. The electronic device of claim 33 , wherein the one or more programs further include instructions for:

determining whether one or more digital assistant dismissal criteria are satisfied; and

in accordance with a determination that the one or more digital assistant dismissal criteria are satisfied, ceasing to display the second object, wherein satisfying the digital assistant dismissal criteria includes at least one of:

determining that the user's attention is not directed to the virtual object;

determining that the user's attention is not directed to the virtual object for at least a second predetermined duration; and

determining that the user has used a gesture to interact with a user interface element displayed in the XR environment, wherein the user interface element does not correspond to the digital assistant.

42. The electronic device of claim 25 , wherein expanding the first object into the list of objects includes:

displaying a third animation of the second object.

43. The method of claim 26 , wherein displaying the first animation of the second object includes displaying a change in a shape, a size, or a color of the second object.

44. The method of claim 26 , wherein displaying the first animation of the second object includes moving the second object away from the list of objects.

45. The method of claim 26 , wherein displaying the first animation of the second object includes:

ceasing to display of the list of objects.

46. The method of claim 26 , wherein displaying the first animation of the second object includes determining, based on the user gaze at the second object, that the first audio input is intended for the digital assistant.

47. The method of claim 26 , further comprising:

while displaying the list of objects, receiving a hand gesture from the user, the hand gesture corresponding to a selection of the second object; and

in response to receiving the hand gesture, displaying an animation of the second object, wherein the animation of the second object indicates that a second digital assistant session is initiated.

48. The method of claim 26 , wherein displaying the second animation of the second object includes expanding and shrinking a size of the second object responsive to the first audio input.

49. The method of claim 26 , further comprising:

displaying a virtual object in response to receiving the first audio input, the virtual object corresponding to a response, by the digital assistant, to the first audio input.

50. The method of claim 49 , wherein the virtual object and the second object are persistent in the current field of view of the XR environment.

51. The method of claim 49 , wherein the second object is persistent in the current field of view of the XR environment while the virtual object is positioned at a fixed location in the XR environment.

52. The method of claim 49 , wherein the virtual object and the second object are positioned at respective fixed locations in the XR environment.

53. The method of claim 49 , wherein the virtual object is displayed below the second object.

54. The method of claim 49 , wherein the virtual object is positioned within a predetermined distance from the second object in the XR environment.

55. The method of claim 49 , further comprising:

receiving, from the user, a request to interact with the virtual object; and

in response to receiving the request to interact with the virtual object, expanding the virtual object into a user interface of an application corresponding to the virtual object.

56. The method of claim 49 , further comprising:

in accordance with displaying the second object:

ceasing to display the second object a predetermined duration after displaying the virtual object; and

displaying the first object.

57. The method of claim 49 , further comprising:

determining whether one or more digital assistant dismissal criteria are satisfied; and

in accordance with a determination that the one or more digital assistant dismissal criteria are satisfied, ceasing to display the second object, wherein satisfying the digital assistant dismissal criteria includes at least one of:

determining that the user's attention is not directed to the virtual object;

determining that the user's attention is not directed to the virtual object for at least a second predetermined duration; and

determining that the user has used a gesture to interact with a user interface element displayed in the XR environment, wherein the user interface element does not correspond to the digital assistant.

58. The method of claim 26 , wherein expanding the first object into the list of objects includes:

displaying a third animation of the second object.

Assignments (1)

Continuity (2)

References Cited (139)