IP Library Granted Patent US 12,136,172
Granted Patent B2
US 12,136,172 · App. 17/945,963 · Granted Nov 5, 2024

Digital assistant for providing visualization of snippet information

Inventors: Shiraz Akmal (Playa Vista, CA); Brad K. Herman (Culver City, CA); St John W. Colon (Los Angeles, CA); Viraaj M. Marathe (Los Angeles, CA); Isar Arason (Lilburn, GA); Scott M. Andrus (Lilburn, GA); Karan M. Daryanani (San Francisco, CA); Zihan Xu (Tarzana, CA); Ryan J. Quintana (Los Angeles, CA); David A. Carson (San Francisco, CA)
Assignee: Apple Inc.
G06T19/003G06T15/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,136,172
App. No.
17/945,963
Granted
Nov 5, 2024
Kind
B2
Abstract

This relates generally to intelligent automated assistants and, more specifically, to provide intelligent visualization of snippet information. An example method includes receiving a user request and in response to receiving the user request, determining a response for the user request, determining a format for a visual representation of the response, identifying one or more virtual objects based on the format for the visual representation of the response, determining a size for each of the one or more virtual objects based on context information associated with the response, and displaying the one or more virtual objects within a background environment based on the size for each of the one or more virtual objects and geometry of the background environment.

Claims (123)

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors, cause an electronic device to:

receive, from a user, a user request;

in response to receiving the user request:

determine a response for the user request;

determine a format for a visual representation of the response;

identify one or more virtual objects based on the format for the visual representation of the response;

determine a size for each of the one or more virtual objects based on context information associated with the response; and

display the one or more virtual objects within a background environment based on the size for each of the one or more virtual objects and geometry of the background environment.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the user request is a natural language input in form of a spoken or textual command presented at the electronic device.

3. The non-transitory computer-readable storage medium of claim 1 , wherein determining the format for the visual representation comprises:

prompting the user to select a format from a list of formats; and

receiving, from the user, a format selected from the list of formats.

4. The non-transitory computer-readable storage medium of claim 3 , wherein determining the format for the visual representation comprises:

determining a format from the list of formats based on user's historical format selections.

5. The non-transitory computer-readable storage medium of claim 1 , wherein the format for the visual representation includes at least one of a compact mode, a real-world environment mode and an immersive mode.

6. The non-transitory computer-readable storage medium of claim 1 , wherein each of the one or more virtual objects is a Graphical User Interface (GUI) component in a virtual three-dimensional space.

7. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

ranking the one or more virtual objects to be displayed within the background environment; and

presenting the one or more virtual objects within the background environment in a sequence based on the ranking, wherein the one or more virtual objects are presented within a predetermined time-period.

8. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

creating an animation involving the one or more virtual objects within the background environment; and

presenting the animation on a user interface within the electronic device.

9. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the one or more virtual objects within the background environment comprises:

scaling the one or more virtual objects based on a size associated with at least one object in the background environment; and

presenting the one or more scaled virtual objects within the background environment.

10. The non-transitory computer-readable storage medium of claim 1 , wherein scaling the one or more virtual objects comprises:

determining a size of each of the one or more virtual objects based on context information associated with the response;

determining a size and shape of at least one object within the background environment based on the geometry of the background environment; and

adjusting the size of each of the one or more virtual objects based on the size and shape of the at least one object in the background environment.

11. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more virtual objects within the background environment are displayed based on one or more capabilities of the electronic device.

12. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

receive a request from the user to move the one or more virtual objects within the background environment; and

move the one or more virtual objects in accordance with the request from the user.

13. The non-transitory computer-readable storage medium of claim 1 , wherein the background environment is a virtual environment or a real-world environment.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the real-world environment is a surrounding environment of the user providing the user request.

15. The non-transitory computer-readable storage medium of claim 13 , wherein the virtual environment is a two-dimensional or three-dimensional image.

16. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more virtual objects are displayed on a portion of a user interface along with the user request and the response.

17. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more virtual objects are displayed on a first user interface, wherein the user request is displayed on a second user interface.

18. The non-transitory computer-readable storage medium of claim 1 , wherein determining whether to present the visual representation of the response based on a user state.

19. The non-transitory computer-readable storage medium of claim 1 , wherein determining the one or more virtual objects based on time it may take to display them to the user.

20. A method performed on an electronic device, the method comprising:

receiving, from a user, a user request;

in response to receiving the user request:

determining a response for the user request;

determining a format for a visual representation of the response;

identifying one or more virtual objects based on the format for the visual representation of the response;

determining a size for each of the one or more virtual objects based on context information associated with the response; and

displaying the one or more virtual objects within a background environment based on the size for each of the one or more virtual objects and geometry of the background environment.

21. The method of claim 20 , wherein the user request is a natural language input in form of a spoken or textual command presented at the electronic device.

22. The method of claim 20 , wherein determining the format for the visual representation comprises:

prompting the user to select a format from a list of formats; and

receiving, from the user, a format selected from the list of formats.

23. The method of claim 22 , wherein determining the format for the visual representation comprises:

determining a format from the list of formats based on user's historical format selections.

24. The method of claim 20 , wherein the format for the visual representation includes at least one of a compact mode, a real-world environment mode and an immersive mode.

25. The method of claim 20 , wherein each of the one or more virtual objects is a Graphical User Interface (GUI) component in a virtual three-dimensional space.

26. The method of claim 20 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

ranking the one or more virtual objects to be displayed within the background environment; and

presenting the one or more virtual objects within the background environment in a sequence based on the ranking, wherein the one or more virtual objects are presented within a predetermined time-period.

27. The method of claim 20 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

creating an animation involving the one or more virtual objects within the background environment; and

presenting the animation on a user interface within the electronic device.

28. The method of claim 20 , wherein displaying the one or more virtual objects within the background environment comprises:

scaling the one or more virtual objects based on a size associated with at least one object in the background environment; and

presenting the one or more scaled virtual objects within the background environment.

29. The method of claim 20 , wherein scaling the one or more virtual objects comprises:

determining a size of each of the one or more virtual objects based on context information associated with the response;

determining a size and shape of at least one object within the background environment based on the geometry of the background environment; and

adjusting the size of each of the one or more virtual objects based on the size and shape of the at least one object in the background environment.

30. The method of claim 20 , wherein the one or more virtual objects within the background environment are displayed based on one or more capabilities of the electronic device.

31. The method of claim 20 , further comprising:

receiving a request from the user to move the one or more virtual objects within the background environment; and

moving the one or more virtual objects in accordance with the request from the user.

32. The method of claim 20 , wherein the background environment is a virtual environment or a real-world environment.

33. The method of claim 32 , wherein the real-world environment is a surrounding environment of the user providing the user request.

34. The method of claim 32 , wherein the virtual environment is a two-dimensional or three-dimensional image.

35. The method of claim 20 , wherein the one or more virtual objects are displayed on a portion of a user interface along with the user request and the response.

36. The method of claim 20 , wherein the one or more virtual objects are displayed on a first user interface, wherein the user request is displayed on a second user interface.

37. The method of claim 20 , wherein determining whether to present the visual representation of the response based on a user state.

38. The method of claim 20 , wherein determining the one or more virtual objects based on time it may take to display them to the user.

39. An electronic device comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving, from a user, a user request;

in response to receiving the user request:

determining a response for the user request;

determining a format for a visual representation of the response;

identifying one or more virtual objects based on the format for the visual representation of the response;

determining a size for each of the one or more virtual objects based on context information associated with the response; and

displaying the one or more virtual objects within a background environment based on the size for each of the one or more virtual objects and geometry of the background environment.

40. The electronic device of claim 39 , wherein the user request is a natural language input in form of a spoken or textual command presented at the electronic device.

41. The electronic device of claim 39 , wherein determining the format for the visual representation comprises:

prompting the user to select a format from a list of formats; and

receiving, from the user, a format selected from the list of formats.

42. The electronic device of claim 41 , wherein determining the format for the visual representation comprises:

determining a format from the list of formats based on user's historical format selections.

43. The electronic device of claim 39 , wherein the format for the visual representation includes at least one of a compact mode, a real-world environment mode and an immersive mode.

44. The electronic device of claim 39 , wherein each of the one or more virtual objects is a Graphical User Interface (GUI) component in a virtual three-dimensional space.

45. The electronic device of claim 39 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

ranking the one or more virtual objects to be displayed within the background environment; and

presenting the one or more virtual objects within the background environment in a sequence based on the ranking, wherein the one or more virtual objects are presented within a predetermined time-period.

46. The electronic device of claim 39 , wherein displaying the one or more virtual objects within the background environment on the electronic device comprises:

creating an animation involving the one or more virtual objects within the background environment; and

presenting the animation on a user interface within the electronic device.

47. The electronic device of claim 39 , wherein displaying the one or more virtual objects within the background environment comprises:

scaling the one or more virtual objects based on a size associated with at least one object in the background environment; and

presenting the one or more scaled virtual objects within the background environment.

48. The electronic device of claim 39 , wherein scaling the one or more virtual objects comprises:

determining a size of each of the one or more virtual objects based on context information associated with the response;

determining a size and shape of at least one object within the background environment based on the geometry of the background environment; and

adjusting the size of each of the one or more virtual objects based on the size and shape of the at least one object in the background environment.

49. The electronic device of claim 39 , wherein the one or more virtual objects within the background environment are displayed based on one or more capabilities of the electronic device.

50. The electronic device of claim 39 , the one or more programs further including instructions for:

receiving a request from the user to move the one or more virtual objects within the background environment; and

moving the one or more virtual objects in accordance with the request from the user.

51. The electronic device of claim 39 , wherein the background environment is a virtual environment or a real-world environment.

52. The electronic device of claim 51 , wherein the real-world environment is a surrounding environment of the user providing the user request.

53. The electronic device of claim 51 , wherein the virtual environment is a two-dimensional or three-dimensional image.

54. The electronic device of claim 39 , wherein the one or more virtual objects are displayed on a portion of a user interface along with the user request and the response.

55. The electronic device of claim 39 , wherein the one or more virtual objects are displayed on a first user interface, wherein the user request is displayed on a second user interface.

56. The electronic device of claim 39 , wherein determining whether to present the visual representation of the response based on a user state.

57. The electronic device of claim 39 , wherein determining the one or more virtual objects based on time it may take to display them to the user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: ARASON, ISAR; HERMAN, BRAD K.; CARSON, DAVID A.; MARATHE, VIRAAJ M.; XU, ZIHAN; QUINTANA, RYAN J.; AKMAL, SHIRAZ; DARYANANI, KARAN M.; ANDRUS, SCOTT M.; COLON, ST. JOHN W.
To: APPLE INC.
Reel/Frame 062059/0729 →
Continuity (2)
Provisional Application 63245749 · Sep 17, 2021
Related Publication 20230087244A1 · Mar 23, 2023