IP Library › Granted Patent US 11,656,884
Granted Patent B2
US 11,656,884 · App. 16/595,962 · Granted May 23, 2023

Application integration with a digital assistant

Inventors: Carey E. Radebaugh (Los Gatos, CA); Brandon J. Newendorp (San Francisco, CA); Corey J. Peterson (Cupertino, CA); Rohit Dasari (San Francisco, CA); Trungtin Tran (Sunnyvale, CA); Vineet Khosla (Cupertino, CA)
Assignee: Apple Inc.
G06F9/453G06F3/167G06F16/245G06F16/3329G06F16/951G10L15/1815G10L15/26G06F3/0482G06F3/0488G06F40/30G06N20/00G10L15/187
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,656,884
App. No.
16/595,962
Filed
Oct 8, 2019
Granted
May 23, 2023
Kind
B2
Art Unit
2179
USPC
715/708
Abstract

Systems and processes for application integration with a digital assistant are provided. In accordance with one example, a method includes receiving an audio input including a natural-language user input and identifying an intent object of a set of intent objects. The intent object may be derived from the natural-language user input. The method further includes identifying a software application associated with the intent object of the set of intent objects, providing the intent object to the software application to cause the software application to perform a task associated with the intent object, receiving a result response indicating whether the task was successfully performed, and providing an output indicating whether the task was performed.

Claims (202)

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by one or more processors of an electronic device having a display, cause the electronic device to:

receive an audio input including a natural-language user input;

determine a user intent based on the natural-language user input;

identify, based on the determined user intent, an intent object of a set of intent objects, wherein the intent object is a data structure that corresponds to the determined user intent and that includes one or more data fields for one or more parameters associated with the determined user intent;

identify, based on the intent object, a software application for performing a task corresponding to the determined user intent;

provide the intent object to the software application;

in accordance with a determination that the task cannot be fulfilled, provide a list of one or more software applications identified for performing the task;

receive an indication of a selected software application from among the one or more listed software applications;

receive, from the selected software application, a response indicating whether the task was successfully performed; and

provide an output indicating whether the task was performed.

2. The non-transitory computer-readable storage medium of claim 1 , wherein:

identifying the software application associated with the determined user intent comprises identifying a first parameter associated with the intent object, and

providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises providing the first parameter to the software application.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the natural-language user input is a first natural-language user input, and wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

receive, from the software application, a request for a second parameter associated with the intent object;

provide a natural-language query based on the request;

receive a second natural-language user input;

identify the second parameter, wherein the second parameter is derived from the second natural-language user input; and

provide the second parameter to the software application.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the output indicates that the software application successfully performed the task.

5. The non-transitory computer-readable storage medium of claim 4 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after providing the output, receive a second input; and

in response to the second input, adjust an operation associated with the task.

6. The non-transitory computer-readable storage medium of claim 5 , wherein adjusting an operation associated with the task comprises:

adjusting the intent object, wherein adjusting the intent object comprises adjusting a parameter of the intent object; and

providing the adjusted intent object to the software application.

7. The non-transitory computer-readable storage medium of claim 5 , wherein receiving the second input comprises detecting a selection of an affordance, and

wherein adjusting an operation associated with the task comprises adjusting the operation associated with the task in response to the selection of the affordance.

8. The non-transitory computer-readable storage medium of claim 1 , wherein the output indicates that the software application failed to perform the task.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after providing the output, receive a third input; and

in response to the third input, provide another intent object to the software application to cause the software application to perform a task associated with the another intent object.

10. The non-transitory computer-readable storage medium of claim 9 , wherein receiving the third input comprises:

detecting a selection of an affordance, and

wherein providing another intent object to the software application to cause the software application to perform the task associated with the another intent object comprises:

providing the another intent object to the software application in response to the selection of the affordance.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the intent object and the another intent object are a same type of intent object and wherein a parameter associated with the intent object and a parameter associated with the another intent object have a same value.

12. The non-transitory computer-readable storage medium of claim 1 , wherein providing an output indicating whether the task was performed comprises:

receiving, from the software application, a set of view controller parameter values; and

providing the output according to the set of view controller parameter values.

13. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

receive a plurality of terms from another electronic device,

wherein receiving a natural-language user input comprises receiving a natural-language user input including a term of the plurality of terms, and

wherein identifying a software application for performing a task corresponding to the determined user intent comprises identifying the software application based on the term of the plurality of terms.

14. The non-transitory computer-readable storage medium of claim 1 , wherein identifying, based on the determined user intent, the intent object of the set of intent objects comprises identifying the intent object based on a type of the electronic device.

15. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

identify a plurality of candidate parameters associated with the determined user intent; and

provide the plurality of candidate parameters to the software application.

16. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

receive a disambiguation request corresponding to a parameter from the software application;

identify a candidate parameter from a set of candidate parameters associated with the parameter; and

provide the selected candidate parameter to the software application.

17. The non-transitory computer-readable storage medium of claim 1 , wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises:

after identifying the software application, determining whether a user input requesting cancellation or modification of the task has been received during a period of time; and

in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application.

18. The non-transitory computer-readable storage medium of claim 17 , wherein in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application comprises:

providing the intent object to the software application without user confirmation of the intent object.

19. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

receive, from the software application, a request for a parameter associated with the determined user intent;

identify, using the electronic device, a natural-language query based on the request, wherein the natural-language query is associated with the software application and the parameter;

provide the natural-language query;

receive a fourth user input;

identify the parameter based on the fourth user input; and

provide the parameter to the software application.

20. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs include further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

transmit the list of one or more software applications for display.

21. A method, comprising:

at an electronic device having one or more processors and a display:

receiving an audio input including a natural-language user input;

determining a user intent based on the natural-language user-input;

identifying, based on the determined user intent, an intent object of a set of intent objects, wherein the intent object is a data structure that corresponds to the determined user intent and that includes one or more data fields for one or more parameters associated with the determined user intent;

identifying, based on the intent object, a software application for performing a task corresponding to the determined user intent;

providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent;

in accordance with a determination that the task cannot be fulfilled, providing a list of one or more software applications identified for performing the task;

receiving an indication of a selected software application from among the one or more listed software applications;

receiving, from the selected software application, a response indicating whether the task was successfully performed; and

providing an output indicating whether the task was performed.

22. The method of claim 21 , further comprising transmitting the list of one or more software applications for display.

23. The method of claim 21 , wherein:

identifying the software application associated with the determined user intent comprises identifying a first parameter associated with the intent object, and

providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises providing the first parameter to the software application.

24. The method of claim 21 , wherein the natural-language user input is a first natural- language user input, and wherein the method further comprises:

receiving, from the software application, a request for a second parameter associated with the intent object;

providing a natural-language query based on the request;

receiving a second natural-language user input;

identifying the second parameter, wherein the second parameter is derived from the second natural-language user input; and

providing the second parameter to the software application.

25. The method of claim 21 , wherein the output indicates that the software application successfully performed the task.

26. The method of claim 25 , further comprising:

after providing the output, receiving a second input; and

in response to the second input, adjusting an operation associated with the task.

27. The method of claim 26 , wherein adjusting an operation associated with the task comprises:

adjusting the intent object, wherein adjusting the intent object comprises adjusting a parameter of the intent object; and

providing the adjusted intent object to the software application.

28. The method of claim 26 , wherein receiving the second input comprises detecting a selection of an affordance, and

wherein adjusting an operation associated with the task comprises adjusting the operation associated with the task in response to the selection of the affordance.

29. The method of claim 21 , wherein the output indicates that the software application failed to perform the task.

30. The method of claim 29 , further comprising:

after providing the output, receiving a third input; and

in response to the third input, providing another intent object to the software application to cause the software application to perform a task associated with the another intent object.

31. The method of claim 30 , wherein receiving the third input comprises:

detecting a selection of an affordance, and

wherein providing another intent object to the software application to cause the software application to perform the task associated with the another intent object comprises:

providing the another intent object to the software application in response to the selection of the affordance.

32. The method of claim 31 , wherein the intent object and the another intent object are a same type of intent object and wherein a parameter associated with the intent object and a parameter associated with the another intent object have a same value.

33. The method of claim 21 , wherein providing an output indicating whether the task was performed comprises:

receiving, from the software application, a set of view controller parameter values; and

providing the output according to the set of view controller parameter values.

34. The method of claim 21 , further comprising:

receiving a plurality of terms from another electronic device,

wherein receiving a natural-language user input comprises receiving a natural-language user input including a term of the plurality of terms, and

wherein identifying a software application for performing a task corresponding to the determined user intent comprises identifying the software application based on the term of the plurality of terms.

35. The method of claim 21 , wherein identifying, based on the determined user intent, the intent object of the set of intent objects comprises identifying the intent object based on a type of the electronic device.

36. The method of claim 21 , further comprising:

identifying a plurality of candidate parameters associated with the determined user intent; and

providing the plurality of candidate parameters to the software application.

37. The method of claim 21 , further comprising:

receiving a disambiguation request corresponding to a parameter from the software application;

identifying a candidate parameter from a set of candidate parameters associated with the parameter; and

providing the selected candidate parameter to the software application.

38. The method of claim 21 , wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises:

after identifying the software application, determining whether a user input requesting cancellation or modification of the task has been received during a period of time; and

in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application.

39. The method of claim 38 , wherein in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application comprises:

providing the intent object to the software application without user confirmation of the intent object.

40. The method of claim 21 , further comprising:

receiving, from the software application, a request for a parameter associated with the determined user intent;

identifying, using the electronic device, a natural-language query based on the request, wherein the natural-language query is associated with the software application and the parameter;

providing the natural-language query;

receiving a fourth user input;

identifying the parameter based on the fourth user input; and

providing the parameter to the software application.

41. An electronic device, comprising:

one or more processors;

a memory;

a display; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving an audio input including a natural-language user input;

determining a user intent based on the natural-language user input;

identifying, based on the determined user intent, an intent object of a set of intent objects, wherein the intent object is a data structure that corresponds to the determined user intent and that includes one or more data fields for one or more parameters associated with the determined user intent;

identifying, based on the intent object, a software application for performing a task corresponding to the determined user intent;

providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent;

in accordance with a determination that the task cannot be fulfilled, providing a list of one or more software applications identified for performing the task;

receiving an indication of a selected software application from among the one or more listed software applications;

receiving, from the selected software application, a response indicating whether the task was successfully performed; and

providing an output indicating whether the task was performed.

42. The electronic device of claim 41 , the one or more programs further comprising instructions for transmitting the list of one or more software applications for display.

43. The electronic device of claim 41 , wherein:

identifying the software application associated with the determined user intent comprises identifying a first parameter associated with the intent object, and

providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises providing the first parameter to the software application.

44. The electronic device of claim 41 , wherein the natural-language user input is a first natural-language user input, the one or more programs further comprising instructions for:

receiving, from the software application, a request for a second parameter associated with the intent object;

providing a natural-language query based on the request;

receiving a second natural-language user input;

identifying the second parameter, wherein the second parameter is derived from the second natural-language user input; and

providing the second parameter to the software application.

45. The electronic device of claim 41 , wherein the output indicates that the software application successfully performed the task.

46. The electronic device of claim 45 , the one or more programs further comprising instructions for:

after providing the output, receiving a second input; and

in response to the second input, adjusting an operation associated with the task.

47. The electronic device of claim 46 , wherein adjusting an operation associated with the task comprises:

adjusting the intent object, wherein adjusting the intent object comprises adjusting a parameter of the intent object; and

providing the adjusted intent object to the software application.

48. The electronic device of claim 46 , wherein receiving the second input comprises detecting a selection of an affordance, and

wherein adjusting an operation associated with the task comprises adjusting the operation associated with the task in response to the selection of the affordance.

49. The electronic device of claim 41 , wherein the output indicates that the software application failed to perform the task.

50. The electronic device of claim 49 , the one or more programs further comprising instructions for:

after providing the output, receiving a third input; and

in response to the third input, providing another intent object to the software application to cause the software application to perform a task associated with the another intent object.

51. The electronic device of claim 50 , wherein receiving the third input comprises:

detecting a selection of an affordance, and

wherein providing another intent object to the software application to cause the software application to perform the task associated with the another intent object comprises:

providing the another intent object to the software application in response to the selection of the affordance.

52. The electronic device of claim 51 , wherein the intent object and the another intent object are a same type of intent object and wherein a parameter associated with the intent object and a parameter associated with the another intent object have a same value.

53. The electronic device of claim 41 , wherein providing an output indicating whether the task was performed comprises:

receiving, from the software application, a set of view controller parameter values; and

providing the output according to the set of view controller parameter values.

54. The electronic device of claim 41 , the one or more programs further comprising instructions for:

receiving a plurality of terms from another electronic device,

wherein receiving a natural-language user input comprises receiving a natural-language user input including a term of the plurality of terms, and

wherein identifying a software application for performing a task corresponding to the determined user intent comprises identifying the software application based on the term of the plurality of terms.

55. The electronic device of claim 41 , wherein identifying, based on the determined user intent, the intent object of the set of intent objects comprises identifying the intent object based on a type of the electronic device.

56. The electronic device of claim 41 , the one or more programs further comprising instructions for:

identifying a plurality of candidate parameters associated with the determined user intent;

and

providing the plurality of candidate parameters to the software application.

57. The electronic device of claim 41 , the one or more programs further comprising instructions for:

receiving a disambiguation request corresponding to a parameter from the software application;

identifying a candidate parameter from a set of candidate parameters associated with the parameter; and

providing the selected candidate parameter to the software application.

58. The electronic device of claim 41 , wherein providing the intent object to the software application to cause the software application to perform the task corresponding to the determined user intent comprises:

after identifying the software application, determining whether a user input requesting cancellation or modification of the task has been received during a period of time; and

in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application.

59. The electronic device of claim 58 , wherein in accordance with a determination that the user input requesting cancellation or modification of the task has not been received during the period of time, providing the intent object to the software application comprises:

providing the intent object to the software application without user confirmation of the intent object.

60. The electronic device of claim 41 , the one or more programs further comprising instructions for:

receiving, from the software application, a request for a parameter associated with the determined user intent;

identifying, using the electronic device, a natural-language query based on the request, wherein the natural-language query is associated with the software application and the parameter;

providing the natural-language query;

receiving a fourth user input;

identifying the parameter based on the fourth user input; and

providing the parameter to the software application.

Continuity (3)
Continuation 15863523 · Jan 5, 2018
Provisional Application 62444162 · Jan 9, 2017
Related Publication 20200042334A1 · Feb 6, 2020
Cited By (1)
US 12,216,963