System and method for selecting and providing available actions from one or more computer applications to a user
A computing system can be configured to input model input that includes context data into a machine-learned model and receive model output that describes one or more semantic entities referenced by the context data. The computing system can be configured to provide data descriptive of the semantic entity or entities to the computer application(s) and receive application output(s) respectively from the computing application(s) in response to providing the data descriptive of semantic entity or entities to the computer application(s). The application output(s) received from each computer application can describe available action(s) of the corresponding computer application with respect to the semantic entity or entities. The computing system can be configured to provide at least one indicator to a user that describes the available action(s) of the corresponding computer applications with respect to the semantic entity or entities.
1. A computing system, comprising:
at least one processor;
a machine-learned model configured to receive a model input that comprises context data, and, in response to receipt of the model input, output a model output that describes one or more semantic entities referenced by the context data;
one or more computer applications; and
at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
inputting the model input into the machine-learned model;
receiving, as an output of the machine-learned model, the model output that describes the one or more semantic entities referenced by the context data;
providing data descriptive of the one or more semantic entities to the one or more computer applications;
receiving one or more application outputs respectively from the one or more computing applications in response to providing the data descriptive of the one or more semantic entities to the one or more computer applications, wherein the application output received from each computer application describes one or more available actions of the corresponding computer application with respect to the one or more semantic entities, wherein the operations of inputting the model input, receiving the model output, providing the data descriptive of the one or more semantic entities, and receiving the one or more application outputs are performed proactively without receipt of a user input requesting their performance; and
providing at least one indicator to a user of the computing system, wherein the at least one indicator describes at least one of the one or more available actions of the corresponding computer applications with respect to the one or more semantic entities.
2. The computing system of claim 1 , wherein the context data comprises at least one of information displayed in a user interface, audio played by the computing system, or ambient audio detected by the computing system.
3. The computing system of claim 1 , wherein the context data comprises at least one of calendar data or a location of a mobile computing device of the computing system.
4. The computing system of claim 1 , wherein the computing system comprises an artificial intelligence system that includes the machine-learned model and that performs the operations, wherein the artificial intelligence system is separate and distinct from the one or more computer applications but capable of communicating with the one or more computer applications.
5. The computing system of claim 1 , wherein the artificial intelligence system provides the data descriptive of the one or more semantic entities to the one or more computer applications and receives the one or more application outputs respectively from the one or more computing applications via a pre-defined application programming interface.
6. The computing system of claim 1 , wherein the at least one indicator comprises:
a graphical indicator presented in the user interface; or
an audio indicator played to the user.
7. The computing system of claim 1 , wherein the operations of providing the at least one indicator is performed reactively in response to a user input.
8. The computing system of claim 1 , wherein providing the at least one indicator to the user of the computing system comprises displaying the at least one indicator in at least one of an operating system-level navigation bar in the user interface or a lock screen in the user interface.
9. The computing system of claim 1 , wherein the operations further comprise selecting the one or more computer applications to provide the data descriptive of the one or more semantic entities from a plurality of applications operable on the computing system based on a comparison between the model output and respective information about the plurality of applications.
10. The computing system of claim 1 , further comprising selecting the at least one or more available actions described by the at least one indicator to provide to the user from the one or more available actions described by the application outputs based on at least one of relevancy to the one or more semantic entities, a location of a mobile computing device of the computing system, past user interactions, a type of the one or more semantic entities, or a type of the one or more available actions.
11. The computing system of claim 1 , further comprising a ranking machine-learned model configured to receive an input that describes the one or more available actions described by the output received from each computer application, and, in response to receipt of the input, output a ranking output that describes a ranking of the one or more available actions, and wherein the operations further comprise:
inputting the input that describes the one or more available actions into the ranking machine-learned model; and
receiving, as an output of the ranking machine-learned model, the ranking output that describes the ranking of the respective outputs.
12. The computing system of claim 1 , wherein the operations further comprise:
receiving a stylization output from the one or more computing applications, the stylization output describing aesthetic features associated with displaying the at least one indicator in the user interface; and
displaying the at least one indicator in the user interface based on the stylization output.
13. A computer-implemented method for selecting and providing available actions from one or more computer applications to a user, the method comprising:
inputting, by one or more computing devices, a model input that comprises context data into a machine-learned model that is configured to receive the model input, and, in response to receipt of the model input, output a model output that describes one or more semantic entities referenced by the context data;
receiving, by the one or more computing devices, as an output of the machine-learned model, the model output that describes the one or more semantic entities referenced by the context data;
providing, by the one or more computing devices, data descriptive of the one or more semantic entities to the one or more computer applications;
receiving, by the one or more computing devices, one or more application outputs respectively from the one or more computing applications in response to providing the data descriptive of the one or more semantic entities to the one or more computer applications, wherein the application output received from each computer application describes one or more available actions of the corresponding computer application with respect to the one or more semantic entities, wherein inputting the model input, receiving the model output, providing the data descriptive of the one or more semantic entities, and receiving the one or more application outputs are performed proactively without receipt of a user input requesting their performance; and
providing, by the one or more computing devices, at least one indicator to a user of the computing system, wherein the at least one indicator describes at least one of the one or more available actions of the corresponding computer applications with respect to the one or more semantic entities.
14. The method of claim 13 , wherein the operations of providing the at least one indicator is performed reactively in response to a user input.
15. The method of claim 13 , wherein providing the at least one indicator to the user of the computing system comprises displaying the at least one indicator in at least one of an operating system-level navigation bar in the user interface or a lock screen in the user interface.
16. The method of claim 13 , further comprising selecting the one or more computer applications to provide the data descriptive of the one or more semantic entities from a plurality of applications operable on the computing system based on a comparison between the model output and respective information about the plurality of applications.
17. The method of claim 13 , further comprising selecting the at least one or more available actions described by the at least one indicator to provide to the user from the one or more available actions described by the application outputs based on at least one of relevancy to the one or more semantic entities, past user interactions, a type of the one or more semantic entities, or a type of the one or more available actions.
18. The method of claim 13 , further comprising:
inputting an input that describes the one or more available actions described by the output received from each computer application into a ranking machine-learned model that is configured to receive the input, and, in response to receipt of the input, output a ranking output that describes a ranking of the one or more available actions; and
receiving, as an output of the ranking machine-learned model, the ranking output that describes the ranking of the respective outputs.