Automated assistant training and/or execution of inter-user procedures
Implementations relate to an automated assistant that can automate repeatedly performed procedures. The automation can involve communicating with different users, organizations, and/or other automated assistants. The automated assistant, with prior permission from respective user(s), can detect repeated performance of a particular series of manually initiated computational actions. Based on this determination, the automated assistant can determine automated assistant computational action(s) that can be performed by the automated assistant in order to reduce latency in performing a procedure, reduce quantity and/or size of transmissions in performing the procedure, and/or reduce an amount of client device resources required for performing the procedure. Such actions can include communicating with an additional automated assistant that may be associated with another user and/or organization. In these and other manners, manually initiated computational actions that include electronic communications amongst users can be converted to backend operations amongst instances of automated assistants to achieve technical benefits.
1. A method implemented by one or more processors, the method comprising:
determining, by a second automated assistant of a second computing device, that performance of an initial activity of an instance of a procedure has occurred,
wherein the initial activity is performed by a first automated assistant of a first computing device in response to input from a first user provided to the first automated assistant;
causing, based on the performance of the initial activity, the first automated assistant to generate first action data accessible by the second automated assistant,
wherein the first action data characterizes one or more initial actions to be initialized by the second automated assistant when performing the initial activity during a subsequent instance of the procedure;
determining, by the first automated assistant subsequent to the performance of the initial activity, that performance of a final activity by the second automated assistant in the instance of the procedure has occurred, wherein performance of the final activity by the second automated assistant is responsive to input from a second user being provided to the second automated assistant;
causing, based on the performance of the final activity, the second automated assistant to generate second action data,
wherein the second action data characterizes one or more final actions to be initialized by the first automated assistant when performing the final activity during the subsequent instance of the procedure; and
subsequent to generating the first action data and the second action data:
causing the first automated assistant and the second automated assistant to initialize performance of the subsequent instance of the procedure using the first action data and the second action data without receiving subsequent input from the first user and/or the second user.
2. The method of claim 1 , wherein the subsequent instance of the procedure is performed without a user directly instructing the first automated assistant and/or the second automated assistant to perform the subsequent instance of the procedure.
3. The method of claim 1 , further comprising:
determining, subsequent to determining that the performance of the initial activity of the instance of the procedure has occurred, that a user has performed a particular activity that is extraneous relative to the procedure,
wherein, subsequent to generating the first action data and the second action data, and based on determining that the particular activity is extraneous, the first automated assistant and/or second automated assistant performs the subsequent instance of the procedure without performing the particular activity.
4. The method of claim 3 , wherein determining that the user has performed the particular activity that is extraneous includes:
processing application data using a classification model,
wherein the classification model is trained based on previous interactions between the user and an application that is accessible via the first computing device and/or the second computing device, and
wherein the application data characterizes one or more inputs provided to the application when the user is performing the particular activity.
5. The method of claim 3 , wherein determining that the user has performed the particular activity that is extraneous includes:
processing activity data using a classification model,
wherein the classification model is trained based on previous interactions between the user and an application that is accessible via the first computing device and/or the second computing device, and
wherein the activity data characterizes one or more features of the first computing device and/or second computing device when the user is performing the particular activity.
6. The method of claim 3 ,
wherein one or more prior instances of the procedure have been performed previously via interactions between one or more users and one or more computing devices, and
wherein determining that the user has performed the particular activity that is extraneous includes:
determining that one or more prior instances of the procedure lack any activity that satisfies a threshold similarity to the particular activity.
7. The method of claim 1 , wherein determining that the performance of the initial activity of the instance of the procedure has occurred includes:
processing application data using a classification model,
wherein the classification model is trained based on previous interactions between a user and an application that is accessible via the first computing device and/or second computing device, and
wherein the application data characterizes one or more inputs provided to the application when the user is performing the initial activity.
8. The method of claim 7 , wherein the application data characterizes natural language content embodied in the one or more inputs to the application.
9. The method of claim 7 ,
wherein the application data characterizes natural language content rendered in a graphical user interface of the application, and
wherein the first action data characterizes one or more inputs to be provided to the graphical user interface of the application during the subsequent instance of the procedure.
10. The method of claim 9 , wherein the one or more inputs provided to the graphical user interface of the application include a gesture input to a touch display interface of the first computing device and/or the second computing device.
11. The method of claim 1 , wherein determining that the performance of the initial activity of the instance of the procedure has occurred includes:
processing application data using a classification model,
wherein the classification model is trained based on a previous interaction between a user and one or more applications, and
wherein the one or more initial actions include retrieving dynamic data from a first application of the one or more applications, and inputting the dynamic data to an interface of a second application of the one or more applications.
12. The method of claim 1 , wherein determining that the performance of the final activity of the instance of the procedure has occurred includes:
processing application data using a classification model,
wherein the classification model is trained based on a previous interaction between the user and a separate user via an application, and
wherein the application data characterizes the user providing, via the application, data to the separate user.
13. The method of claim 12 , wherein the data is different than other data that was previously provided by the user to the separate user during the previous interaction.
14. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause to the one or more computers to perform operations comprising:
determining, by a second automated assistant of a second computing device, that performance of an initial activity of an instance of a procedure has occurred,
wherein the initial activity is performed by a first automated assistant of a first computing device in response to input from a first user provided to the first automated assistant;
causing, based on the performance of the initial activity, the first automated assistant to generate first action data accessible by the second automated assistant,
wherein the first action data characterizes one or more initial actions to be initialized by the second automated assistant when performing the initial activity during a subsequent instance of the procedure;
determining, by the first automated assistant subsequent to the performance of the initial activity, that performance of a final activity by the second automated assistant in the instance of the procedure has occurred, wherein performance of the final activity by the second automated assistant is responsive to input from a second user being provided to the second automated assistant;
causing, based on the performance of the final activity, the second automated assistant to generate second action data,
wherein the second action data characterizes one or more final actions to be initialized by the first automated assistant when performing the final activity during the subsequent instance of the procedure; and
subsequent to generating the first action data and the second action data:
causing the first automated assistant and the second automated assistant to initialize performance of the subsequent instance of the procedure using the first action data and the second action data without receiving subsequent input from the first user and/or the second user.
15. The system of claim 14 , wherein the subsequent instance of the procedure is performed without a user directly instructing the first automated assistant and/or the second automated assistant to perform the subsequent instance of the procedure.
16. The system of claim 14 , further comprising:
determining, subsequent to determining that the performance of the initial activity of the instance of the procedure has occurred, that a user has performed a particular activity that is extraneous relative to the procedure,
wherein, subsequent to generating the first action data and the second action data, and based on determining that the particular activity is extraneous, the first automated assistant and/or second automated assistant performs the subsequent instance of the procedure without performing the particular activity.
17. The system of claim 16 , wherein determining that the user has performed the particular activity that is extraneous includes:
processing application data using a classification model,
wherein the classification model is trained based on previous interactions between the user and an application that is accessible via the first computing device and/or the second computing device, and
wherein the application data characterizes one or more inputs provided to the application when the user is performing the particular activity.
18. The system of claim 16 , wherein determining that the user has performed the particular activity that is extraneous includes:
processing activity data using a classification model,
wherein the classification model is trained based on previous interactions between the user and an application that is accessible via the first computing device and/or the second computing device, and
wherein the activity data characterizes one or more features of the first computing device and/or second computing device when the user is performing the particular activity.
19. The system of claim 16 ,
wherein one or more prior instances of the procedure have been performed previously via interactions between one or more users and one or more computing devices, and
wherein determining that the user has performed the particular activity that is extraneous includes:
determining that one or more prior instances of the procedure lack any activity that satisfies a threshold similarity to the particular activity.
20. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to initiate operations comprising:
determining, by a second automated assistant of a second computing device, that performance of an initial activity of an instance of a procedure has occurred,
wherein the initial activity is performed by a first automated assistant of a first computing device in response to input from a first user provided to the first automated assistant;
causing, based on the performance of the initial activity, the first automated assistant to generate first action data accessible by the second automated assistant,
wherein the first action data characterizes one or more initial actions to be initialized by the second automated assistant when performing the initial activity during a subsequent instance of the procedure;
determining, by the first automated assistant subsequent to the performance of the initial activity, that performance of a final activity by the second automated assistant in the instance of the procedure has occurred, wherein performance of the final activity by the second automated assistant is responsive to input from a second user being provided to the second automated assistant;
causing, based on the performance of the final activity, the second automated assistant to generate second action data,
wherein the second action data characterizes one or more final actions to be initialized by the first automated assistant when performing the final activity during the subsequent instance of the procedure; and
subsequent to generating the first action data and the second action data:
causing the first automated assistant and the second automated assistant to initialize performance of the subsequent instance of the procedure using the first action data and the second action data without receiving subsequent input from the first user and/or the second user.