Voice assistant-enabled client application with user view context
Various embodiments discussed herein enable client applications to be heavily integrated with a voice assistant in order to both perform commands associated with voice utterances of users via voice assistant functionality and also seamlessly cause client applications to automatically perform native functions as part of executing the voice utterance. For example, some embodiments can automatically and intelligently cause a switch to a page the user needs and automatically and intelligently cause a population of particular fields of the page the user needs based on a user view context and the voice utterance.
1. A system comprising:
one or more processors; and
one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, at a first client application, utterance data of a user;
extracting one or more values from at least one of: a first page of the first client application or a second page of a second client application;
based at least in part on the utterance data and the extracting of the one or more values, receiving a user intent of the user of the first client application; and
in response to the receiving of the user intent, automatically populating data at the first page or another page of the client application, the automatic populating of the data is at least partially indicative of performing an operation associated with the user intent.
2. The system of claim 1 , wherein the extracting of the one or more values occurs while at least one of the first page and the second page are not displayed.
3. The system of claim 2 , wherein at least one of the first page and second page has been closed by the user or is running in a background.
4. The system of claim 1 , wherein the receiving of user intent of the user of the first client application includes receiving a prediction of what user interface task the user is trying to accomplish within first client application according to the utterance data and the one or more values.
5. The system of claim 1 , wherein the operations further comprising:
subsequent to the automatic populating of the data, receiving second utterance data of the user at the first client application;
extracting one or more values from at least one of: the first page, the second page, or the another page of the client application;
based at least in part on the first utterance data, the second utterance data, and the extracting of the one or more values from the first page or the another page, receiving a second user intent of the user of the first client application; and
in response to the receiving of the second user intent, automatically populating second data at the first page or the another page of the client application, the automatic populating of the second data is at least partially indicative of performing a second operation associated with the second user intent.
6. The system of claim 5 , wherein the receiving of the second user intent is further based on a determination of information about the user using a graph structure.
7. The system of claim 1 , wherein the automatically populating includes automatically switching from the first page to the second page and automatically populating the second page with the data.
8. The system of claim 1 , wherein the first page includes an opened or closed window, and wherein the automatically populating includes automatically populating a field within the opened or closed window.
9. The system of claim 1 , wherein the operations further comprising: detecting, at the client application, wake word issued by the user, and wherein the receiving of the utterance data occurs subsequent to the detection of the wake word.
10. The system of claim 1 , wherein the operations further comprising: detecting, at the client application, a selection of a user interface element at the client application, and wherein the receiving of utterance data occurs subsequent to the selection of the user interface element.
11. The system of claim 1 , wherein the receiving of the user intent is based on transmitting the first page or the second page and the voice utterance to a user intent understanding service and receiving, from the user intent understanding service, the user intent.
12. A computer-implemented method comprising:
receiving utterance data of a user;
receiving one or more values that have been extracted from at least one of: a first page of a first client application or a second page of a second client application;
based at least in part on the utterance data and the one or more values, determining a user intent of the user of the first client application by predicting what user interface task the user is trying to accomplish within first client application according to the utterance data and the one or more values; and
in response to the determining of the user intent, causing the first client application to automatically populate data at the first page or another page of the client application.
13. The computer-implemented method of claim 12 , wherein at least one of the first page and second page are not displayed during the receiving of the one or more values.
14. The computer-implemented method of claim 13 , wherein at least one of the first page and second page has been closed by the user or is running in a background.
15. The computer-implemented method of claim 12 , further comprising:
subsequent to the automatic populating of the data, receiving second utterance data of the user at the first client application;
receiving one or more second values from the first page or the another page of the client application;
based at least in part on the first utterance data, the second utterance data, and the one or more second values, determining a second user intent of the user of the first client application; and
in response to the determining of the second user intent, causing the first client application to automatically populate second data at the first page or the another page of the client application, the automatic populating of the second data is at least partially indicative of performing a second operation associated with the user intent.
16. The computer-implemented method of claim 15 , wherein the determination of the second user intent is further based on a determination of information about the user using a graph structure.
17. The computer-implemented method of claim 12 , wherein the automatically populating includes automatically switching from the first page to the second page and automatically populating the second page with the data.
18. One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, cause the one or more processors to perform operations comprising:
extracting utterance data of a user;
receiving one or more values from at least one of: a first page of a first client application or a second page of a second client application;
based at least in part on the utterance data and the one or more values, receiving a user intent of the user; and
in response to the receiving of the user intent, causing the first client application to automatically populate data at the first page or another page of the client application.
19. The one or more computer storage media of claim 18 , wherein at least one of the first page and second page are not displayed during the receiving of the one or more values.
20. The one or more computer storage media of claim 18 , wherein at least one of the first page and second page has been closed by the user or is running in a background.