Optimizing display engagement in action automation
Embodiments of the present invention provide systems, methods, and computer storage media directed to optimizing engagement with a display during digital assistant-performed operations in response to a received command. The digital assistant generates an overlay having user interface elements that present information determined to be relevant to a user based on the received command and contextual data. The overlay is presented over the underlying operations performed on corresponding applications to mask the visible steps of the operations being performed. In this way, the digital assistant optimizes display resources that are typically rendered useless during the processing of digital assistant-performed operations.
1. A system supported by a user device, the system comprising:
at least one processor;
computer storage media storing computer-usable instructions that, when used by the at least one processor, cause the at least one processor to:
receive a command by a digital assistant of a mobile device;
select an action dataset associated with the received command,
wherein an action dataset defines multiple automated operations to be performed by an application of the user device;
determine that a first automated action of the multiple automated operations defined by the action dataset can be executed in response to the received command;
generate an overlay interface that includes a first user interface element configured to present content determined to be contextually relevant to one or more parameters of the received command;
cause performance of the multiple automated operations defined by the action dataset; and
during performance of the multiple automated operations, cause the generated overlay interface to be presented via the application of the user device,
wherein the generated overlay interface masks visual output data generated by the application of the user device.
2. The system of claim 1 , wherein the generated overlay interface is displayed for at least a latency period that corresponds to a duration of the performance of the multiple automated operations.
3. The system of claim 1 , wherein the performance of the multiple automated operations defined by the action dataset is caused by initiating the action dataset.
4. The system of claim 1 , wherein the at least one processor removes, after presenting the generated overlay interface, to cause the application to display resulting visual output data based on a completed performance of the multiple automated operations.
5. The system of claim 1 , wherein the multiple automated operations include an emulated touch event, an invocation of a deep link, an automated inclusion of at least one parameter, or an automated entry of input data.
6. The system of claim 1 , wherein the command is received based on generated speech-to-text data.
7. The system of claim 1 , wherein the overlay interface is based on the action dataset.
8. A method performed by a digital assistant of a user device, the method comprising:
receiving a command from a user of the user device;
selecting an action dataset associated with the received command,
wherein an action dataset defines multiple automated operations to be performed by an application of the user device;
determining that a first automated action can be executed in response to the received command;
generating an overlay interface that includes a first user interface element configured to present content determined to be contextually relevant to one or more parameters of the received command and to mask visual output data generated by the first automated action;
causing performance of the multiple automated operations defined by the action dataset; and
during performance of the multiple automated operations, causing the generated overlay interface to be presented via the application of the user device.
9. The method of claim 8 , wherein the generated overlay interface is displayed for at least a latency period that corresponds to a duration of the performance of the multiple automated operations.
10. The method of claim 8 , wherein the generated overlay interface masks visual output data generated by the application of the user device.
11. The method of claim 8 , wherein the performance of the multiple automated operations defined by the action dataset is caused by initiating the action dataset.
12. The method of claim 8 , wherein the at least one processor removes, after presenting the generated overlay interface, to cause the application to display resulting visual output data based on a completed performance of the multiple automated operations.
13. The method of claim 8 , wherein the multiple automated operations include an emulated touch event, an invocation of a deep link, an automated inclusion of at least one parameter, or an automated entry of input data.
14. The method of claim 8 , wherein the command is received based on generated speech-to-text data.
15. The method of claim 8 , wherein the overlay interface is based on the action dataset.
16. A non-transitory computer-readable medium, whose contents, when executed by a digital assistant of a user device, cause the digital assistant to perform a method, the method comprising:
receiving a command from a user of the user device;
selecting an action dataset associated with the received command,
wherein an action dataset defines multiple automated operations to be performed by an application of the user device;
generating a masking overlay interface that includes content determined to be contextually relevant to one or more parameters of the received command;
causing performance of the multiple automated operations defined by the action dataset; and
during performance of the multiple automated operations, causing the generated overlay interface to be presented via the application of the user device.
17. The non-transitory computer-readable medium of claim 16 , wherein the content of the overlay interface includes content determined to be contextually relevant to the user of the user device.
18. The non-transitory computer-readable medium of claim 16 , wherein the content of the overlay interface includes content representing additional actions to be performed by the application for the user after performance of the multiple automated operations.
19. The non-transitory computer-readable medium of claim 16 , wherein the content of the overlay interface includes content representing additional actions to be performed by a different application for the user after performance of the multiple automated operations.