Talk back from actions in applications
Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.
1. A non-transitory computer storage medium storing computer-useable instructions that, when used by at least one processor, cause the at least one processor to:
determine that a received command corresponds to an action associated with an application, the action including a workflow of the application and a talk back object that corresponds to a graphical user interface (GUI) object presented via the application at a specific state of the workflow;
emulate the workflow in response to the determination that the received command corresponds to the action; and
decode, at the specific state of the emulated workflow, the talk back object to provide at least one output that corresponds to the talk back object, the at least one output being different than the GUI object.
2. The medium of claim 1 , wherein the talk back object is associated with metadata and the metadata is included in the at least one output.
3. The medium of claim 1 , wherein the talk back object is decoded based on metadata associated with the talk back object.
4. The medium of claim 3 , wherein the instructions further cause the at least one processor to:
determine an entity type of the talk back object based on the metadata, and
wherein a context associated with the determined entity type is included in the at least one output.
5. The medium of claim 1 , wherein the at least one output that corresponds to the talk back object is provided to a remote computing device configured to audibly provide the at least one output.
6. The medium of claim 1 , wherein the at least one output that corresponds to the talk back object is provided as input to at least one field of another application.
7. The medium of claim 1 , wherein the at least one output includes a verbal communication.
8. The medium of claim 1 , wherein the at least one output includes a visual presentation based on a determination that audio capability is disabled.
9. The medium of claim 1 , wherein the at least one output includes a sound and/or a color that corresponds to the talk back object.
10. A computer-implemented method for providing talk back automation, the method comprising:
determining, by a computing device, that a received command corresponds to an action associated with an application, the action including a workflow of the application and a talk back object that corresponds to a graphical user interface (GUI) object presented via the application at a specific state of the workflow;
emulating, by the computing device, the workflow in response to the determination that the received command corresponds to the action; and
decoding, by the computing device, the talk back object at the specific state of the emulated workflow based on metadata associated with the talk back object, the talk back object being decoded to provide at least one audible output that corresponds at least in part to the talk back object.
11. The method of claim 10 , wherein the at least one audible output further corresponds to the metadata.
12. The method of claim 11 , wherein the metadata corresponds to one or more of a unit of measurement, a date, a starting point and an ending point, or a template for constructing the at least one audible output.
13. The method of claim 11 , wherein the determination that the received command corresponds to the action associated with the application is based at least in part on obtained contextual data.
14. The method of claim 13 , wherein the command is received as a voice command.
15. A computer-implemented method, comprising:
determining that a received command corresponds to an action associated with an application,
wherein the action includes a workflow of the application and a talk back object that corresponds to a graphical user interface (GUI) object presented via the application at a specific state of the workflow;
emulating the workflow in response to the determination that the received command corresponds to the action; and
decoding, at the specific state of the emulated workflow, the talk back object to provide at least one output that corresponds to the talk back object,
wherein the at least one output is different than the GUI object.
16. The method of claim 15 , wherein the talk back object is associated with metadata and the metadata is included in the at least one output.
17. The method of claim 15 , wherein the talk back object is decoded based on metadata associated with the talk back object.
18. The method of claim 17 , further comprising:
determining an entity type of the talk back object based on the metadata,
wherein a context associated with the determined entity type is included in the at least one output.
19. The method of claim 15 , wherein the at least one output that corresponds to the talk back object is provided as input to at least one field of another application.
20. The method of claim 15 , wherein the at least one output includes a verbal communication or a visual presentation.