Talk back from actions in applications
Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.
1. A non-transitory computer storage medium storing computer-useable instructions that, when used by at least one computing device, cause the at least one computing device to perform operations to facilitate talk back automation, comprising:
receiving, by a digital assistant executing on a mobile device, a verbal talk back add command to add a talk back feature that corresponds to an output of a target application installed on the mobile device, without the verbal talk back add command specifying the target application;
storing the talk back feature on a server;
initiating, by the digital assistant, a talk back recording mode based on the received verbal talk back add command, wherein the digital assistant in the talk back recording mode is configured to record a set of events that corresponds to a workflow of the target application;
analyzing, by the digital assistant, a view of the target application at a specific state to identify objects within the view, the specific state being selected by a user;
receiving, by the digital assistant, a selection of a talk back object selected from the identified objects, the talk back object being an output of the target application at the specific state and having metadata;
decoding the talk back object to provide a specific audible sound; and
generating, by the digital assistant, a talk back output corresponding to the target application based on the recorded set of events, the specific audible sound, and the selected talk back object.
2. The medium of claim 1 , further comprising, receiving, by the digital assistant, one or more commands selected by the user that, when communicated to the digital assistant by the user, initiate the workflow of the target application and provide the talk back output to the user.
3. The medium of claim 2 , wherein the digital assistant is configured to invoke a particular talk back feature responsive to a detected utterance that corresponds to a particular command of the one or more commands.
4. The medium of claim 2 , wherein the talk back output is provided on a device other than the mobile device.
5. The medium of claim 2 , wherein the talk back output is provided on the mobile device.
6. The medium of claim 2 , wherein the talk back output is communicated as an input to another application.
7. The medium of claim 1 , further comprising enabling access to the talk back feature stored on the server by other users via digital assistants corresponding to the other users.
8. The medium of claim 1 , further comprising determining an entity type for the talk back object.
9. The medium of claim 8 , wherein the entity type is determined by the digital assistant based on the metadata.
10. The medium of claim 8 , wherein the entity type is determined by a selection made by the user.
11. The medium of claim 8 , further comprising adding context to the talk back output, the context corresponding to the entity type.
12. The medium of claim 11 , further comprising employing machine learning to (1) suggest the context to add to the talk back output and (2) learn a dialect from the user.
13. The medium of claim 12 , further comprising receiving, by the digital assistant, a selection of the context made by the user to add to the talk back output.
14. The medium of claim 1 , wherein the talk back feature is embodied in a JSON data structure, and wherein the talk back output was generated without the user manually interacting with an interface or the target application.
15. The medium of claim 2 , wherein each command of the one or more commands includes a string representation of a detected audio input.
16. The medium of claim 15 , wherein each command of the one or more commands is generated by the mobile device employing a speech-to-text operation, and wherein the detected audio input is a spoken utterance detected by the digital assistant via at least one microphone of the mobile device.
17. The medium of claim 16 , wherein each command of the one or more commands includes at least one parameter.
18. A computer-implemented method for providing talk back automation, the method comprising:
receiving, by a digital assistant executing on a mobile device, a verbal command from a user, the verbal command including a string representation of a detected audio input and generated by the mobile device employing a speech-to- text operation, and wherein the detected audio input is a spoken utterance detected by the digital assistant via at least one microphone of the mobile device, and wherein the verbal command was received without the user manually interacting with an interface or a target application;
in response to the received verbal command, initiating a workflow of the target application, the workflow corresponding to a set of events;
receiving a selection of a talk back object selected from a view of the target application at a specific state, and wherein the talk back object includes metadata that enables an entity type to be determined;
decoding the talk back object to provide a specific audible sound; and
providing a talk back output corresponding to the target application, the talk back output including context based on the entity type, wherein the talk back output is based on the set of events, the specific audible sound, and the selection of the talk back object.
19. A computerized system for providing talk back automation, the system comprising:
at least one processor; and
computer readable memory storing computer usable instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, by a digital assistant executing on a mobile device, a verbal talk back add command to add a talk back feature that corresponds to an output of a target application installed on the mobile device, without the verbal talk back add command specifying the target application;
initiate, by the digital assistant, a talk back recording mode based on the received verbal talk back add command, wherein the digital assistant in the talk back recording mode is configured to record a set of events that corresponds to a workflow of the target application;
analyze, by the digital assistant, a view of the target application at a specific state to identify objects within the view, the specific state being selected by a user;
receive, by the digital assistant, a selection of a talk back object selected from the identified objects, the talk back object being an output of the target application at the specific state and having metadata;
decoding the talk back object to provide a specific audible sound; and
generate, by the digital assistant, talk back output corresponding to the target application based on the recorded set of events, the specific audible sound, and the selection of the selected talk back object, the talk back output embodied in a JSON data structure.