Voice enabling applications
Systems and methods for voice control of computing devices are disclosed. Applications may be downloaded and/or accessed by a device having a display, and content associated with the applications may be displayed. Many applications do not allow for voice commands to be utilized to interact with the displayed content. Improvements described herein allow for non-voice-enabled applications to utilize voice commands to interact with displayed content by determining screen data displayed by the device and utilizing the screen data to determine an intent associated with the application. Directive data to perform an action corresponding to the intent may be sent to the device and may be utilized to perform the action on an object associated with the displayed content.
1. A system comprising:
one or more processors; and
non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving audio data representing a user utterance;
determining intent data based at least in part on the audio data;
identifying first directive data corresponding to the intent data, the first directive data configured to, when sent to a device, cause the device to perform a first operation with respect to a portion of content displayed by the device;
identifying second directive data corresponding to the intent data, the second directive data configured to, when sent to the device, cause the device to perform a second operation with respect to the portion of the content;
determining a first priority associated with the first directive data from a first content type associated with the portion of the content, the first content type being text content;
determining a second priority associated with the second directive data from a second content type associated with the portion of the content, the second content type being icon content;
determining that the first directive data is prioritized over the second directive data based on the first content type being prioritized over the second content type;
selecting the first directive data based on the first directive data being prioritized over the second directive data; and
sending the first directive data to the device to cause an action to be performed with respect to the portion of the content.
2. The system of claim 1 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based on:
determining that the first directive data has been sent to the device a first number of times;
determining that the second directive data has been sent to the device a second number of times; and
determining that the first number of times is greater than the second number of times.
3. The system of claim 1 , wherein the portion of the content comprises a first portion of the content, and determining that that the first directive data is prioritized over the second directive data comprises determining that that the first directive data is prioritized over the second directive data based on:
determining a first rendering size of the first portion of the content with respect to a screen of the device;
determining a second rendering size of a second portion of the content with respect to the screen; and
determining that the first rendering size is greater than the second rendering size.
4. The system of claim 1 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based on contextual data indicating that, for an application associated with the intent data, the first directive data is prioritized over the second directive data.
5. A method, comprising:
receiving audio data representing a user utterance;
determining, based at least in part on the audio data, intent data associated with the user utterance;
identifying first directive data corresponding to the intent data, the first directive data configured to, when received by a device, cause the device to perform a first operation with respect to a portion of content of an application;
identifying second directive data corresponding to the intent data, the second directive data configured to, when received by the device, cause the device to perform a second operation with respect to the portion of the content;
determining that the first directive data is prioritized over the second directive data;
selecting the first directive data based at least in part on the first directive data being prioritized over the second directive data; and
sending the first directive data to the device to cause an action to be performed on the portion of the content.
6. The method of claim 5 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on:
determining that the first directive data has been sent to the device a first number of times;
determining that the second directive data has been sent to the device a second number of times; and
determining that the first number of times is greater than the second number of times.
7. The method of claim 5 , wherein the portion of the content comprises a first portion of the content, and determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on:
determining a first rendering size of the first portion of the content with respect to a screen of the device;
determining a second rendering size of a second portion of the content with respect to the screen; and
determining that the first rendering size is greater than the second rendering size.
8. The method of claim 5 , wherein determining that that the first directive data is prioritized over the second directive data comprises determining that that the first directive data is prioritized over the second directive data based on contextual data indicating that, for the application, the first directive data is prioritized over the second directive data.
9. The method of claim 5 , wherein the content comprises first content, and further comprising:
receiving an indication that second content is displayed by the device instead of the first content;
determining, based at least in part on the indication, that the second directive data is prioritized over the first directive data;
selecting, based at least in part on the second directive data being prioritized over the first directive data, the second directive data; and
sending the second directive data to the device.
10. The method of claim 5 , wherein the audio data comprises first audio data that represents a first user utterance, and further comprising:
determining, based at least in part on second audio data representing a second user utterance, second intent data associated with the second user utterance;
determining that at least one of the first directive data or the second directive data corresponds to the second intent data;
identifying third directive data corresponding to the second intent data, the third directive data being content independent;
selecting the third directive data based at least in part on the third directive data being content independent; and
sending the third directive data to the device.
11. The method of claim 5 , further comprising:
determining that the first directive data is associated with a value associated with the intent data, the value indicating a first portion of the content on which to perform the first operation, the first portion of the content associated with a first content type;
determining that the value is associated with a second portion of the content associated with a second content type; and
wherein determining that the first directive data is prioritized over the second directive data is based at least in part on prioritizing the first content type over the second content type.
12. The method of claim 5 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on previous directive data sent to the device.
13. A system, comprising:
one or more processors; and
non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
determining, based at least in part on audio data representing a user utterance, intent data associated with the user utterance;
identifying first directive data corresponding to the intent data, the first directive data configured to, when sent to a device, cause the device to perform a first operation with respect to a portion of content of an application;
identifying second directive data corresponding to the intent data, the second directive data configured to, when sent to the device, cause the device to perform a second operation with respect to the portion of the content;
determining that the first directive data is prioritized over the second directive data; and
sending the first directive data to the device based at least in part on determining that the first directive data is prioritized over the second directive data.
14. The system of claim 13 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on:
determining that the first directive data has been sent to the device a first number of times;
determining that the second directive data has been sent to the device a second number of times; and
determining that the first number of times is greater than the second number of times.
15. The system of claim 13 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on:
determining a first rendering size of the first portion of the content with respect to a screen of the device;
determining a second rendering size of a second portion of the content with respect to the screen; and
determining that the first rendering size is greater than the second rendering size.
16. The system of claim 13 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based on contextual data indicating that, for the application, the first directive data is prioritized over the second directive data.
17. The system of claim 13 , wherein the content comprises first content, and the operations further comprising:
receiving an indication that second content is displayed by the device instead of the first content;
determining, based at least in part on the indication, that the second directive data is prioritized over the first directive data;
selecting, based at least in part on the second directive data being prioritized over the first directive data, the second directive data; and
sending the second directive data to the device.
18. The system of claim 13 , wherein the audio data comprises first audio data that represents a first user utterance, and the operations further comprising:
determining, based at least in part on second audio data representing a second user utterance, second intent data associated with the second user utterance;
determining that at least one of the first directive data or the second directive data corresponds to the second intent data;
identifying third directive data corresponding to the second intent data, the third directive data being content independent;
selecting the third directive data based at least in part on the third directive data being content independent; and
sending the third directive data to the device.
19. The system of claim 13 , the operations further comprising:
determining that the first directive data is associated with a value associated with the intent data, the value indicating a first portion of the content on which to perform the first operation, the first portion of the content associated with a first content type;
determining that the value is associated with a second portion of the content associated with a second content type; and
wherein determining that the first directive data is prioritized over the second directive data is based at least in part on prioritizing the first content type over the second content type.
20. The system of claim 13 , wherein determining that the first directive data is prioritized over the second directive data comprises determining that the first directive data is prioritized over the second directive data based at least in part on previous directive data sent to the device.