Action suggestions for user-selected content
Systems and methods are provided for suggesting actions for selected text based on content displayed on a mobile device. An example method can include converting a selection made via a display device into a query, providing the query to an action suggestion model that is trained to predict an action given a query, each action being associated with a mobile application, receiving one or more predicted actions, and initiating display of the one or more predicted actions on the display device. Another example method can include identifying, from search records, queries where a website is highly ranked, the website being one of a plurality of websites in a mapping of websites to mobile applications. The method can also include generating positive training examples for an action suggestion model from the identified queries, and training the action suggestion model using the positive training examples.
1. A mobile device comprising:
a display device;
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the mobile device to:
receive, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application,
generate, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device,
further train, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein in further training the action suggestion model the mobile device is to train the action suggestion model using the additional training examples to generate a personalized action suggestion model that is personalized to the user,
store, at the mobile device, the personalized action suggestion model, and
subsequent to further training the action suggestion model to generate the personalized action suggestion model:
receive, at the mobile device, a user selection of textual content that is displayed via the display device of the mobile device and that is not in query form; and
in response to receiving the user selection of the textual content:
convert, at the mobile device, the user selection of the textual content, that is not in query form, made via the display device into a particular query, that is in query form, for the textual content;
provide, at the mobile device, the particular query to the personalized action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and
output, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device.
2. The mobile device of claim 1 , wherein in generating the additional training examples based on the search records, the mobile device is to:
identify whitelisted websites in the search records; and
generate the training examples from queries that include the whitelisted websites as highly ranked.
3. The mobile device of claim 1 , wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:
initiate display of the one or more predicted actions on the display device.
4. The mobile device of claim 3 , wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:
receive a selection of one of the one or more predicted actions; and
initiate an intent using the query for the mobile application associated with the selection.
5. The mobile device of claim 1 , wherein initiating display of the one or more predicted actions includes for each predicted action, displaying an icon for the mobile application associated with the predicted action.
6. The mobile device of claim 5 , wherein the icons for the one or more predicted actions overlay content displayed on the display device.
7. The mobile device of claim 1 , wherein the selection is of the textual content that is displayed via the display device, and wherein the particular query is for the textual content.
8. The mobile device of claim 7 , wherein the selection of the textual content is received via touch input directed to the display device, and wherein, in converting the selection made into the particular query for the textual content, the mobile device is to:
identify, based on the touch input, one or more terms; and
use the one or more terms as the particular query for the textual content.
9. The mobile device of claim 7 , wherein the selection of the textual content is received via a voice command, and wherein, in converting the selection made into the particular query for the textual content, the mobile device is to:
perform word recognition, on the voice command, to generate one or more recognized words; and
use the one or more recognized words as the particular query for the textual content.
10. A method implemented by one or more processors of a mobile device, the method comprising:
receiving, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application;
generating, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device;
further training, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein further training comprises using the additional training examples to generate a personalized action suggestion model that is personalized to the user;
storing, at the mobile device, the personalized action suggestion model;
subsequent to further training the action suggestion model to generate the personalized action suggestion model:
receiving, at the mobile device, a user selection of textual content that is displayed via a display device of the mobile device and that is not in query form; and
in response to receiving the user selection of the textual content or the image:
converting, at the mobile device, the user selection of the textual content, that is not in query form, made via the display device into a particular query, that is in query form, for the textual content;
providing, at the mobile device, the particular query to the personalized action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and
outputting, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device of the mobile device.
11. The method of claim 10 , wherein generating the additional training examples comprises:
identifying whitelisted websites in the search records; and
generating the training examples from queries that include the whitelisted websites as highly ranked.
12. The method of claim 10 , further comprising:
initiating display of the one or more predicted actions on the display device.
13. The method of claim 12 , further comprising:
receiving a selection of one of the one or more predicted actions; and
initiating an intent using the query for the mobile application associated with the selection.
14. The method of claim 10 , wherein initiating display of the one or more predicted actions comprises for each predicted action, displaying an icon for the mobile application associated with the predicted action.
15. The method of claim 14 , wherein the icons for the one or more predicted actions overlay content displayed on the display device.
16. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform operations, the operations comprising:
receiving, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application;
generating, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device;
further training, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein further training comprises using the additional training examples to generate a personalized action suggestion model that is personalized to the user;
storing, at the mobile device, the personalized action suggestion model;
subsequent to further training the action suggestion model to generate the personalized action suggestion model:
receiving, at the mobile device, a user selection of textual content that is displayed via a display device of the mobile device and that is not in query form; and
in response to receiving the user selection of the textual content:
converting, at the mobile device, the user selection of the textual content, that is not in query form, made via the display device into a particular query, that is in query form, for the textual content;
providing, at the mobile device the particular query to the personalized action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and
outputting, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device of the mobile device.