Granted Patent B2

US 12,505,384 · App. 18/669,259 · Granted Dec 23, 2025

Inventors: Matthew Sharifi (Kilchberg, CH); Daniel Ramage (Seattle, WA); David Petrou (Brooklyn, NY)

Assignee: GOOGLE LLC

G06N20/00G06F3/0481G06F3/0482G06F3/0484G06F3/04895G06F9/453G06F16/245G06F16/3322G06N5/02

Abstract

Claims (79)

1 . A mobile device comprising:

a display device;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to:

receive, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application,

store, at the mobile device, the action suggestion model, and

subsequent to storing the action suggestion model at the mobile device:

receive, at the mobile device, a user selection of an image that is displayed via the display device of the mobile device and that is not in query form; and

in response to receiving the user selection of the image:

convert, at the mobile device, the user selection of the image, that is not in query form, made via the display device into a particular query, that is in query form, for the image;

provide, at the mobile device, the particular query to the action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and

output, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device.

2 . The mobile device of claim 1 , wherein, in converting the user selection of the image into the particular query, the mobile device is to:

identify an entity that is included in the image; and

use the entity as the particular query for the image.

3 . The mobile device of claim 1 , wherein, in converting the user selection of the image into the particular query, the mobile device is to:

identify a text description of the image; and

use the text description of the image as the particular query for the image.

4 . The mobile device of claim 1 , wherein, in converting the user selection of the image into the particular query, the mobile device is to:

identify one or more image tags that are associated with the image; and

use one or more of the image tags that are associated with the image as the particular query for the image.

5 . The mobile device of claim 1 , wherein the instructions further cause the mobile device to:

generate, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device,

further train, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein in further training the action suggestion model the mobile device is to train the action suggestion model using the additional training examples to generate a personalized action suggestion model that is personalized to the user, and

replace, at the mobile device, the action suggestion model with the personalized action suggestion model.

6 . The mobile device of claim 5 , wherein, in generating the additional training examples based on the search records, the mobile device is to:

identify whitelisted websites in the search records; and

generate the training examples from queries that include the whitelisted websites as highly ranked.

7 . The mobile device of claim 1 , wherein the instructions further cause the mobile device to:

initiate display of the one or more predicted actions on the display device.

8 . The mobile device of claim 7 , wherein the instructions further cause the mobile device to:

receive a selection of one of the one or more predicted actions; and

initiate an intent using the query for the mobile application associated with the selection.

9 . The mobile device of claim 8 , wherein, in initiating display of the one or more predicted actions, the mobile device is to:

output, for each predicted action, an icon for the mobile application associated with the predicted action.

10 . The mobile device of claim 9 , wherein the icons for the one or more predicted actions overlay content displayed on the display device.

11 . A method implemented by one or more processors of a mobile device, the method comprising:

receiving, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application,

storing, at the mobile device, the action suggestion model, and

subsequent to storing the action suggestion model at the mobile device:

receiving, at the mobile device, a user selection of an image that is displayed via a display device of the mobile device and that is not in query form; and

in response to receiving the user selection of the image:

converting, at the mobile device, the user selection of the image, that is not in query form, made via the display device into a particular query, that is in query form, for the image;

providing, at the mobile device, the particular query to the action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and

outputting, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device.

12 . The method of claim 11 , wherein converting the user selection of the image into the particular query comprises:

identifying an entity that is included in the image; and

using the entity as the particular query for the image.

13 . The method of claim 11 , wherein converting the user selection of the image into the particular query comprises:

identifying a text description of the image; and

using the text description of the image as the particular query for the image.

14 . The method of claim 11 , wherein converting the user selection of the image into the particular query comprises:

identifying one or more image tags that are associated with the image; and

using one or more of the image tags that are associated with the image as the particular query for the image.

15 . The method of claim 11 , further comprising:

generating, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device,

further training, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein in further training the action suggestion model the mobile device is to train the action suggestion model using the additional training examples to generate a personalized action suggestion model that is personalized to the user, and

replacing, at the mobile device, the action suggestion model with the personalized action suggestion model.

16 . The method of claim 15 , wherein generating the additional training examples based on the search records comprises:

identifying whitelisted websites in the search records; and

generating the training examples from queries that include the whitelisted websites as highly ranked.

17 . The method of claim 11 , further comprising

initiating display of the one or more predicted actions on the display device;

receiving a selection of one of the one or more predicted actions; and

initiating an intent using the query for the mobile application associated with the selection.

18 . The method of claim 17 , wherein initiating display of the one or more predicted actions comprises:

outputting, for each predicted action, an icon for the mobile application associated with the predicted action.

19 . The method of claim 18 , wherein the icons for the one or more predicted actions overlay content displayed on the display device.

20 . A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a mobile device to perform operations, the operations comprising:

receiving, at the mobile device and from a remote server, an action suggestion model, the action suggestion model being trained, using training examples obtained from search records of previously submitted queries from a plurality of users, to predict one or more actions in response to a corresponding query, each of the one or more actions being associated with a corresponding mobile application;

generating, at the mobile device, additional training examples based on the search records of previously submitted queries from a user of the mobile device;

further training, at the mobile device, the action suggestion model subsequent to receiving the action suggestion model, wherein further training comprises using the additional training examples to generate a personalized action suggestion model that is personalized to the user;

storing, at the mobile device, the personalized action suggestion model;

subsequent to further training the action suggestion model to generate the personalized action suggestion model:

receiving, at the mobile device, a user selection of an image that is displayed via a display device of the mobile device and that is not in query form; and

in response to receiving the user selection of the image:

converting, at the mobile device, the user selection of the image, that is not in query form, made via the display device into a particular query, that is in query form, for the image;

providing, at the mobile device the particular query to the personalized action suggestion model, that is stored at the mobile device, to predict at least one of the one or more actions based on the particular query; and

outputting, at the mobile device, the at least one of the one or more actions that are predicted based on the particular query via the display device of the mobile device.

Assignments (2)

Continuity (3)

References Cited (259)