Granted Patent B1

US 12,481,705 · App. 18/745,530 · Granted Nov 25, 2025

Inventors: Ahmet Emre Barut (Boston, MA); Chengwei Su (Belmont, MA); Weitong Ruan (Revere, MA); Wael Hamza (Yorktown Heights, NY)

Assignee: AMAZON TECHNOLOGIES, INC.

G06F16/90332G06F16/532G06F16/583G06V20/20G06N20/00

Abstract

Claims (134)

1 . A computer-implemented method comprising:

receiving input query data comprising a word token;

receiving first image data;

generating first embedding data comprising the word token and a corresponding token based on the input query data;

identifying, by an object detector model, a first image region representing at least a first portion of the first image data, and a second image region representing at least a second portion of the first image data;

generating second embedding data comprising the first image region;

generating cluster centroid data representing the first image region;

generating a spatial attention score by applying a first activation function to a product of a representation of the first image region and the cluster centroid data;

generating a spatial attention map by multiplying the spatial attention score and the representation of the first image region;

generating a down-sampled spatial attention map by down-sampling the spatial attention map;

calculating a channel attention score by applying a second activation function to a product of the down-sampled spatial attention map and a channel attention weight,

generating a channel attention amp by multiplying the channel attention score and the representation of the first image region;

generating third embedding data comprising the channel attention map;

generating fourth embedding data comprising a representation of the second image region;

storing the first embedding data, the second embedding data, the third embedding data, and the fourth embedding data in at least one non-transitory computer readable memory;

inputting the first embedding data, the second embedding data, the third embedding data, and the fourth embedding data into a multi-modal natural language understanding model to determine an output score quantifying how the input query data relates to the first image region;

determining derived query data representing the first image region, wherein the derived query data comprises a search engine query;

inputting the derived query data into a search interface;

receiving a first search result from the search interface; and

outputting the first search result in response to the input query data.

2 . The computer-implemented method of claim 1 , further comprising:

determining third data based at least in part on a distance between the derived query data and the input query data in an embedding space;

determining an item listing associated with the derived query data; and

generating output data comprising the item listing.

3 . The computer-implemented method of claim 1 , further comprising:

sending the first image data to an object detector; and

determining location data identifying a location of the first image region in the first image data, wherein the second embedding data represents pixels of a portion of the first image data identified by the location data.

4 . The computer-implemented method of claim 1 , further comprising:

determining that the input query data represents a query about a first attribute of the first image region;

generating a graph search query comprising a first data identifying the first image region and second data comprising a request for the first attribute related to the first image region;

performing a graph search using the graph search query; and

determining attribute data representing the first attribute of the first image region.

5 . The computer-implemented method of claim 1 , further comprising:

determining that the first natural input query data comprises a request to select the first image region on a graphical user interface represented by the first image data; and

generating control input data effective to select the image region on the graphical user interface.

6 . The computer-implemented method of claim 1 , further comprising:

determining a latent variable associated with the first image region; and

determining the spatial attention score based at least in part on the latent variable.

7 . The computer-implemented method of claim 1 , further comprising:

determining second image data representing at least the second e image region, wherein the first image data and the second image data are included in a single frame of image data;

determining, for the first image data by the multi-modal natural language understanding model, a first score indicating a first probability that the input query data corresponds to the first image region; and

determining, for the second image data by the multi-modal natural language understanding model, a second score indicating a second probability that the input query data corresponds to the second image region, wherein the determining, by the multi-modal natural language understanding model, that the input query data relates to the first image region comprises determining that the first probability is greater than the second probability.

8 . The computer-implemented method of claim 1 , further comprising:

determining, using a first component configured to function as a recurrent neural network, word embeddings representing at least each word of the input query data, wherein the first embedding data comprises the word embeddings; and

determining, using a second component configured to function as a convolutional neural network, visual feature embeddings representing attributes of the first image data, wherein the second embedding data comprises the visual feature embeddings.

9 . The computer-implemented method of claim 1 , further comprising:

determining, by an object detector, a plurality of objects in the first image data, wherein each object of the plurality of objects corresponds to a respective image region by the object detector model; and

computing spatial attention and channel attention for each of the plurality of objects.

10 . A system comprising:

at least one processor; and

at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to:

receive input query data comprising a word token;

receive first image data;

generate first embedding data comprising the word token and a corresponding positional token based on the input query data;

identify, by an object detector model, a first image region representing at least a first portion of the first image data, and a second image region representing at least a second portion of the first image data;

generate second embedding data representing the first image region;

generating cluster centroid data representing the first image region;

generate a spatial attention score by applying a first activation function to a product of a representation of the first image region and the cluster centroid data;

generate a spatial attention map by multiplying the spatial attention score and the representation of the first image region;

generate a down-sampled spatial attention map by down-sampling the spatial attention map;

calculate a channel attention score by applying a second activation function to a product of the down-sampled spatial attention map and a channel attention weight;

generate a channel attention map by multiplying the channel attention score and the representation of the first image region;

generate third embedding data comprising the channel attention map;

generate fourth embedding data comprising a representation of the second image region;

store the first embedding data, the second embedding data, the third embedding data, and the fourth embedding data in the at least one non-transitory computer-readable memory;

input the first embedding data, the second embedding data, the third embedding data, and the fourth embedding data into a multi-modal natural language understanding model to determine an output score quantifying how the input query data relates to the first image region;

determine derived query data representing the first image region, wherein the derived query data comprises a search engine query;

input the derived query data into a search interface;

receive a first search result from the search interface; and

output the first search result in response to the input query data.

11 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine third data based at least in part on a distance between the derived query data and the input query data in an embedding space;

determine an item listing associated with the derived query data; and

generate output data comprising the item listing.

12 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

send the first image data to an object detector; and

determine location data identifying a location of the first image region in the first image data, wherein the second embedding data represents pixels of a portion of the first image data identified by the location data.

13 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine that the input query data represents a query about a first attribute of the first image region;

generate a graph search query comprising a first data identifying the first image region and second data comprising a request for the first attribute related to the first image region;

perform a graph search using the graph search query; and

determine attribute data representing the first attribute of the first image region.

14 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine that the input query data comprises a request to select the first image region on a graphical user interface represented by the first image data; and

generate control input data effective to select the first image region on the graphical user interface.

15 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine a latent variable associated with the first image region; and

determine the spatial attention score based at least in part on the latent variable.

16 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine second image data representing at least the second image region, wherein the first image data and the second image data are included in a single frame of image data;

determine, for the first image data by the multi-modal natural language understanding model, a first score indicating a first probability that the input query data corresponds to the first image region; and

determine, for the second image data by the multi-modal natural language understanding model, a second score indicating a second probability that the input query data corresponds to the second image region, wherein the determining, by the multi-modal natural language understanding model, that the input query data relates to the first image region comprises determining that the first probability is greater than the second probability.

17 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine, using a first component configured to function as a recurrent neural network, word embeddings representing at least each word of the input query data, wherein the first embedding data comprises the word embeddings; and

determine, using a second component configured to function as a convolutional neural network, visual feature embeddings representing attributes of the first image data, wherein the second embedding data comprises the visual feature embeddings.

18 . The system of claim 10 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine, by an object detector, a plurality of objects in the first image data, wherein each object of the plurality of objects corresponds to a respective image region by the object detector model; and

compute spatial attention and channel attention for each of the plurality of objects.

19 . A method comprising:

receiving input query data comprising a word token;

receiving first image data;

generating first embedding data comprising the word token and a corresponding positional token based on the input query data;

identifying, by an object detector model, a first image region representing at least a first portion of the first image data, and a second imager region representing at least a second portion of the first image data;

generating cluster centroid data representing the first image region;

generating a spatial attention score by applying a first activation function to a product of a representation of the first image region and the cluster centroid data;

generating a spatial attention map by multiplying the spatial attention score and the representation of the first image region;

generating, for a first filter channel of the object detector model, a channel attention score using a pooled representation of the spatial attention map;

generating, for the first image region, second embedding data by multiplying the representation of the first image region by the channel attention score;

determining, by a multi-modal natural language understanding model using the first embedding data and the second embedding data, that the input query data relates to the first image region;

determining first data representing the first image region;

determining second data representing an action to take with respect to the first image region; and

performing the action based at least in part on the first data and the second data.

20 . The method of claim 19 , further comprising:

dividing a first subdivided image region corresponding to the second image region into K grid locations;

determining visual feature data V representing a first grid location of the K grid locations;

determining the spatial attention score a for the first grid location;

determining a visual representation V k of the first grid location, where V k =aV;

down-sampling the visual representation V k to generate

V

k

p

⁢

o

⁢

o

⁢

l

⁢

e

⁢

d

;

determining a channel attention score b for the first grid location using V k pooled , and

determining a visual embedding bV for the first grid location by multiplying the channel attention score b by the visual feature data V.

Assignments (1)

Continuity (1)

References Cited (45)