Interactive retrieval using visual semantic matching
Techniques are generally described for interactive image retrieval using visual semantic matching. Image data and text data are encoded into a single shared visual semantic embedding space. A prediction model is trained using reference inputs, target outputs, and modification text describing changes to the reference inputs to obtain the target outputs. The prediction model can be used to perform image-to-text, text-to-image, and interactive retrieval.
1 . A method of interactive shopping assistance, said method comprising:
training a machine learning product prediction model based at least in part on:
determining, using at least one processor, a predicted output vector based on an encoding of a reference image input and an encoding of a modification text input describing a modification to the reference image input that results in a target image output;
determining, using the at least one processor, a target output vector based at least in part on an encoding of the target image output; and
determining, using the at least one processor, a compositional matching loss based at least in part on a difference between the predicted output vector and the target output vector;
receiving image data from a user, the image data representing an image of an article of clothing;
receiving a modification input from the user, the modification input describing a desired modification to the article of clothing; and
processing the image data and the modification input with the machine learning product prediction model to identify a target product corresponding to the desired modification to the article of clothing; and
sending image data of the target product to the user.
2 . The method according to claim 1 , further comprising:
training an embedding model based at least in part on:
determining a first representation of reference image data;
determining a second representation of a reference textual description describing the reference image data;
determining a third representation of negative reference image data different than the reference image data;
determining a fourth representation of a negative reference description describing the negative reference image data; and
determining an embedding loss based at least in part on the first representation, the second representation, the third representation, and the fourth representation; and
generating the encoding of the reference image input using the embedding model; and
generating the encoding of the target image output using the embedding model.
3 . The method according to claim 1 , further comprising:
for each product of a plurality of products in product catalog, determining a corresponding product vector in an embedding space based at least in part on an encoding of a corresponding image of the product;
processing the image data and the modification input with the machine learning product prediction model to generate a predicted output vector in the embedding space, the predicted output vector corresponding to the desired modification to the article of clothing; and
identifying the target product of the plurality of products by determining that a first product vector corresponding to the target product is closest of all of the product vectors to the predicted output vector in the embedding space.
4 . A method, comprising:
training a machine learning prediction model based at least in part on:
determining, using at least one processor, a predicted output vector based at least in part on an encoding of a reference input and an encoding of a modification input describing a modification to the reference input;
determining, using the at least one processor, a target output vector based at least in part on an encoding of a target output; and
determining, using the at least one processor, a compositional matching loss based at least in part on a difference between the predicted output vector and the target output vector;
receiving a query modification input describing a modification to a query reference input; and
processing the query modification input and the query reference input with the machine learning prediction model to generate a predicted query output vector.
5 . The method according to claim 4 , further comprising:
for each of a plurality of objects in a database, determining a corresponding result vector based at least in part on an encoding of the object; and
identifying a first object of the plurality of objects by determining that a first result vector corresponding to the first object is closest of all of the result vectors corresponding to the plurality of objects in the database to the predicted query output vector in an embedding space.
6 . The method according to claim 4 , wherein the training the machine learning prediction model comprises:
determining the predicted output vector based at least in part on an encoding of a reference image data input and an encoding of a modification text input describing the modification to the reference image data input; and
determining the target output vector based at least in part on an encoding of a target image data output.
7 . The method according to claim 6 , further comprising:
generating the encoding of the reference image data input by sending the reference image data input to a convolutional neural network (CNN) and applying an image projection model; and
generating the encoding of the target image data output by sending the target image data output to a CNN and applying the image projection model.
8 . The method according to claim 6 , further comprising:
generating the encoding of the modification text input by sending the modification text input to a long short term memory (LSTM) and applying a text projection model.
9 . The method according to claim 4 , further comprising:
determining the compositional matching loss according to:
L
=
L
v
s
e
+
L
i
m
+
L
t
m
wherein L represents the compositional matching loss, L vse represents an embedding loss, L im represents a compositional image matching loss, and L tm represents a compositional text matching loss.
10 . The method according to claim 4 , further comprising:
training an embedding model based at least in part on:
determining a first representation of reference image data;
determining a second representation of a reference description describing the reference image data;
determining a third representation of negative reference image data different than the reference image data;
determining a fourth representation of a negative reference description describing the negative reference image data; and
determining an embedding loss based at least in part on the first representation, the second representation, the third representation, and the fourth representation; and
generating the encoding of the reference input using the embedding model; and
generating the encoding of the target output using the embedding model.
11 . The method according to claim 10 , further comprising:
determining the embedding loss based at least in part on a first difference between the first representation and the fourth representation, and a second difference between the second representation and the third representation.
12 . The method according to claim 10 , further comprising:
determining the embedding loss according to:
L
v
s
e
=
d
v,t
−
d
v,t
⎺
+
m
+
+
d
v,t
−
d
v
⎺
,
t
+
m
+
wherein L vse represents the embedding loss, v represents the first representation of the reference image data, t represents the second representation of the reference description, ν- represents the third representation of the negative reference image data, and t- represents the fourth representation of the negative reference description, d represents a distance between representations in an embedding space defined by the embedding model, and m represents a margin.
13 . The method according to claim 4 , wherein the training the machine learning prediction model further comprises:
determining a compositional image matching loss based at least in part on a difference between the predicted output vector and the target output vector representing a target image.
14 . The method according to claim 4 , wherein the training the machine learning prediction model further comprises:
determining a compositional text matching loss based at least in part on a difference between the predicted output vector and the target output vector representing a target text description.
15 . A system, comprising:
at least one processor; and
at least one non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to program the processor to perform a method comprising:
training a prediction model based at least in part on:
determining a predicted output vector based at least in part on an encoding of a reference input and an encoding of a modification input describing a modification to the reference input;
determining a target output vector based at least in part on an encoding of a target output; and
determining a compositional matching loss based at least in part on a difference between the predicted output vector and the target output vector;
receiving a query modification input describing a modification to a query reference input; and
processing the query modification input and the query reference input with the prediction model to generate a predicted query output vector.
16 . The system of claim 15 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to program the at least one processor to perform the method further comprising:
for each of a plurality of objects in a database, determining a corresponding result vector based at least in part on an encoding of the object; and
identifying a first object of the plurality of objects by determining that a first result vector corresponding to the first object is closest of all of the result vectors corresponding to the plurality of objects in the database to the predicted query output vector in an embedding space.
17 . The system of claim 15 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to program the at least one processor to perform the method further comprising:
determining the predicted output vector based at least in part on an encoding of a reference image data input and an encoding of a modification text input describing the modification to the reference image data input; and
determining the target output vector based at least in part on an encoding of a target image data output.
18 . The system of claim 17 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to program the at least one processor to perform the method further comprising:
generating the encoding of the reference image data input by sending the reference image data input to a convolutional neural network (CNN) and applying an image projection model; and
generating the encoding of the target image data output by sending the target image data output to a CNN and applying the image projection model.
19 . The system of claim 17 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to program the at least one processor to perform the method further comprising:
generating the encoding of the modification text input by sending the modification text input to a long short term memory (LSTM) and applying a text projection model.
20 . The system of claim 15 , the at least one non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to program the at least one processor to perform the method further comprising:
determining the compositional matching loss according to:
L
=
L
v
s
e
+
L
i
m
+
L
t
m
wherein L represents the compositional matching loss, L vse represents an embedding loss, L im represents a compositional image matching loss, and L tm represents a compositional text matching loss.