Training image and text embedding models
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an image embedding model. In one aspect, a method comprises: obtaining training data comprising a plurality of training examples, wherein each training example comprises: an image pair comprising a first image and a second image; and selection data indicating one or more of: (i) a co-click rate of the image pair, and (ii) a similar-image click rate of the image pair; and using the training data to train an image embedding model having a plurality of image embedding model parameters.
1. A method performed by one or more data processing apparatus, the method comprising:
obtaining training data comprising a plurality of training examples, wherein each training example comprises:
an image pair comprising a first image and a second image; and
selection data indicating a click rate of the image pair; and
using the training data to train an image embedding model having a plurality of image embedding model parameters, wherein the training comprises, for each of the plurality of training examples:
processing the first image of the training example using the image embedding model to generate an embedding of the first image;
processing the second image of the training example using the image embedding model to generate an embedding of the second image;
determining a measure of similarity between the embedding of the first image and the embedding of the second image; and
adjusting the image embedding model parameters based at least in part on: (i) the measure of similarity between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example.
2. The method of claim 1 , wherein the click rate of the image pair comprises a co-click rate of the image pair, wherein the co-click rate of the image pair characterizes how often users select both the first image and the second image in response to both the first image and the second image being concurrently identified by search results for a search query.
3. The method of claim 2 , wherein the co-click rate for each training example indicates a fraction of times users selected both the first image and the second image of the training example in response to both the first image and the second image of the training example being concurrently identified by search results for a search query.
4. The method of claim 1 , wherein the click rate of the image pair comprises a similar-image click rate of the image pair, wherein the similar-image click rate of the image pair characterizes how often users select either the first image or the second image in response to the first image or the second image being identified by a search result for a search query respectively comprising either the second image or the first image.
5. The method for claim 4 , wherein the similar-image click rate for each training example indicates a fraction of times users selected either the first image or the second image in response to the first image or the second image being identified by a search result for a search query respectively comprising either the second image or the first image.
6. The method of claim 1 , wherein the training data is generated using a historical query log of a web search system.
7. The method of claim 1 , wherein the image embedding model comprises a convolution neural network.
8. The method of claim 7 , wherein adjusting the image embedding model parameters comprises:
determining a gradient of a loss function that depends on: (i) the measure of similarity between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example; and
using the gradient to adjust the image embedding model parameters.
9. The method for claim 8 , wherein the loss function comprises a multiplicative scaling factor that is determined based on the selection data of the training example.
10. The method for claim 9 , wherein the multiplicative scaling factor is determined as a linear combination of a co-click rate and a similar-image click rate of the training example.
11. The method of claim 1 , wherein determining a measure of similarity between the embedding of the first image and the embedding of the second image comprises:
determining a Euclidean distance between the embedding of the first image and the embedding of the second image.
12. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining training data comprising a plurality of training examples wherein each training example comprises:
an image pair comprising a first image and a second image; and
selection data indicating a click rate of the image pair; and
using the training data to train an image embedding model having a plurality of image embedding model parameters, wherein the training comprises, for each of the plurality of training examples:
processing the first image of the training example using the image embedding model to generate an embedding of the first image;
processing the second image of the training example using the image embedding model to generate an embedding of the second image;
determining a measure of similarity between the embedding of the first image and the embedding of the second image; and
adjusting the image embedding model parameters based at least in part on: (i) the measure of similarity between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example.
13. The system of claim 12 , wherein the click rate of the image pair comprises a co-click rate of the image pair, wherein the co-click rate of the image pair characterizes how often users select both the first image and the second image in response to both the first image and the second image being concurrently identified by search results for a search query.
14. The system of claim 13 , wherein the co-click rate for each training example indicates a fraction of times users selected both the first image and the second image of the training example in response to both the first image and the second image of the training example being concurrently identified by search result for a search query.
15. The system of claim 12 , wherein the click rate of the image pair comprises a similar-image click rate of the image pair, wherein the similar-image click rate of the image pair characterizes how often users select either the first image or the second image in response to the first image or the second image being identified by a search result for a search query respectively comprising either the second image or the first image.
16. The system of claim 15 , wherein the similar-image click rate for each training example indicates a fraction of time users selected either the first image or the second image in response to the first image or the second image being identified by a search result for a search query respectively comprising either the second image or the first image.
17. The system of claim 12 , wherein the training data is generated using a historical query log of a web search system.
18. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining training data comprising a plurality of training examples, wherein each training example comprises:
an image pair comprising a first image and a second image; and
selection data indicating a click rate of the image pair; and
using the training data to train an image embedding model having a plurality of image embedding model parameters wherein the training comprises, for each of the plurality of training examples:
processing the first image of the training example using the image embedding model to generate an embedding of the first image;
processing the second image of the training example using the image embedding model to generate an embedding of the second image;
determining a measure of similarity between the embedding of the first image and the embedding of the second image; and
adjusting the image embedding model parameters based at least in part on: (i) the measure of similarity between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example.
19. The non-transitory computer storage media of claim 18 , wherein the click rate of the image pair comprises a co-click rate of the image pair, wherein the co-click rate of the image pair characterizes how often users select both the first image and the second image in response to both the first image and the second image being concurrently identified by search results for a search query.
20. The non-transitory computer storage media of claim 18 , wherein the click rate of the image pair comprises a similar-image click rate of the image pair, wherein the similar-image click rate of the image pair characterizes how often users select either the first image or the second image in response to the first image or the second image being identified by a search result for a search query respectively comprising either the second image or the first image.