Utilizing visual and textual aspects of images with recommendation systems
Described herein are systems and methods for generating an embedding—a learned representation—for an image. The embedding for the image is derived to capture visual aspects, as well as textual aspects, of the image. An encoder-decoder is trained to generate the visual representation of the image. An optical character recognition (OCR) algorithm is used to identify text/words in the image. From these words, an embedding is derived by performing an average pooling operation on pre-trained embeddings that map to the identified words. Finally, the embedding representing the visual aspects of the image is combined with the embedding representing the textual aspects of the image to generate a final embedding for the image.
1. A computer-implemented method comprising:
with a machine learning algorithm, training an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
generating a first embedding for an image by:
detecting words present in the image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image;
generating a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model,
generating from the image with the encoder the second embedding for the image;
combining the first embedding with the second embedding to derive for the image a final embedding; and
storing the final embedding for the image.
2. The computer-implemented method of claim 1 , wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, the method further comprising:
generating the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.
3. The computer-implemented method of claim 2 , wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.
4. The computer-implemented method of claim 2 , wherein the encoder of the encoder-decoder model is implemented based on the Inception v3 model.
5. The computer-implemented method of claim 1 , further comprising:
deriving with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
combining the confidence score with the first embedding and the second embedding to generate the final embedding for the image.
6. The computer-implemented method of claim 1 , wherein combining the first embedding with the second embedding to derive for the image a final embedding comprises:
concatenating the first embedding with the second embedding to derive for the image the final embedding.
7. The computer-implemented method of claim 1 , further comprising:
using the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.
8. A system comprising:
a memory storage device for storing computer-executable instructions; and
at least one processor, which, when executing the computer-executable instructions, causes the system to:
with a machine learning algorithm, train an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
generate a first embedding for the image by:
detecting words present in an image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image;
generate a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model, generate from the image with the encoder the second embedding for the image;
concatenate the first embedding with the second embedding to derive for the image a final embedding; and
store the final embedding for the image.
9. The system of claim 8 , wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, and iii) the computer-executable instructions, when executed by the processor, cause the system to:
generate the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.
10. The system of claim 9 , wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.
11. The system of claim 9 , wherein the encoder of the encoder-decoder model is implemented based on the Inception v3model.
12. The system of claim 9 , wherein the computer-executable instructions, when executed by the at least on processor, cause the system to:
derive with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
combine the confidence score with the first embedding and the second embedding to generate the final embedding for the image.
13. The system of claim 8 , wherein the computer-executable instructions, when executed by the at least on processor, cause the system to:
use the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.
14. A system comprising:
means for training an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
means for generating a first embedding for the image by:
detecting words present in an image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image; means for generating a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model, generating from the image with the encoder the second embedding for the image;
means for combining the first embedding with the second embedding to derive for the image a final embedding; and
means for storing the final embedding for the image.
15. The system of claim 14 , wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, the system further comprising:
means for generating the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.
16. The system method of claim 15 , wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.
17. The system of claim 15 , wherein the encoder of the encoder-decoder model is implemented based on the Inception v3 model.
18. The system of claim 14 , further comprising:
means for deriving with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
means for combining the confidence score with the first embedding and the second embedding to generate the final embedding for the image.
19. The system of claim 14 , further comprising:
means for concatenating the first embedding with the second embedding to derive for the image the final embedding.
20. The system of claim 14 , further comprising:
means for using the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.