Automatic product description generation
Systems, devices, and techniques are disclosed for automatic product description generation. A first set of features including labels including words may be generated from an image using a first feature extraction model. A second set of features including labels including words may be generated from the image using a second feature extraction model. A text description of a product depicted in the image may be generated by inputting the image and metadata for the image to a description generating model. The text description may include words. Each of the words may be generated by assigning probabilities to candidate words, boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features, and selecting one of the candidate words based on the assigned probabilities after the boosting as a word of the text description.
1. A computer-implemented method comprising:
generating, by a computing device, from an image, using a first feature extraction model, a first set of features comprising labels comprising words;
generating, by a computing device, from the image, using a second feature extraction model, a second set of features comprising labels comprising words;
generating, by a computing device, a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated by:
assigning, by a computing device, probabilities to candidate words,
boosting, by a computing device, the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features, and
selecting, by a computing device, one of the candidate words based on the assigned probabilities after the boosting as a word of the text description;
determining by the computing device a quality level of the text description based on metrics for user interactions with a webpage for the product depicted in the image wherein the webpage comprises the text description of the product depicted in the image generated by inputting the image and metadata for the image to the description generating model; and
adjusting by the computing device the description generating model based on the determined quality level of the text description.
2. The computer-implemented method of claim 1 , wherein boosting the assigned probabilities of candidate words that are similar to words of labels the first set of features or words of labels of the second set of features further comprises determining levels of similarity between the candidate words and the words of the labels of the first set of features and the words of the labels the second set of features using a similarity model.
3. The computer-implemented method of claim 1 , further comprising
generating an extracted feature set from the first set of features and the second set of features by:
conflict checking the words of the labels of the first set of features and the words of the labels of the second set of features against data in a basic feature description of the product depicted in the image,
removing from the first set of features and the second set of features any labels comprising words that conflict with data in the basic feature description, and
combining labels not removed from the first set of features and the second set of features into an extracted feature set.
4. The computer-implemented method of claim 3 , wherein boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features comprises the boosting the assigned probabilities of candidate words that are similar to words of labels of the extracted feature set.
5. The computer-implemented method of claim 1 , wherein generating a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated, further comprises, after selecting a first word for the text description, inputting the text description with all selected words to the description generating model before assigning probabilities to candidate words.
6. The computer-implemented method of claim 1 , wherein generating a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated, further comprises decreasing the assigned probabilities of candidate words that are dissimilar to words of labels of the first set of features or words of labels of the second set of features.
7. The computer-implemented method of claim 1 , wherein selecting one of the candidate words based on the assigned probabilities after the boosting as a word of the text description comprises selecting a candidate word with the highest assigned probability.
8. The computer-implemented method of claim 1 , wherein the labels of the first set of features and the labels of the second set of features have assigned probabilities, and further comprising using a top-K restriction on the labels of the first set of features and the labels of the second set of features based on the assigned probabilities when boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features.
9. A computer-implemented system for localization of matrix factorization models trained with global data comprising:
one or more storage devices; and
a processor that generates from an image, using a first feature extraction model, a first set of features comprising labels comprising words,
generates from the image, using a second feature extraction model, a second set of features comprising labels comprising words,
generates a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated by:
assigning probabilities to candidate words,
boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features, and
selecting one of the candidate words based on the assigned probabilities after the boosting as a word of the text description, and
determines a quality level of the text description based on metrics for user interactions with a webpage for the product depicted in the image wherein the webpage comprises the text description of the product depicted in the image generated by inputting the image and metadata for the image to the description generating model, and
adjusts the description generating model based on the determined quality level of the text description.
10. The computer-implemented system of claim 9 , wherein the processor boosts the assigned probabilities of candidate words that are similar to words of labels the first set of features or words of labels of the second set of features further by determining levels of similarity between the candidate words and the words of the labels of the first set of features and the words of the labels the second set of features using a similarity model.
11. The computer-implemented system of claim 9 , wherein the processor further generates an extracted feature set from the first set of features and the second set of features by:
conflict checking the words of the labels of the first set of features and the words of the labels of the second set of features against data in a basic feature description of the product depicted in the image,
removing from the first set of features and the second set of features any labels comprising words that conflict with data in the basic feature description, and
combining labels not removed from the first set of features and the second set of features into an extracted feature set.
12. The computer-implemented system of claim 11 , wherein the processor boosts the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features by boosting the assigned probabilities of candidate words that are similar to words of labels of the extracted feature set.
13. The computer-implemented system of claim 9 , wherein the processor further generates a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated further by, after selecting a first word for the text description, inputting the text description with all previously selected words for the text description to the description generating model before assigning probabilities to candidate words.
14. The computer-implemented system of claim 9 , wherein the processor generates a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated, further by decreasing the assigned probabilities of candidate words that are dissimilar to words of labels of the first set of features or words of labels of the second set of features.
15. The computer-implemented system of claim 9 , wherein the processor selects one of the candidate words based on the assigned probabilities after the boosting as a word of the text description by selecting a candidate word with the highest assigned probability.
16. The computer-implemented system of claim 9 , wherein the labels of the first set of features and the labels of the second set of features have assigned probabilities, and wherein the processor further uses a top-K restriction on the labels of the first set of features and the labels of the second set of features based on the assigned probabilities when boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features.
17. A system comprising: one or more computers and one or more storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
generating from an image, using a first feature extraction model, a first set of features comprising labels comprising words;
generating from the image, using a second feature extraction model, a second set of features comprising labels comprising words;
generating a text description of a product depicted in the image by inputting the image and metadata for the image to a description generating model, wherein the text description comprises words and each of the words is generated by:
assigning probabilities to candidate words,
boosting the assigned probabilities of candidate words that are similar to words of labels of the first set of features or words of labels of the second set of features, and
selecting one of the candidate words based on the assigned probabilities after the boosting as a word of the text description;
determining a quality level of the text description based on metrics for user interactions with a webpage for the product depicted in the image wherein the webpage comprises the text description of the product depicted in the image generated by inputting the image and metadata for the image to the description generating model; and
adjusting the description generating model based on the determined quality level of the text description.