Machine-learning models for extracting and classifying image content, and augmenting image based on same
An online system that displays items from an item catalog to users supplements content displayed for one or more of the items with information extracted from images of the items. For a particular item in the item catalog, the online system performs image processing, such as optical character recognition, on one or more images of the item to extract text phrases from the images. For each extracted text phrase, the system then uses a trained model to score the text phrase as being a viable informational message. If the score for a text phrase is above a threshold, the online system augments content displayed in a user interface for the item with the text phrase. The online system may decide whether to supplement content for the item with an extracted text phrase based on the output of a predictive model.
1 . A method, performed at a computer system comprising a processor and a non-transitory computer readable medium, comprising:
obtaining, at the computer system, one or more images of an item;
extracting, by the computer system, text from each image of the item;
generating one or more phrases for each image of the item, wherein the one or more phrases are generated by applying a trained large language model to the extracted text;
generating a phrase embedding for each phrase;
generating a probability of each phrase being an informational phrase that increases a likelihood of a user performing a specific interaction with the item by applying a phrase classification model to a phrase embedding of a phrase, an embedding for the item, and an embedding for an entity associated with the item, the phrase classification model trained by:
obtaining a training dataset including a plurality of training examples, each training example including a training phrase embedding for a training phrase, an embedding for a training item, and an embedding for an entity associated with the training item, each training example having a label indicating whether the training phrase is the informational phrase;
applying the phrase classification model to each training example of the training dataset to generate a predicted probability of the training phrase in a training example being the informational phrase;
scoring the phrase classification model using a loss function and the label of the training example; and
updating one or more parameters of the phrase classification model by backpropagation based on the scoring until one or more criteria are satisfied;
selecting a phrase for the item based on the generated probabilities;
modifying content describing the item to include the selected phrase;
receiving a request for content describing the item at the computer system from a client device; and
transmitting an interface including the modified content describing the item to the client device, wherein transmitting the interface causes the client device to display the interface.
2 . The method of claim 1 , wherein modifying content describing the item to include the selected phrase comprises:
overlaying the selected phrase in a specific portion of the content describing the item.
3 . The method of claim 1 , wherein modifying content describing the item to include the selected phrase comprises:
generating a probability of the user performing the specific interaction with the item for the content describing the item without the selected phrase based on the embedding of the item and the embedding of the user;
generating a modified probability of the user performing the specific interaction with the item for the modified content describing the item including the selected phrase based on the embedding of the item, the embedding of the user, and the phrase embedding for the selected phrase; and
modifying the content describing the item to include the selected phrase in response to the modified probability exceeding the probability.
4 . The method of claim 1 , wherein extracting text from each image of the item by the computer system comprises:
applying one or more optical character recognition models to an image of the item by the computer system to extract text from the image.
5 . The method of claim 1 , wherein extracting text from each image of the item by the computer system comprises:
applying a multimodal large language model to an image of the item, the multimodal large language model generating text from the image.
6 . The method of claim 1 , wherein the specific interaction with the item comprises including the item in an order fulfilled by the computer system.
7 . The method of claim 1 , wherein obtaining one or more images of an item at the computer system comprises:
obtaining an image of each portion of packaging of the item at the computer system.
8 . The method of claim 1 , wherein each of the one or more images includes a portion of packaging of the item.
9 . The method of claim 8 , wherein different images include different portions of packaging of the item.
10 . The method of claim 1 , wherein selecting the phrase for the item based on the generated probabilities comprises:
ranking the one or more phrases based on the generated probabilities; and
selecting a phrase having at least a threshold position in the ranking.
11 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
obtaining one or more images of an item;
extracting text from each image of the item;
generating one or more phrases for each image of the item, wherein the one or more phrases are generated by applying a trained large language model to the extracted text;
generating a phrase embedding for each phrase;
generating a probability of each phrase being an informational phrase that increases a likelihood of a user performing a specific interaction with the item by applying a phrase classification model to a phrase embedding of a phrase, an embedding for the item, and an embedding for an entity associated with the item, the phrase classification model trained by:
obtaining a training dataset including a plurality of training examples, each training example including a training phrase embedding for a training phrase, an embedding for a training item, and an embedding for an entity associated with the training item, each training example having a label indicating whether the training phrase is the informational phrase;
applying the phrase classification model to each training example of the training dataset to generate a predicted probability of the training phrase in a training example being the informational phrase;
scoring the phrase classification model using a loss function and the label of the training example; and
updating one or more parameters of the phrase classification model by backpropagation based on the scoring until one or more criteria are satisfied;
selecting a phrase for the item based on the generated probabilities;
modifying content describing the item to include the selected phrase;
receiving a request for content describing the item from a client device; and
transmitting an interface including the modified content describing the item to the client device, wherein transmitting the interface causes the client device to display the interface.
12 . The computer program product of claim 11 , wherein modifying content describing the item to include the selected phrase comprises:
overlaying the selected phrase in a specific portion of the content describing the item.
13 . The computer program product of claim 11 , wherein modifying content describing the item to include the selected phrase comprises:
generating a probability of the user performing the specific interaction with the item for the content describing the item without the selected phrase based on the embedding of the item and the embedding of the user;
generating a modified probability of the user performing the specific interaction with the item for the modified content describing the item including the selected phrase based on the embedding of the item, the embedding of the user, and the phrase embedding for the selected phrase; and
modifying the content describing the item to include the selected phrase in response to the modified probability exceeding the probability.
14 . The computer program product of claim 11 , wherein extracting text from each image of the item comprises:
applying one or more optical character recognition models to an image of the item to extract text from the image.
15 . The computer program product of claim 11 , wherein extracting text from each image of the item comprises:
applying one or more multimodal large language models to an image of the item to extract text from the image.
16 . The computer program product of claim 11 , wherein obtaining one or more images of an item comprises:
obtaining an image of each portion of packaging of the item.
17 . The computer program product of claim 11 , wherein each of the one or more images includes a portion of packaging of the item.
18 . The computer program product of claim 17 , wherein different images include different portions of packaging of the item.
19 . The computer program product of claim 11 , wherein selecting the phrase for the item based on the generated probabilities comprises:
ranking the one or more phrases based on the generated probabilities; and
selecting a phrase having at least a threshold position in the ranking.
20 . A system comprising:
a processor; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the processor to perform steps comprising:
obtaining one or more images of an item;
extracting text from each image of the item;
generating one or more phrases for each image of the item, wherein the one or more phrases are generated by applying a trained large language model to the extracted text;
generating a phrase embedding for each phrase;
generating a probability of each phrase being an informational phrase that increases a likelihood of a user performing a specific interaction with the item by applying a phrase classification model to a phrase embedding of a phrase, an embedding for the item, and an embedding for an entity associated with the item, the phrase classification model trained by:
obtaining a training dataset including a plurality of training examples, each training example including a training phrase embedding for a training phrase, an embedding for a training item, and an embedding for an entity associated with the training item, each training example having a label indicating whether the training phrase is the informational phrase;
applying the phrase classification model to each training example of the training dataset to generate a predicted probability of the training phrase in a training example being the informational phrase;
scoring the phrase classification model using a loss function and the label of the training example; and
updating one or more parameters of the phrase classification model by backpropagation based on the scoring until one or more criteria are satisfied;
selecting a phrase for the item based on the generated probabilities;
modifying content describing the item to include the selected phrase;
receiving a request for content describing the item from a client device; and
transmitting an interface including the modified content describing the item to the client device, wherein transmitting the interface causes the client device to display the interface.