Generating realistic machine learning-based product images for online catalogs
An online concierge system trains a fine-tuned generative image model for distinct categories of items based on a generative image model that takes a textual query as input and outputs and an associated image. Training of the fine-tuned generative image model is additionally based on a small set of representative images associated with the various categories, as well as textual tokens associated with the categories. Once trained, the fine-tuned generative image model can be used to generate realistic representative images for items in a database of the online concierge system that are lacking associated images. The fine-tuned model permits the generation of different variants of an item, such as different quantities or amounts, different packaging or packing density, and the like.
1 . A method performed at a computer system comprising a processor and a computer-readable medium, the method for generating and managing synthetic images of items and comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to others of the item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item in an order lacking a corresponding representative image in an database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for the item for display of the generated representative image on the picker device.
2 . The method of claim 1 , further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.
3 . The method of claim 1 , wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.
4 . The method of claim 1 , wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.
5 . The method of claim 1 , wherein the token is randomly generated.
6 . The method of claim 1 , wherein the fine-tuned generative diffusion model is generated using DREAMBOOTH.
7 . A non-transitory computer-readable storage medium containing instructions that when executed by one or more processors perform actions comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to any of the other item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item lacking a corresponding representative image in a database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for the item for display of the generated representative image on the picker device.
8 . The non-transitory computer-readable storage medium of claim 7 , the actions further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.
9 . The non-transitory computer-readable storage medium of claim 7 , wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.
10 . The non-transitory computer-readable storage medium of claim 7 , wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.
11 . The non-transitory computer-readable storage medium of claim 7 , wherein the token is randomly generated.
12 . The non-transitory computer-readable storage medium of claim 7 , wherein the fine-tuned generative diffusion model is generated using DREAMBOOTH.
13 . A computer system comprising:
one or more computer processors; and
a computer-readable storage medium storing instructions that when executed by the one or more computer processors perform actions comprising:
accessing a generative diffusion model that accepts a textual query as input and generates a synthetic image as output;
for each of a plurality of categories from a taxonomy of item categories:
generating a token corresponding to the category and not corresponding to any of the other item categories;
generating a fine-tuned generative diffusion model for the categories using the generative diffusion model, sets of images corresponding to the categories, and the tokens corresponding to the categories, the fine-tuned generative diffusion model accepting a textual query as input and generating a synthetic image as output;
identifying an item lacking a corresponding representative image in a database;
generating a representative image for the item using the fine-tuned generative diffusion model by:
identifying a category, from the plurality of categories arranged within a hierarchy of taxonomy, corresponding to the item in an order lacking the corresponding representative image,
identifying a token associated with that category,
generating a textual query for the category, wherein the textual query includes the token, and
specifying the textual query as input to the fine-tuned generative diffusion model;
storing the generated representative image in the database in association with the item; and
transmitting, to a picker device associated with a picker fulfilling the order through obtainment of items in the order, the order inclusive of the generated representative image for them for display of the generated representative image on the picker device.
14 . The computer system of claim 13 , the actions further comprising:
receiving a query of a customer for items from the database;
identifying items of the database corresponding to the query, the items including the item for which the representative image was generated; and
causing display of data about the items, the data including the representative image in visual association with the item.
15 . The computer system of claim 13 , wherein the textual query contains the token corresponding to the category of the item lacking the corresponding representative image.
16 . The computer system of claim 13 , wherein the textual query contains at least one of: an amount of the item, a quantity of the item, a packaging density of the item, or a packaging type of the item.
17 . The computer system of claim 13 , wherein the fine-tuned diffusion model is generated using DREAMBOOTH.