Product image generation based on diffusion model
Methods and systems are disclosed for generating an extended reality (XR) try-on experience based on an image produced by a diffusion model. The system receives an image depicting a real-world object and generates a prompt comprising a textual description of a fashion item. The system analyzes the image and the textual description of the fashion item using a generative machine learning model to generate an artificial image that depicts an artificial object that resembles the real-world object wearing an artificial fashion item matching the textual description of the fashion item. The system identifies an object comprising a real-world product image that matches visual attributes of the artificial fashion item and replaces the artificial fashion item in the artificial image with the object to generate an output image.
1 . A method comprising:
receiving an image depicting a real-world object;
generating a prompt comprising a textual description of a fashion item;
analyzing the image and the textual description of the fashion item using a generative machine learning model to generate an artificial image that depicts an artificial object that resembles the real-world object wearing an artificial fashion item having visual attributes specified by the textual description of the fashion item;
searching a database of images of real-world products based on the prompt;
comparing visual attributes of extracted pixels of the artificial fashion item with visual attributes of the images of the real-world products stored in the database;
identifying an object comprising an individual image depicting an individual real-world product of the images of the real-world products for which a difference between a set of visual attributes of the individual image of the individual real-world product and the visual attributes of the artificial fashion item is smaller than a reference value; and
replacing the artificial fashion item in the artificial image with the object comprising the individual image of the individual real-world product to generate an output image.
2 . The method of claim 1 , wherein the prompt further comprises a textual description of a background, and wherein the artificial image further depicts the artificial object with an artificial background having visual features specified by the textual description of the background.
3 . The method of claim 1 , further comprising:
overlaying the object on the artificial fashion item in the artificial image.
4 . The method of claim 1 , further comprising:
analyzing the image and the textual description using the generative machine learning model to generate an artificial video comprising the artificial image, wherein the artificial fashion item depicted in a plurality of frames of the artificial video is replaced with the object.
5 . The method of claim 4 , wherein a pose of the artificial object changes across the plurality of frames of the video, and wherein the object is adjusted based on the pose of the artificial object in the plurality of frames of the video.
6 . The method of claim 1 , wherein the generative machine learning model comprises a diffusion machine learning model.
7 . The method of claim 1 , further comprising:
analyzing the artificial image using an additional machine learning model to validate the artificial image.
8 . The method of claim 7 , wherein the additional machine learning model is trained to perform operations comprising:
generating a set of body features of the artificial object depicted in the artificial image;
determining that the set of body features corresponds to a specified threshold; and
validating the image based on determining that the set of body features corresponds to the specified threshold.
9 . The method of claim 8 , wherein the set of body features comprise a set of body parts, a body mass index (BMI), and joint lengths of the artificial object.
10 . The method of claim 9 , wherein the specified threshold includes a minimum number of body parts, a minimum BMI, and a relative dimension of joint lengths for a given object.
11 . The method of claim 1 , further comprising:
receiving input from a user that specifies one or more parameters of the prompt.
12 . The method of claim 1 , further comprising:
processing the artificial image to generate a segmentation of the artificial fashion item;
extracting the artificial fashion item from the artificial image using the segmentation; and
searching a plurality of objects of different real-world product images based on the extracted artificial fashion item.
13 . The method of claim 12 , further comprising:
comparing each of the plurality of objects to the extracted artificial fashion item.
14 . The method of claim 12 , wherein the artificial fashion item in the artificial image is replaced with the object based on the segmentation of the artificial fashion item.
15 . The method of claim 1 , wherein the real-world object comprises a real-world person, further comprising:
generating a first face mask for the real-world person depicted in the received image;
cropping a portion of the received image comprising a face of the real-world person based on the first face mask;
generating a second face mask for a face of the artificial object depicted in the artificial image; and
replacing the face of the artificial object with the cropped portion of the received image comprising the face of the real-world person using the second face mask.
16 . The method of claim 15 , further comprising:
generating a fashion item mask for the object that replaced the artificial fashion item in the artificial image; and
occluding portions of the cropped portion of the received image comprising the face of the real-world person with the object based on the fashion item mask.
17 . The method of claim 15 , further comprising:
blending the object and the cropped portion of the received image comprising the face of the real-world person in the artificial image.
18 . The method of claim 17 , further comprising:
performing color correction in the artificial image in response to blending the object and the cropped portion of the received image.
19 . A system comprising:
at least one processor; and
at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving an image depicting a real-world object;
generating a prompt comprising a textual description of a fashion item;
analyzing the image and the textual description of the fashion item using a generative machine learning model to generate an artificial image that depicts an artificial object that resembles the real-world object wearing an artificial fashion item matching having visual attributes specified by the textual description of the fashion item;
searching a database of images of real-world products based on the prompt;
comparing visual attributes of extracted pixels of the artificial fashion item with visual attributes of the images of the real-world products stored in the database;
identifying an object comprising an individual image depicting an individual real-world product of the images of the real-world products for which a difference between a set of visual attributes of the individual image of the individual real-world product and the visual attributes of the artificial fashion item is smaller than a reference value; and
replacing the artificial fashion item in the artificial image with the object comprising the individual image of the individual real-world product to generate an output image.
20 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving an image depicting a real-world object;
generating a prompt comprising a textual description of a fashion item;
analyzing the image and the textual description of the fashion item using a generative machine learning model to generate an artificial image that depicts an artificial object that resembles the real-world object wearing an artificial fashion item having visual attributes specified by the textual description of the fashion item;
searching a database of images of real-world products based on the prompt;
comparing visual attributes of extracted pixels of the artificial fashion item with visual attributes of the images of the real-world products stored in the database;
identifying an object comprising an individual image depicting an individual real-world product of the images of the real-world products for which a difference between a set of visual attributes of the individual image of the individual real-world product and the visual attributes of the artificial fashion item is smaller than a reference value; and
replacing the artificial fashion item in the artificial image with the object comprising the individual image of the individual real-world product to generate an output image.