Pixel-based multi-view garment transfer
Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a person in an individual pose and receive a first source image depicting a first view of a target fashion item and a second source image depicting a second view of the target fashion item. The methods and systems process, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of existence and location of each pixel of the one or more images relative to the first and second source images. The methods and systems modify a portion of the one or more images to overlay the target fashion item on the person.
1 . A method comprising:
receiving, by one or more processors, one or more images depicting a person in an individual pose;
receiving a first source image depicting a first view of a target fashion item;
receiving a second source image depicting a second view of the target fashion item;
processing, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of: existence, and location of each pixel of the one or more images relative to the first and second source images; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the person in the individual pose including a first portion of the target fashion item depicted in the first source image and a second portion of the target fashion item depicted in the second source image.
2 . The method of claim 1 , further comprising:
applying a pose estimation machine learning model to the one or more images to generate first pose estimation information representing the individual pose of the person; and
applying the pose estimation machine learning model to the first and second source images to generate respectively second and third pose estimation information representing the poses of the target fashion item depicted in the first and second source images.
3 . The method of claim 2 , wherein the first pose estimation information comprises an identification of body parts of the person that are depicted in the one or more images.
4 . The method of claim 2 , wherein the second pose estimation information comprises an identification of body parts associated with the first view, and the third pose estimation information comprises an identification of body parts associated with the second view.
5 . The method of claim 2 , further comprising:
processing, by a flow estimation machine learning model, the first pose estimation information, the second pose estimation information, and the third pose estimation information, to generate the flow field indicating the likelihood of existence and the location of each pixel of the one or more images relative to the first and second source images.
6 . The method of claim 5 , wherein the flow field indicates that a first pixel of the one or more images is associated with a first likelihood of existence in the first source image and the location of the first pixel, and wherein the flow field indicates that the first pixel of the one or more images is associated with a second likelihood of existence in the second source image and the location of the first pixel.
7 . The method of claim 6 , further comprising:
determining that the first likelihood is greater than the second likelihood; and
in response to determining that the first likelihood is greater than the second likelihood, replacing the first pixel with a pixel value associated with the location of the first pixel in the first source image.
8 . The method of claim 6 , further comprising:
determining that the first likelihood is lower than the second likelihood; and
in response to determining that the first likelihood is lower than the second likelihood, replacing the first pixel with a pixel value associated with the location of the first pixel in the second source image.
9 . The method of claim 6 , further comprising:
determining that the first and second likelihoods fail to transgress a minimum likelihood of existence threshold; and
in response to determining that the first and second likelihoods fail to transgress the minimum likelihood of existence threshold, applying a generative machine learning model to the one or more images, the first source image, and the second source image to generate a pixel value for the first pixel.
10 . The method of claim 9 , further comprising:
replacing the first pixel with the pixel value generated by the generative machine learning model; and
replacing a first set of pixels of the one or more images with a first portion of the first source image and a second set of pixels of the one or more images with a second portion of the second source image.
11 . The method of claim 1 , further comprising:
generating a first confidence map for the first source image that indicates the likelihood of existence and location of each pixel of the one or more images in the first source image; and
generating a second confidence map for the second source image that indicates the likelihood of existence and location of each pixel of the one or more images in the second source image.
12 . The method of claim 11 , further comprising:
selectively replacing individual pixels in the one or more images with either pixels of the first source image or pixels of the second source image based on the first and second confidence maps.
13 . The method of claim 1 , further comprising:
sampling one or more pixels of the first and second source images based on the flow field to extract and adjust a pose of the target fashion item to match the individual pose of the person depicted in the one or more images.
14 . The method of claim 1 , wherein the one or more machine learning models comprise a convolutional neural network associated with a fashion item extended reality (XR) experience.
15 . The method of claim 14 , wherein the one or more machine learning models are trained by performing training operations comprising:
accessing training data comprising a first training image depicting a first training object in a training pose, a second training image depicting a first training view of a second training object, a third training image depicting a second training view of the second training object, and a ground truth flow field for the second and third training images;
analyzing, using the one or more machine learning models, the first, second, and third training images to estimate a flow field for the second and third training images;
computing a loss based on a deviation between the estimated flow field for the second and third training images and the ground truth flow field; and
updating one or more parameters of the one or more machine learning models based on the computed loss.
16 . The method of claim 15 , further comprising repeating the training operations for additional training data until a stopping criterion is met, wherein the ground truth flow field indicates whether a body part depicted in the first training object exists in the second or third training images.
17 . The method of claim 1 , wherein the target fashion item is a virtual object.
18 . The method of claim 1 , wherein the target fashion item is a real-world fashion item.
19 . A system comprising:
at least one processor; and
at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving one or more images depicting a person in an individual pose;
receiving a first source image depicting a first view of a target fashion item;
receiving a second source image depicting a second view of the target fashion item;
processing, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of: existence, and location of each pixel of the one or more images relative to the first and second source images; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the person in the individual pose including a first portion of the target fashion item depicted in the first source image and a second portion of the target fashion item depicted in the second source image.
20 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving one or more images depicting a person in an individual pose;
receiving a first source image depicting a first view of a target fashion item;
receiving a second source image depicting a second view of the target fashion item;
processing, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of: existence, and location of each pixel of the one or more images relative to the first and second source images; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the person in the individual pose including a first portion of the target fashion item depicted in the first source image and a second portion of the target fashion item depicted in the second source image.