Pixel-based deformation of fashion items
Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a first person in a first pose and receive a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose. The methods and systems process, using one or more machine learning models, the one or more images together with the source image to generate a flow field indicating existence and location of each pixel of the one or more images in the source image and modify, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person including one or more portions of the target fashion item that extend beyond a body of the first person.
1 . A method comprising:
receiving, by one or more processors, one or more images depicting a first person in a first pose;
receiving a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose, the target fashion item comprising one or more portions that extend beyond the body of the second person;
processing, using one or more machine learning models, the one or more images together with the source image to generate a flow field, the flow field indicating existence and location of each pixel of the one or more images in the source image; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person in the first pose including the one or more portions of the target fashion item that extend beyond a body of the first person.
2 . The method of claim 1 , further comprising:
determining, based on the flow field, pose modification information for pixels of the target fashion item including the one or more portions that extend beyond the body of the second person.
3 . The method of claim 2 , further comprising:
adjusting, based on the pose modification information, a pose of the target fashion item including the one or more portions that extend beyond the body of the second person to match the first pose of the first person.
4 . The method of claim 1 , further comprising:
applying a pose estimation machine learning model to the one or more images to generate first pose estimation information representing the first pose of the first person; and
applying the pose estimation machine learning model to the source image to generate second pose estimation information representing the second pose of the second person.
5 . The method of claim 4 , further comprising:
processing, by a flow estimation machine learning model, the first pose estimation information and the second pose estimation information, to generate the flow field indicating the existence and the location of each pixel of the one or more images in the source image.
6 . The method of claim 5 , wherein the flow field indicates that a first pixel of the one or more images exists in the source image and the location of the first pixel, and wherein the flow field indicates that a second pixel of the one or more images fails to exist in the source image.
7 . The method of claim 5 , wherein the flow estimation machine learning model estimates a fit between each pixel of the body of the first person and the body of the second person and between one or more pixels outside of the body of the first person and the body of the second person.
8 . The method of claim 1 , further comprising:
sampling one or more pixels of the source image based on the flow field to extract and adjust a pose of the target fashion item to match the first pose of the first person.
9 . The method of claim 1 , wherein the one or more machine learning models comprise a convolutional neural network associated with a fashion item extended reality (XR) experience.
10 . The method of claim 1 , wherein the one or more machine learning models are trained by performing training operations comprising:
accessing training data comprising a first training image depicting a first training object in a first training pose, a second training image depicting a second training object in a second training pose, and a ground truth flow field for the first and second training images;
analyzing, using the one or more machine learning models, the first and second training images to estimate a flow field for the first and second training images;
computing a loss based on a deviation between the estimated flow field for the first and second training images and the ground flow field; and
updating one or more parameters of the one or more machine learning models based on the computed loss.
11 . The method of claim 10 , further comprising repeating the training operations for additional training data until a stopping criterion is met.
12 . The method of claim 1 , wherein the one or more machine learning models are trained by performing training operations comprising:
accessing training data comprising first training pose information, second training pose information, and a ground truth flow field for the first training pose information and the second training pose information;
analyzing, using the one or more machine learning models, the first training pose information and the second training pose information to estimate a flow field for the first training pose information and second training pose information;
computing a loss based on a deviation between the estimated flow field for the first training pose information and second training pose information and the ground flow field; and
updating one or more parameters of the one or more machine learning models based on the computed loss.
13 . The method of claim 12 , further comprising generating the first and second training pose information by performing operations comprising:
accessing a first synthetic image;
applying an augmented reality fashion item to an object depicted in the first synthetic image;
modifying a pose of the object and the augmented reality fashion item to generate a second synthetic image; and
computing the ground truth flow field indicating existence and location in the first synthetic image of the object and the augmented reality fashion item depicted in the second synthetic image.
14 . The method of claim 13 , further comprising:
storing the training data comprising the first synthetic image, the second synthetic image and the ground truth flow information.
15 . The method of claim 13 , further comprising:
storing the training data comprising pose information associated with the first synthetic image, pose information associated with the second synthetic image, and the ground truth flow information.
16 . The method of claim 12 , further comprising generating the first and second training pose information by performing operations comprising:
accessing a first image depicting an object in a real-world environment;
modifying the first image to generate a second image depicting the object, the modifying comprising at least one of rotating a portion of the first image, rendering a different view of the object depicted in the first image, cropping the portion of the first image, or applying one or more virtual elements to the first image; and
computing the ground truth flow field indicating existence and location in the first image of the object depicted in the second image.
17 . The method of claim 1 , wherein the target fashion item is a virtual object.
18 . The method of claim 1 , wherein the target fashion item is a real-world fashion item.
19 . A system comprising:
at least one processor; and
at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving one or more images depicting a first person in a first pose;
receiving a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose, the target fashion item comprising one or more portions that extend beyond the body of the second person;
processing, using one or more machine learning models, the one or more images together with the source image to generate a flow field, the flow field indicating existence and location of each pixel of the one or more images in the source image; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person in the first pose including the one or more portions of the target fashion item that extend beyond a body of the first person.
20 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving one or more images depicting a first person in a first pose;
receiving a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose, the target fashion item comprising one or more portions that extend beyond the body of the second person;
processing, using one or more machine learning models, the one or more images together with the source image to generate a flow field, the flow field indicating existence and location of each pixel of the one or more images in the source image; and
modifying, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person in the first pose including the one or more portions of the target fashion item that extend beyond a body of the first person.