Mirroring 3D assets for virtual experiences
Methods and systems are disclosed for generating mirrored 3D assets for an extended reality (XR) experience. The system receives a three-dimensional (3D) object comprising a target and analyzes the 3D object using one or more machine learning models to generate data associated with a mirrored version of the target of the 3D object. The system applies the mirrored version of the target to a mirrored version of the 3D object using the generated data and generates a new 3D object comprising the mirrored version of the 3D object and the mirrored version of the target.
1 . A method comprising:
receiving a three-dimensional (3D) object comprising a target;
analyzing the 3D object using one or more machine learning models to generate data associated with a mirrored version of the target of the 3D object, the one or more machine learning models comprising a convolutional neural network trained by performing training operations comprising:
accessing training data comprising a training pair of fashion items including a training fashion item and a ground truth mirrored version of the training fashion item;
analyzing the training fashion item to estimate a mirrored version of the training fashion item;
computing a deviation between the estimated mirrored version of the training fashion item and the ground truth mirrored version of the training fashion item; and
updating one or more parameters of the convolutional neural network based on the computed deviation;
applying the mirrored version of the target to a mirrored version of the 3D object using the generated data; and
generating a new 3D object comprising the mirrored version of the 3D object and the mirrored version of the target.
2 . The method of claim 1 , wherein the 3D object and the mirrored version of the 3D object represent a pair of fashion items, and wherein the target and the mirrored version of the target comprise one or more graphical elements or text on the pair of fashion items.
3 . The method of claim 1 , further comprising:
extracting the 3D object from an image; and
generating the mirrored version of the 3D object from the extracted 3D object, wherein the 3D object comprises a first virtual or real-world object and the mirrored version of the 3D object comprises a second virtual object, wherein the mirrored version of the 3D object is separately generated from the generated data associated with the mirrored version of the target.
4 . The method of claim 3 , further comprising:
identifying a first bounding box of the 3D object that includes the target;
determining a second bounding box in the mirrored version of the 3D object that corresponds to the first bounding box; and
replacing visual content in the second bounding box with the mirrored version of the 3D object to generate the mirrored version of the 3D object comprising the mirrored version of the target.
5 . The method of claim 1 , further comprising:
applying a first machine learning model to the 3D object to identify a bounding box comprising the target; and
applying a second machine learning model to generate a segmentation of the target in the bounding box, the segmentation distinguishing pixels in the 3D object of the bounding box of the target from other pixels in the bounding box.
6 . The method of claim 5 , further comprising:
applying a flip transformation to the segmentation of the target in the bounding box, the flip transformation flipping the segmentation of the target around a flipping axis of the bounding box thereby to generate a flipped segmentation of the target.
7 . The method of claim 6 , further comprising:
blending pixels of the flipped segmentation of the target to a specified portion of the mirrored version of the 3D object.
8 . The method of claim 7 , wherein the pixels of the flipped segmentation are blended to the specified portion while preserving original texture features of the mirrored version of the 3D object.
9 . The method of claim 7 , wherein the pixels are blended according to at least one of Poisson blending techniques or a blending map.
10 . The method of claim 7 , further comprising:
after blending the pixels, identifying an area in the mirrored version of the 3D object with unknown pixel values; and
applying inpainting techniques to populate the unknown pixel values with predicted values.
11 . The method of claim 1 , further comprising:
analyzing the 3D object comprising the target and the mirrored version of the 3D object comprising the mirrored version of the target with a quality control machine learning model to generate a score representing match accuracy between the mirrored version of the object comprising the mirrored version of the target and the 3D object comprising the target.
12 . The method of claim 11 , wherein the quality control machine learning model comprises a discriminator trained by performing training operations comprising:
analyzing the training pair of fashion items using the discriminator to generate a prediction indicative of whether the mirrored version of the training fashion item is a real mirrored version of the training fashion item or a fake mirrored version of the training fashion item;
accessing ground truth labels for the training pair of fashion items indicating whether the mirrored version of the training fashion item is real or fake; and
updating one or more parameters of the discriminator based on accuracy of the prediction in relation to the ground truth labels.
13 . The method of claim 12 , further comprising:
accessing a plurality of 3D objects representing pairs of fashion items;
selecting a subset of the plurality of 3D objects;
adding noise to an individual fashion item of each pair of fashion items in the subset of the plurality of 3D objects;
associating a first set of ground truth labels for each of the subset of the plurality of 3D objects indicating that the mirrored version of the fashion item in each of the subset of the plurality of 3D objects is fake; and
associating a second set of ground truth labels for each of the plurality of 3D objects outside of the subset of the plurality of 3D objects indicating that the mirrored version of the fashion item in each of the plurality of 3D objects outside of the subset of the plurality of 3D objects is real.
14 . The method of claim 1 , further comprising:
generating an extended reality (XR) experience using the 3D object and the new 3D object.
15 . The method of claim 1 , further comprising:
generating an artificial image depicting the mirrored version of the 3D object, the artificial image being modified using at least a portion of the new 3D object.
16 . The method of claim 1 , further comprising repeating the training operations for additional training data until a stopping criterion is met.
17 . The method of claim 1 , wherein the convolutional neural network is trained to establish a relationship between mapping of pixels in a first fashion item in a first pair of fashion items to pixels in a second fashion item in the first pair of fashion items.
18 . A system comprising:
at least one processor; and
at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving a three-dimensional (3D) object comprising a target;
analyzing the 3D object using one or more machine learning models to generate data associated with a mirrored version of the target of the 3D object, the one or more machine learning models comprising a convolutional neural network trained by performing training operations comprising:
accessing training data comprising a training pair of fashion items including a training fashion item and a ground truth mirrored version of the training fashion item;
analyzing the training fashion item to estimate a mirrored version of the training fashion item;
computing a deviation between the estimated mirrored version of the training fashion item and the ground truth mirrored version of the training fashion item; and
updating one or more parameters of the convolutional neural network based on the computed deviation;
applying the mirrored version of the target to a mirrored version of the 3D object using the generated data; and
generating a new 3D object comprising the mirrored version of the 3D object and the mirrored version of the target.
19 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a three-dimensional (3D) object comprising a target;
analyzing the 3D object using one or more machine learning models to generate data associated with a mirrored version of the target of the 3D object, the one or more machine learning models comprising a convolutional neural network trained by performing training operations comprising:
accessing training data comprising a training pair of fashion items including a training fashion item and a ground truth mirrored version of the training fashion item;
analyzing the training fashion item to estimate a mirrored version of the training fashion item;
computing a deviation between the estimated mirrored version of the training fashion item and the ground truth mirrored version of the training fashion item; and
updating one or more parameters of the convolutional neural network based on the computed deviation;
applying the mirrored version of the target to a mirrored version of the 3D object using the generated data; and
generating a new 3D object comprising the mirrored version of the 3D object and the mirrored version of the target.
20 . The non-transitory computer-readable storage medium of claim 19 , the operations comprising:
presenting a virtual object comprising the new 3D object in an augmented reality experience.