Recognizing combinations of body shape, pose, and clothing in three-dimensional input images
Certain embodiments involve recognizing combinations of body shape, pose, and clothing in three-dimensional input images. For example, synthetic training images are generated based on user inputs. These synthetic training images depict different training figures with respective combinations of a body pose, a body shape, and a clothing item. A machine learning algorithm is trained to recognize the pose-shape-clothing combinations in the synthetic training images and to generate feature descriptors describing the pose-shape-clothing combinations. The trained machine learning algorithm is outputted for use by an image manipulation application. In one example, an image manipulation application uses a feature descriptor, which is generated by the machine learning algorithm, to match an input figure in an input image to an example image based on a correspondence between a pose-shape-clothing combination of the input figure and a pose-shape-clothing combination of an example figure in the example image.
1. A method for training machine learning algorithms to match input images to example images based on combinations of body shape, body pose, and clothing item, the method comprising:
accessing, by a processing device, synthetic training images having known combinations of a training body pose, a training clothing item, and a training body shape;
training, by the processing device, a machine learning algorithm to generate training feature vectors describing the known combinations of the training body pose, the training clothing item, and the training body shape, wherein training the machine learning algorithm comprises:
selecting a set of the synthetic training images with known body pose variations and known body shape variations,
generating input depth maps based on clothing depicted in the selected set of synthetic training images,
accessing a neural network for encoding the input depth maps into the training feature vectors and decoding output depth maps from the training feature vectors, and
performing an iterative adjustment of the neural network, wherein:
(a) in a first iteration prior to an adjustment, a first training feature vector encoded by the neural network includes (i) a first set of dimension values for a first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of dimension values for a second set of dimensions representing body shape from the selected set of synthetic training images,
(b) a first difference value indicates a difference between (i) an input depth map encoded into the first training feature vector and (ii) a first output depth map decoded from the first training feature vector,
(c) in a second iteration subsequent to the adjustment, a second training feature vector encoded by the neural network includes (i) a first set of modified dimension values for the first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of modified dimension values for the second set of dimensions representing body shape from the selected set of synthetic training images,
(d) a second difference value indicates a difference between the input depth map and a second output depth map decoded from the second training feature vector, and
(e) the iterative adjustment of the neural network is completed based on the second difference value being less than the first difference value; and
outputting, by the processing device, the trained machine learning algorithm for matching (i) an input image having an input body shape and input body pose to (ii) an example image having a known combination of example body shape and example body pose.
2. The method of claim 1 , further comprising:
accessing the input image from a computer-readable medium;
computing a feature vector from the accessed input image, the feature vector having a first portion that describes the input body pose, a second portion that describes the input body shape, and a third portion that describes an input clothing item in the input image;
generating a query from the feature vector; and
selecting the example image from a database using the query.
3. The method of claim 2 , wherein the input image comprises a three-dimensional scan of a person, wherein a figure in the input image having the input body pose, the input body shape, and the input clothing item corresponds to the person.
4. The method of claim 2 , further comprising:
accessing example data describing an example body shape of an example figure from the example image and an example body pose of the example figure;
identifying, based on the example data, a difference between at least one of:
(i) the input body shape and the example body shape; or
(ii) the input body pose and the example body pose;
modifying, based on the identified difference, data describing at least one of the input body shape or the input body pose, wherein the modified data is usable for at least one of animating an input figure in the input image or modifying clothing of the input figure.
5. The method of claim 1 , further comprising generating the synthetic training images, wherein generating the synthetic training images comprises, for each synthetic training image:
accessing, with a graphical application, a deformable body model from a computer-readable medium;
modifying, with the graphical application, a body pose of the deformable body model and a body shape of the deformable body model;
accessing, with the graphical application, graphics data depicting clothing items;
applying, with the graphical application, at least some of the graphics data depicting the clothing items to the deformable body model with the modified body pose and the modified body shape; and
storing respective training data for the synthetic training image describing the deformable body model with the modified body pose, the modified body shape, and the applied graphics data, wherein the respective training data indicates the respective training combination of the training body pose, the training clothing item, and the training body shape for the synthetic training image.
6. The method of claim 1 , wherein training the machine learning algorithm further comprises:
providing the input depth maps to the machine learning algorithm;
encoding, based on the neural network, the input depth maps as feature vectors;
decoding the feature vectors into synthesized depth maps;
modifying a structure of the neural network based on differences between the synthesized depth maps and the input depth maps.
7. The method of claim 1 , wherein training the machine learning algorithm comprises:
selecting viewpoint training images, shape training images, pose training images, and clothing training images, wherein:
the viewpoint training images are a first subset of the synthetic training images and include (i) a fixed combination of pose, shape, and clothing and (ii) respective viewpoint variations,
the shape training images are a second subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body shape variations,
the pose training images are a third subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body pose variations, and
the clothing training images are a fourth subset of the synthetic training images and include (i) a fixed combination of viewpoint, pose, and shape and (ii) respective clothing variations;
modifying, based on depth maps of the viewpoint training images, a structure of the neural network such that the modified structure of the neural network encodes a viewpoint in a first feature vector portion;
modifying, based on depth maps of the shape training images, the structure of the neural network such that the modified structure of the neural network encodes a body shape in a second feature vector portion;
modifying, based on depth maps of the pose training images, the structure of the neural network such that the modified structure of the neural network encodes a body pose in a third feature vector portion; and
modifying, based on surface normal maps of the clothing training images, the structure of the neural network such that the modified structure of the neural network encodes clothing in a fourth feature vector portion.
8. A system comprising:
a non-transitory computer-readable medium storing synthetic training images having known combinations of a training body pose, a training clothing item, and a training body shape;
means for selecting a set of the synthetic training images with known body pose variations and known body shape variations;
means for encoding, via a neural network, input depth maps generated from the selected set of synthetic training images into training feature vectors and decoding output depth maps from the training feature vectors; and
means for performing an iterative adjustment of the neural network, wherein:
(a) in a first iteration prior to an adjustment, a first training feature vector encoded by the neural network includes (i) a first set of dimension values for a first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of dimension values for a second set of dimensions representing body shape from the selected set of synthetic training images,
(b) a first difference value indicates a difference between (i) an input depth map encoded into the first training feature vector and (ii) a first output depth map decoded from the first training feature vector,
(c) in a second iteration subsequent to the adjustment, a second training feature vector encoded by the neural network includes (i) a first set of modified dimension values for the first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of modified dimension values for the second set of dimensions representing body shape from the selected set of synthetic training images,
(d) a second difference value indicates a difference between the input depth map and a second output depth map decoded from the second training feature vector, and
(e) the iterative adjustment of the neural network is completed based on the second difference value being less than the first difference value; and
means for matching, with the iteratively adjusted neural network, an input image having an input body shape and input body pose to an example image having a known combination of example body shape and example body pose.
9. The system of claim 8 , wherein the means for matching comprises means for:
accessing the input image from the non-transitory computer-readable medium or another non-transitory computer-readable medium;
computing a feature vector from the accessed input image, the feature vector having a first portion that describes the input body pose, a second portion that describes the input body shape, and a third portion that describes an input clothing item in the input image;
generating a query from the feature vector; and
selecting the example image from a database using the query.
10. The system of claim 9 , wherein the input image comprises a three-dimensional scan of a person, wherein a figure in the input image having the input body pose, the input body shape, and the input clothing item corresponds to the person.
11. The system of claim 9 , further comprising a processing device configured for:
accessing example data describing an example body shape of an example figure from the example image and an example body pose of the example figure;
identifying, based on the example data, a difference between at least one of:
(i) the input body shape and the example body shape; or
(ii) the input body pose and the example body pose;
modifying, based on the identified difference, data describing at least one of the input body shape or the input body pose, wherein the modified data is usable for at least one of animating an input figure in the input image or modifying clothing of the input figure.
12. The system of claim 8 , further comprising a processing device configured for generating the synthetic training images, wherein generating the synthetic training images comprises, for each synthetic training image:
accessing, with a graphical application, a deformable body model from a computer-readable medium;
modifying, with the graphical application, a body pose of the deformable body model and a body shape of the deformable body model;
accessing, with the graphical application, graphics data depicting clothing items;
applying, with the graphical application, at least some of the graphics data depicting the clothing items to the deformable body model with the modified body pose and the modified body shape; and
storing respective training data for the synthetic training image describing the deformable body model with the modified body pose, the modified body shape, and the applied graphics data, wherein the respective training data indicates the respective training combination of the training body pose, the training clothing item, and the training body shape for the synthetic training image.
13. The system of claim 8 , wherein iteratively adjusting the neural network comprises, for one or more iterations:
encoding, based on the neural network, the input depth maps as feature vectors;
decoding the feature vectors into synthesized depth maps;
modifying a structure of the neural network based on differences between the synthesized depth maps and the input depth maps.
14. The system of claim 8 , wherein iteratively adjusting the neural network comprises, for one or more iterations:
selecting viewpoint training images, shape training images, pose training images, and clothing training images, wherein:
the viewpoint training images are a first subset of the synthetic training images and include (i) a fixed combination of pose, shape, and clothing and (ii) respective viewpoint variations,
the shape training images are a second subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body shape variations,
the pose training images are a third subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body pose variations, and
the clothing training images are a fourth subset of the synthetic training images and include (i) a fixed combination of viewpoint, pose, and shape and (ii) respective clothing variations;
modifying, based on depth maps of the viewpoint training images, a structure of the neural network such that the modified structure of the neural network encodes a viewpoint in a first feature vector portion;
modifying, based on depth maps of the shape training images, the structure of the neural network such that the modified structure of the neural network encodes a body shape in a second feature vector portion;
modifying, based on depth maps of the pose training images, the structure of the neural network such that the modified structure of the neural network encodes a body pose in a third feature vector portion; and
modifying, based on surface normal maps of the clothing training images, the structure of the neural network such that the modified structure of the neural network encodes clothing in a fourth feature vector portion.
15. A non-transitory computer-readable medium having program code that is stored thereon and that is executable by a processing device for performing operations, the operations comprising:
accessing synthetic training images having known combinations of a training body pose, a training clothing item, and a training body shape;
training a machine learning algorithm to generate training feature vectors describing the known combinations of the training body pose, the training clothing item, and the training body shape, wherein training the machine learning algorithm comprises:
selecting a set of the synthetic training images with known body pose variations and known body shape variations,
generating input depth maps based on clothing depicted in the selected set of synthetic training images,
accessing a neural network for encoding the input depth maps into the training feature vectors and decoding output depth maps from the training feature vectors, and
performing an iterative adjustment of the neural network, wherein:
(a) in a first iteration prior to an adjustment, a first training feature vector encoded by the neural network includes (i) a first set of dimension values for a first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of dimension values for a second set of dimensions representing body shape from the selected set of synthetic training images,
(b) a first difference value indicates a difference between (i) an input depth map encoded into the first training feature vector and (ii) a first output depth map decoded from the first training feature vector,
(c) in a second iteration subsequent to the adjustment, a second training feature vector encoded by the neural network includes (i) a first set of modified dimension values for the first set of dimensions representing body pose from the selected set of synthetic training images and (ii) a second set of modified dimension values for the second set of dimensions representing body shape from the selected set of synthetic training images,
(d) a second difference value indicates a difference between the input depth map and a second output depth map decoded from the second training feature vector, and
(e) the iterative adjustment of the neural network is completed based on the second difference value being less than the first difference value; and
matching, with the trained machine learning algorithm, (i) an input image having an input body shape and input body pose to (ii) an example image having a known combination of example body shape and example body pose.
16. The non-transitory computer-readable medium of claim 15 , wherein the matching comprises:
accessing the input image from a computer-readable medium;
computing a feature vector from the accessed input image, the feature vector having a first portion that describes the input body pose, a second portion that describes the input body shape, and a third portion that describes an input clothing item in the input image;
generating a query from the feature vector; and
selecting the example image from a database using the query.
17. The non-transitory computer-readable medium of claim 16 , the operations further comprising:
accessing example data describing an example body shape of an example figure from the example image and an example body pose of the example figure;
identifying, based on the example data, a difference between at least one of:
(i) the input body shape and the example body shape; or
(ii) the input body pose and the example body pose;
modifying, based on the identified difference, data describing at least one of the input body shape or the input body pose, wherein the modified data is usable for at least one of animating an input figure in the input image or modifying clothing of the input figure.
18. The non-transitory computer-readable medium of claim 15 , the operations further comprising generating the synthetic training images, wherein generating the synthetic training images comprises, for each synthetic training image:
accessing, with a graphical application, a deformable body model from a computer-readable medium;
modifying, with the graphical application, a body pose of the deformable body model and a body shape of the deformable body model;
accessing, with the graphical application, graphics data depicting clothing items;
applying, with the graphical application, at least some of the graphics data depicting the clothing items to the deformable body model with the modified body pose and the modified body shape; and
storing respective training data for the synthetic training image describing the deformable body model with the modified body pose, the modified body shape, and the applied graphics data, wherein the respective training data indicates the respective training combination of the training body pose, the training clothing item, and the training body shape for the synthetic training image.
19. The non-transitory computer-readable medium of claim 15 , wherein training the machine learning algorithm further comprises:
providing the input depth maps to the machine learning algorithm;
encoding, based on the neural network, the input depth maps as feature vectors;
decoding the feature vectors into synthesized depth maps;
modifying a structure of the neural network based on differences between the synthesized depth maps and the input depth maps.
20. The non-transitory computer-readable medium of claim 15 , wherein training the machine learning algorithm comprises:
selecting viewpoint training images, shape training images, pose training images, and clothing training images, wherein:
the viewpoint training images are a first subset of the synthetic training images and include (i) a fixed combination of pose, shape, and clothing and (ii) respective viewpoint variations,
the shape training images are a second subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body shape variations,
the pose training images are a third subset of the synthetic training images and include (i) a fixed combination of viewpoint, shape, and clothing and (ii) respective body pose variations, and
the clothing training images are a fourth subset of the synthetic training images and include (i) a fixed combination of viewpoint, pose, and shape and (ii) respective clothing variations;
modifying, based on depth maps of the viewpoint training images, a structure of the neural network such that the modified structure of the neural network encodes a viewpoint in a first feature vector portion;
modifying, based on depth maps of the shape training images, the structure of the neural network such that the modified structure of the neural network encodes a body shape in a second feature vector portion;
modifying, based on depth maps of the pose training images, the structure of the neural network such that the modified structure of the neural network encodes a body pose in a third feature vector portion; and
modifying, based on surface normal maps of the clothing training images, the structure of the neural network such that the modified structure of the neural network encodes clothing in a fourth feature vector portion.