Machine learning-based generation of three-dimensional models
Systems and techniques are disclosed for generating a three-dimensional (3D) model. For example, a process can include estimating a plurality of features associated with at least a portion of images; inverse warping the plurality of features into reference pose features having a reference pose; generating filtered reference pose features by selecting features from the reference pose features based on a distance of the selected features from corresponding features from the reference pose; generating modified reference pose features by modifying the filtered reference pose features based on a feature grid associated with a reference model associated with the reference pose; projecting the filtered reference pose features into one or more two dimensional (2D) planes; identifying first features associated with the person from the one or more 2D planes; and generating a 3D model of the person having a pose using the first features, the modified reference pose features, and pose information.
1 . An apparatus for generating a mutable three-dimensional (3D) model, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
estimate a plurality of features associated with images of a person;
inverse warp the plurality of features into reference pose features having a reference pose;
generate filtered reference pose features by selecting features from the reference pose features based on a distance of the selected features from corresponding features from the reference pose;
generate modified reference pose features by modifying the filtered reference pose features based on corresponding features of a reference model associated with the reference pose, wherein the reference model is mapped to a feature grid;
project the filtered reference pose features into one or more two dimensional (2D) planes;
identify first features associated with the person from the one or more 2D planes; and
generate a 3D model of the person having a pose using the first features from the one or more 2D planes, the modified reference pose features, and pose information corresponding to the pose.
2 . The apparatus of claim 1 , wherein the at least one processor is configured to:
select one or more features from the filtered reference pose features corresponding to a first reference feature of the reference model; and
interpolate the one or more features to represent the first reference feature in the modified reference pose features.
3 . The apparatus of claim 1 , wherein the at least one processor is configured to:
estimate a position of a plurality of bones in an image using a first model; and
estimate positions of the plurality of features based on the position of the plurality of bones using a second model, wherein the plurality of features corresponds to a surface of the person in the image.
4 . The apparatus of claim 1 , wherein the reference pose features comprise a T-pose.
5 . The apparatus of claim 1 , wherein the feature grid associated with the reference model comprises features anchored on a skinned multi-person linear model in the reference pose.
6 . The apparatus of claim 1 , wherein the at least one processor is configured to:
add a positional encoding to each of the modified reference pose features.
7 . The apparatus of claim 1 , wherein the at least one processor is configured to:
fine tune parameters of a machine learning model based on the first features extracted from one or more of the images.
8 . The apparatus of claim 1 , wherein the reference model includes a machine learning model trained from a dataset including images of people having different physical characteristics.
9 . The apparatus of claim 8 , wherein the machine learning model includes a multilayer perceptron configured to generate the 3D model based on the first features, the modified reference pose features, and the pose information.
10 . The apparatus of claim 8 , wherein the at least one processor is configured to:
warp, use the machine learning model, points in the modified reference pose features and the first features based on the pose information.
11 . The apparatus of claim 1 , wherein the at least one processor is configured to:
query each plane of the one or more 2D planes for information related to a feature in the filtered reference pose features to identify the first features.
12 . The apparatus of claim 1 , wherein the at least one processor is configured to:
combine the filtered reference pose features and the first features into combined features; and
provide the combined features to a machine learning model to generate the 3D model of the person.
13 . A method for generating a mutable three-dimensional (3D) model, comprising:
estimating a plurality of features associated with images of a person;
inverse warping the plurality of features into reference pose features having a reference pose;
generating filtered reference pose features by selecting features from the reference pose features based on a distance of the selected features from corresponding features from the reference pose;
generating modified reference pose features by modifying the filtered reference pose features based on corresponding features of a reference model associated with the reference pose, wherein the reference model is mapped to a feature grid;
projecting the filtered reference pose features into one or more two dimensional (2D) planes;
identifying first features associated with the person from the one or more 2D planes; and
generating a 3D model of the person having a pose using the first features from the one or more 2D planes, the modified reference pose features, and pose information corresponding to the pose.
14 . The method of claim 13 , wherein modifying the filtered reference pose features based on the feature grid comprises:
selecting one or more features from the filtered reference pose features corresponding to a first reference feature of the reference model; and
interpolating the one or more features to represent the first reference feature in the modified reference pose features.
15 . The method of claim 13 , wherein estimating the plurality of features comprises:
estimating a position of a plurality of bones in an image using a first model; and
estimating positions of the plurality of features based on the position of the plurality of bones using a second model, wherein the plurality of features corresponds to a surface of the person in the image.
16 . The method of claim 13 , wherein the feature grid associated with the reference model comprises features anchored on a skinned multi-person linear model in the reference pose.
17 . The method of claim 13 , further comprising:
adding a positional encoding to each of the modified reference pose features.
18 . The method of claim 13 , wherein identifying the first features comprises:
fine tuning parameters of a machine learning model based on the first features extracted from one or more of the images.
19 . The method of claim 13 , wherein the reference model includes a machine learning model trained from a dataset including images of people having different physical characteristics.
20 . The method of claim 19 , further comprising:
warping, using the machine learning model, points in the modified reference pose features and the first features based on the pose information.