Neural network based reconstruction of three-dimensional representation of human body from an image
A system reconstructs accurate three-dimensional (3D) representation of human body from a single-view RGB-D image that includes color information and depth information. The system uses the depth information to generate a three-dimensional voxel grid and align image features with the voxels to generate voxel aligned features. The system learns geometric details of a human body from both pixel-aligned features and voxel-aligned features. The system integrates surface normal and human body semantic information to increase the accuracy of the reconstructed human body shapes. The system generates a high-fidelity 3D human shape that present high level of details that are significantly close to 3D scan captures.
1 . A method comprising:
receiving, by a model, an input image showing a human body shape, the input image including color information and depth information of pixels, wherein the model is trained to predict a signed depth field of the human body shape shown in the input image;
generating a set of high-resolution features from the color information of the input image;
generating a set of low-resolution features from the color information of the input image;
generating a three-dimensional voxel grid from the depth information of the input image;
generating a set of voxel-aligned features from the three-dimensional voxel grid and the set of low-resolution features;
generating the signed depth field of the human body shape shown in the input image based on the set of high-resolution features and the set of voxel-aligned features, wherein the signed depth field represents a value based on a distance of a point from a surface of the human body shape; and
rendering, in a virtual reality system, a three-dimensional human body representation reconstructed based on the signed depth field.
2 . The method of claim 1 , wherein the set of high-resolution features is generated using a high-resolution stacked hourglass neural network component of the model and the set of low-resolution features is generated using a low-resolution stacked hourglass neural network component of the model.
3 . The method of claim 1 , wherein the set of voxel-aligned features is generated from the three-dimensional voxel grid and the set of low-resolution features using a sparse convolution network component of the model.
4 . The method of claim 1 , further comprising:
performing bilinear interpolation of the set of high-resolution features; and
performing trilinear interpolation of the set of voxel-aligned features.
5 . The method of claim 1 , wherein the signed depth field of the human body shape shown in the input image is generated using a multi-layered perceptron component of the model configured to receive the set of high-resolution features and the set of voxel-aligned features as input.
6 . The method of claim 1 , further comprising:
determining body parts associated with different portions of the input image;
determining a set of three-dimensional points based on the input image, wherein a density of the set of three-dimensional points associated with a portion of the input image is based on a type of body part associated with the portion of the input image; and
training the model via the set of three-dimensional points.
7 . The method of claim 1 , wherein the model is trained by minimizing:
a first loss function representing a measure of a difference between an estimated signed depth field and a signed depth field available as ground truth of a training data; and
a second loss function representing a measure of a difference between three-dimensional points predicted using the model and three-dimensional points estimated as a point cloud determined from the input image.
8 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving, by a model, an input image showing a human body shape, the input image including color information and depth information of pixels, wherein the model is trained to predict a signed depth field of the human body shape shown in the input image;
generating a set of high-resolution features from the color information of the input image;
generating a set of low-resolution features from the color information of the input image;
generating a three-dimensional voxel grid from the depth information of the input image;
generating a set of voxel-aligned features from the three-dimensional voxel grid and the set of low-resolution features;
generating the signed depth field of the human body shape shown in the input image based on the set of high-resolution features and the set of voxel-aligned features, wherein the signed depth field represents a value based on a distance of a point from a surface of the human body shape; and
rendering, in a virtual reality system, a three-dimensional human body representation reconstructed based on the signed depth field.
9 . The computer program product of claim 8 , wherein the set of high-resolution features is generated using a high-resolution stacked hourglass neural network component of the model and the set of low-resolution features is generated using a low-resolution stacked hourglass neural network component of the model.
10 . The computer program product of claim 8 , wherein the set of voxel-aligned features is generated from the three-dimensional voxel grid and the set of low-resolution features using a sparse convolution network component of the model.
11 . The computer program product of claim 8 , wherein the instructions further cause the one or more processors to perform steps comprising:
performing bilinear interpolation of the set of high-resolution features; and
performing trilinear interpolation of the set of voxel-aligned features.
12 . The computer program product of claim 8 , wherein the signed depth field of the human body shape shown in the input image is generated using a multi-layered perceptron component of the model configured to receive the set of high-resolution features and the set of voxel-aligned features as input.
13 . The computer program product of claim 8 , wherein the instructions further cause the one or more processors to perform steps comprising:
determining body parts associated with different portions of the input image;
determining a set of three-dimensional points based on the input image, wherein a density of the set of three-dimensional points associated with a portion of the input image is based on a type of body part associated with the portion of the input image; and
training the model via the set of three-dimensional points.
14 . The computer program product of claim 8 , wherein the model is trained by minimizing:
a first loss function representing a measure of a difference between an estimated signed depth field and a signed depth field available as ground truth of a training data; and
a second loss function representing a measure of a difference between three-dimensional points predicted using the model and three-dimensional points estimated as a point cloud determined from the input image.
15 . A system comprising:
one or more processors; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:
receiving, by a model, an input image showing a human body shape, the input image including color information and depth information of pixels, wherein the model is trained to predict a signed depth field of the human body shape shown in the input image;
generating a set of high-resolution features from the color information of the input image;
generating a set of low-resolution features from the color information of the input image;
generating a three-dimensional voxel grid from the depth information of the input image;
generating a set of voxel-aligned features from the three-dimensional voxel grid and the set of low-resolution features;
generating the signed depth field of the human body shape shown in the input image based on the set of high-resolution features and the set of voxel-aligned features, wherein the signed depth field represents a value based on a distance of a point from a surface of the human body shape; and
rendering, in a virtual reality system, a three-dimensional human body representation reconstructed based on the signed depth field.
16 . The system of claim 15 , wherein the set of high-resolution features is generated using a high-resolution stacked hourglass neural network component of the model, the set of low-resolution features is generated using a low-resolution stacked hourglass neural network component of the model, and the set of voxel-aligned features is generated from the three-dimensional voxel grid and the set of low-resolution features using a sparse convolution network component of the model.
17 . The system of claim 15 , wherein the instructions further cause the one or more processors to perform steps comprising:
performing bilinear interpolation of the set of high-resolution features; and
performing trilinear interpolation of the set of voxel-aligned features.
18 . The system of claim 15 , wherein the signed depth field of the human body shape shown in the input image is generated using a multi-layered perceptron component of the model configured to receive the set of high-resolution features and the set of voxel-aligned features as input.
19 . The system of claim 15 , wherein the instructions further cause the one or more processors to perform steps comprising:
determining body parts associated with different portions of the input image;
determining a set of three-dimensional points based on the input image, wherein a density of the set of three-dimensional points associated with a portion of the input image is based on a type of body part associated with the portion of the input image; and
training the model via the set of three-dimensional points.
20 . The system of claim 15 , wherein the model is trained by minimizing:
a first loss function representing a measure of a difference between an estimated signed depth field and a signed depth field available as ground truth of a training data; and
a second loss function representing a measure of a difference between three-dimensional points predicted using the model and three-dimensional points estimated as a point cloud determined from the input image.