Three dimensional rendering systems and methods from monocular image

1. A human rendering system, comprising:

a feature module configured to generate a feature map based on a single image taken from a point of view (POV) including a human based on features of the human visible in the image and non-visible features of the human;

a pixel features module configured to generate pixel features based on the feature map and a target POV that is different than the POV of the image;

a feature mesh module configured to generate a feature mesh for the human based on the feature map;

a geometry module configured to:

generate voxel features based on the feature mesh for the human; and

generate a density value based on the voxel features and the pixel features;

a texture module configured to generate red green blue (RGB) colors for pixels based on the density value and the pixel features; and

a rendering module configured to generate a three dimensional (3D) rendering of the human in the image from the target POV based on the RGB colors and the density value.

2. The human rendering system of claim 1 wherein the rendering module is configured to generate the 3D rendering of the human using a volume rendering algorithm.

3. The human rendering system of claim 1 wherein the feature module includes the ResNet-18 convolutional neural network.

4. The human rendering system of claim 1 wherein the texture module is configured to generate the RGB colors for the pixels further based on a distance to a root joint of the human.

5. The human rendering system of claim 4 wherein the root joint is a pelvis of the human.

6. The human rendering system of claim 4 wherein the texture module is configured to generate the RGB colors for the pixels further based on the target POV.

7. The human rendering system of claim 1 wherein the geometry module is configured to generate the density value further based on a distance to a root joint of the human.

8. The human rendering system of claim 1 wherein the texture module is configured to generate the RGB colors for the pixels using a multilayer perceptron (MLP) network.

9. The human rendering system of claim 1 wherein the geometry module is configured to generate the density value using a multilayer perceptron (MLP) network.

10. The human rendering system of claim 9 wherein the MLP network includes N layers where N is an integer greater than 2 and is configured to output the density value from a second to last one of the N layers of the MLP network.

11. The human rendering system of claim 1 further comprising an inpainter module configured to determine colors for vertices, respectively, of the feature mesh.

12. The human rendering system of claim 11 wherein the inpainter module determine the colors for the vertices, respectively, and generates the density value using a multilayer perceptron (MLP) network.

13. The human rendering system of claim 11 further comprising a training module configured to selectively adjust one or more parameters of the feature mesh module based on minimizing differences between (a) the colors for the vertices of the feature mesh and (b) ground truth colors for the vertices of the feature mesh.

14. The human rendering system of claim 1 further comprising a training module configured to selectively adjust one or more parameters of the texture module based on minimizing differences between (a) the RGB colors for the pixels and (b) ground truth RGB colors for the pixels.

15. The human rendering system of claim 1 wherein the rendering module is configured to generate the 3D rendering of the human in the image from the target POV based on the RGB colors and the density value without the human rendering system receiving any other images including the human.

16. The human rendering system of claim 15 further comprising a training module configured to train the rendering module using two loss functions (a) an inpainting loss and (b) a rendering loss.

17. The human rendering system of claim 1 wherein the feature mesh module is configured to generate the feature mesh for the human further based on a pre-estimated mesh.

18. A body rendering system, comprising:

a feature module configured to generate a feature map based on a single image taken from a point of view (POV) including a body based on features of the body visible in the image and non-visible features of the body in the image;

a pixel features module configured to generate pixel features based on the feature map and a target POV that is different than the POV of the image;

a feature mesh module configured to generate a feature mesh for the body based on the feature map;

a geometry module configured to:

generate voxel features based on the feature mesh for the body; and

generate a density value based on the voxel features and the pixel features;

a texture module configured to generate pixel values for pixels based on the density value and the pixel features; and

a rendering module configured to generate a three dimensional (3D) rendering of the body in the image from the target POV based on the pixel values and the density value.

19. The body rendering system of claim 18 wherein the body visible in the image is a human body.

20. The body rendering system of claim 18 wherein the pixel values correspond to one of RGB color values and grayscale values.

21. A human rendering method, comprising:

generating a feature map based on a single image taken from a point of view (POV) including a human based on features of the human visible in the image and non-visible features of the human;

generating pixel features based on the feature map and a target POV that is different than the POV of the image;

generating a feature mesh for the human based on the feature map;

generating voxel features based on the feature mesh for the human;

generating a density value based on the voxel features and the pixel features;

generating red green blue (RGB) colors for pixels based on the density value and the pixel features; and

generating a three dimensional (3D) rendering of the human in the image from the target POV based on the RGB colors and the density value.

22. The human rendering method of claim 21 wherein generating the 3D rendering includes generating the 3D rendering of the human using a volume rendering algorithm.

23. The human rendering method of claim 21 wherein generating the feature map includes generating the feature map using the ResNet-18 convolutional neural network.

24. The human rendering method of claim 21 wherein generating the RGB colors includes generating the RGB colors for the pixels further based on a distance to a root joint of the human.

25. The human rendering method of claim 24 wherein the root joint is a pelvis of the human.

26. The human rendering method of claim 24 wherein generating the RGB colors includes generating the RGB colors for the pixels further based on the target POV.

27. The human rendering method of claim 21 wherein generating the density value includes generating the density value further based on a distance to a root joint of the human.

28. The human rendering method of claim 21 wherein generating the RGB colors includes generating the RGB colors for the pixels using a multilayer perceptron (MLP) network.

29. The human rendering method of claim 21 wherein generating the density value includes generating the density value using a multilayer perceptron (MLP) network.

30. The human rendering method of claim 29 wherein the MLP network includes N layers where N is an integer greater than 2 and is configured to output the density value from a second to last one of the N layers of the MLP network.

31. The human rendering method of claim 21 further comprising determining colors for vertices, respectively, of the feature mesh.

32. The human rendering method of claim 31 wherein determining the colors includes determining the colors for the vertices, respectively, and the human rendering method further includes generating the density value using a multilayer perceptron (MLP) network.

33. The human rendering method of claim 31 further comprising selectively adjusting one or more parameters based on minimizing differences between (a) the colors for the vertices of the feature mesh and (b) ground truth colors for the vertices of the feature mesh.

34. The human rendering method of claim 21 further comprising selectively adjusting one or more parameters based on minimizing differences between (a) the RGB colors for the pixels and (b) ground truth RGB colors for the pixels.

35. The human rendering method of claim 21 wherein generating the 3D rendering includes generating the 3D rendering of the human in the image from the target POV based on the RGB colors and the density value independently of any other images including the human.

36. The human rendering method of claim 35 further comprising training for the 3D rendering of the human using two loss functions (a) an inpainting loss and (b) a rendering loss.

37. The human rendering method of claim 21 wherein generating the feature mesh includes generating the feature mesh for the human further based on a pre-estimated mesh.

38. A body rendering method, comprising:

generating a feature map based on a single image taken from a point of view (POV) including a body based on features of the body visible in the image and non-visible features of the body in the image;

generating pixel features based on the feature map and a target POV that is different than the POV of the image;

generating a feature mesh for the body based on the feature map;

generating voxel features based on the feature mesh for the body;

generating a density value based on the voxel features and the pixel features;

generating pixel values for pixels based on the density value and the pixel features; and

generating a three dimensional (3D) rendering of the body in the image from the target POV based on the pixel values and the density value.

39. The body rendering method of claim 38 wherein the body visible in the image is a human body.

40. The body rendering method of claim 38 wherein the pixel values correspond to one of RGB color values and grayscale values.