3D reconstruction of a moving object

1. A method comprising, by a computing device:

reconstructing a three-dimensional shape of a moving object that is at a first distance from a camera capturing image and depth data;

creating a first two-dimensional normal map for the three-dimensional shape of the moving object based on first image and depth data captured by the camera, wherein the first two-dimensional normal map comprises pixel elements that correspond to surface locations on the moving object, and each of the pixel elements specifying a surface normal at the corresponding surface location, and wherein the first two-dimensional normal map has a first resolution corresponding to the first distance;

maintaining a second two-dimensional normal map of a second resolution corresponding to a second distance from the camera, wherein the second resolution is lower than the first resolution;

capturing one or more frames of second image and depth data of the moving object that has moved from the first distance to the second distance from the camera;

generating a normal data associated with the moving object using the second image and depth data;

creating a third two-dimensional normal map of the second resolution for the three-dimensional shape of the moving object based on the normal data and the surface normals corresponding to pixel elements in the maintained second two-dimensional normal map;

updating the first two-dimensional normal map by projecting updated content of the third two-dimensional normal map in comparison with the second two-dimensional normal map to the first two-dimensional normal map; and

rendering an output image based on the three-dimensional shape of the moving object and the first two-dimensional normal map.

2. The method of claim 1 , wherein reconstructing the three-dimensional shape of the moving object comprises performing a Non-rigid Signed Distance Function (SDF) fusion of a stream of image data for the moving object.

3. The method of claim 1 , wherein creating the third two-dimensional normal map comprises minimizing an energy function of a data term and a smooth term using gradient descent.

4. The method of claim 3 , wherein the data term measures differences between a normal on the second two-dimensional normal map and an observed normal in the normal data generated using the second image and depth data.

5. The method of claim 3 , wherein the smooth term is for ensuring consistency between neighboring pixels.

6. The method of claim 1 , wherein rendering the output image comprises rendering a surface of the reconstructed three-dimensional shape of the moving object based on the first two-dimensional normal map.

7. The method of claim 1 , wherein the moving object is a human body.

8. The method of claim 1 , further comprising rendering an avatar of the moving object based on the reconstruction of the moving object.

9. The method of claim 8 , wherein, when the avatar is animated to a position, the recorded normal map when the moving object is in the animated position is used for rendering a surface of the animated avatar.

10. The method of claim 1 , further comprising:

maintaining one or more two-dimensional normal maps of one or more resolutions in addition to the first resolution and the second resolution, each of the one or more resolutions corresponding to a distance between the moving object and the camera capturing the image and depth data; and

updating each maintained two-dimensional normal map by projecting the updated content of the third two-dimensional normal map in comparison with the second two-dimensional normal map to the maintained two-dimensional normal map.

11. The method of claim 10 , wherein the first resolution is a finest resolution among resolutions corresponding to maintained two-dimensional normal maps.

12. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

reconstruct a three-dimensional shape of a moving object that is at a first distance from a camera capturing image and depth data;

create a first two-dimensional normal map for the three-dimensional shape of the moving object based on first image and depth data captured by the camera, wherein the first two-dimensional normal map comprises pixel elements that correspond to surface locations on the moving object, and each of the pixel elements specifying a surface normal at the corresponding surface location, and wherein the first two-dimensional normal map has a first resolution corresponding to the first distance;

maintain a second two-dimensional normal map of a second resolution corresponding to a second distance from the camera, wherein the second resolution is lower than the first resolution;

capture one or more frames of second image and depth data of the moving object that has moved from the first distance to the second distance from the camera;

generate a normal data associated with the moving object using the second image and depth data;

create a third two-dimensional normal map of the second resolution for the three-dimensional shape of the moving object based on the normal data and the surface normals corresponding to pixel elements in the maintained second two-dimensional normal map;

update the first two-dimensional normal map by projecting updated content of the third two-dimensional normal map in comparison with the second two-dimensional normal map to the first two-dimensional normal map; and

render an output image based on the three-dimensional shape of the moving object and the first two-dimensional normal map.

13. The computer-readable non-transitory storage media of claim 12 , wherein reconstructing the three-dimensional shape of the moving object comprises performing a Non-rigid Signed Distance Function (SDF) fusion of a stream of image data for the moving object.

14. The computer-readable non-transitory storage media of claim 12 , wherein creating the third two-dimensional normal map comprises minimizing an energy function of a data term and a smooth term using gradient descent.

15. The computer-readable non-transitory storage media of claim 14 , wherein the data term measures differences between a normal on the second two-dimensional normal map and an observed normal in the normal data generated using the second image and depth data.

16. The computer-readable non-transitory storage media of claim 14 , wherein the smooth term is for ensuring consistency between neighboring pixels.

17. The computer-readable non-transitory storage media of claim 12 , wherein rendering the output image comprises rendering a surface of the reconstructed three-dimensional shape of the moving object based on the first two-dimensional normal map.

18. The computer-readable non-transitory storage media of claim 12 , wherein the moving object is a human body.

19. The computer-readable non-transitory storage media of claim 12 , further comprising rendering an avatar of the moving object based on the reconstruction of the moving object.

20. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

reconstruct a three-dimensional shape of a moving object that is at a first distance from a camera capturing image and depth data;

create a first two-dimensional normal map for the three-dimensional shape of the moving object based on first image and depth data captured by the camera, wherein the first two-dimensional normal map comprises pixel elements that correspond to surface locations on the moving object, and each of the pixel elements specifying a surface normal at the corresponding surface location, and wherein the first two-dimensional normal map has a first resolution corresponding to the first distance;

maintain a second two-dimensional normal map of a second resolution corresponding to a second distance from the camera, wherein the second resolution is lower than the first resolution;

capture one or more frames of second image and depth data of the moving object that has moved from the first distance to the second distance from the camera;

generate a normal data associated with the moving object using the second image and depth data;

create a third two-dimensional normal map of the second resolution for the three-dimensional shape of the moving object based on the normal data and the surface normals corresponding to pixel elements in the maintained second two-dimensional normal map;

update the first two-dimensional normal map by projecting updated content of the third two-dimensional normal map in comparison with the second two-dimensional normal map to the first two-dimensional normal map; and

render an output image based on the three-dimensional shape of the moving object and the first two-dimensional normal map.