3D reconstruction of a moving object

1. A method comprising, by a computing device:

reconstructing a three-dimensional shape of a target object that is moving when captured in a stream of image and depth data;

creating a first two-dimensional normal map for the three-dimensional shape of the target object based on data corresponding to one or more frames, wherein the first two-dimensional normal map comprises first pixel elements that correspond to surface locations on the target object, and each of the first pixel elements specifying a first surface normal at the corresponding surface location;

rendering a first output image based on the three-dimensional shape of the target object and the first two-dimensional normal map;

accessing image and depth data associated with the target object corresponding to a first frame after the one or more frames;

generating a first normal data associated with the target object using the image and depth data corresponding to the first frame;

creating a second two-dimensional normal map for the three-dimensional shape of the target object based on the first normal data and the first surface normals corresponding to the first pixel elements in the first two-dimensional normal map created based on the data corresponding to the one or more frames preceding the first frame, wherein the second two-dimensional normal map comprises second pixel elements that correspond to the surface locations on the target object, and each of the second pixel elements specifying a second surface normal at the corresponding surface location; and

rendering a second output image based on the three-dimensional shape of the target object and the second two-dimensional normal map.

2. The method of claim 1 , wherein reconstructing the three-dimensional shape of the target object comprises performing a Non-rigid Signed Distance Function (SDF) fusion of a stream of image data for the target object.

3. The method of claim 1 , wherein creating the second two-dimensional normal map comprises minimizing an energy function of a data term and a smooth term using gradient descent.

4. The method of claim 3 , wherein the data term measures differences between a normal on the first two-dimensional normal map and an observed normal in the first normal data generated using the image and depth data corresponding to the first frame.

5. The method of claim 3 , wherein the smooth term is for ensuring consistency between neighboring pixels.

6. The method of claim 1 , wherein rendering the second output image comprises rendering a surface of the reconstructed three-dimensional shape of the target object based on the second two-dimensional normal map.

7. The method of claim 1 , wherein the target object is a human body.

8. The method of claim 1 , further comprising rendering an avatar of the target object based on the reconstruction of the target object.

9. The method of claim 8 , wherein, when the avatar is animated to a position, the recorded normal map when the target object is in the animated position is used for rendering a surface of the animated avatar.

10. The method of claim 1 , further comprising:

maintaining a plurality of normal maps of a plurality of resolutions, each resolution corresponding to a distance between the target object and a camera capturing the image and depth data associated with the target object;

detecting the target object moves from a first distance to a second distance from the camera, wherein the second distance does not correspond to a finest resolution among the plurality of resolutions; and

determining a first resolution for the normal map corresponding to the second distance of the target object.

11. The method of claim 10 , wherein creating the second two-dimensional normal map comprises:

generating a second normal data associated with the target object using the image and depth data corresponding to a second frame;

creating a two-dimensional normal map of the first resolution corresponding to the second distance of the target object based on the second normal data and surface normals corresponding to pixel elements in a maintained two-dimensional normal map of the first resolution corresponding to the second distance of the target object; and

projecting content of the created two-dimensional normal map of the first resolution to other normal maps among the plurality of normal maps.

12. The method of claim 11 , wherein the other normal maps comprise normal maps of finer resolutions than the first resolution.

13. The method of claim 10 , wherein rendering the second output image comprises rendering a surface of the reconstructed three-dimensional shape based on a second resolution normal map that is a finest resolution normal map among the plurality of normal maps.

14. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

reconstruct a three-dimensional shape of a target object that is moving when captured in a stream of image and depth data;

create a first two-dimensional normal map for the three-dimensional shape of the target object based on data corresponding to one or more frames, wherein the first two-dimensional normal map comprises first pixel elements that correspond to surface locations on the target object, and each of the first pixel elements specifying a first surface normal at the corresponding surface location;

render a first output image based on the three-dimensional shape of the target object and the first two-dimensional normal map;

access image and depth data associated with the target object corresponding to a first frame after the one or more frames;

generate a first normal data associated with the target object using the image and depth data corresponding to the first frame;

create a second two-dimensional normal map for the three-dimensional shape of the target object based on the first normal data and the first surface normals corresponding to the first pixel elements in the first two-dimensional normal map created based on the data corresponding to the one or more frames preceding the first frame, wherein the second two-dimensional normal map comprises second pixel elements that correspond to the surface locations on the target object, and each of the second pixel elements specifying a second surface normal at the corresponding surface location; and

render a second output image based on the three-dimensional shape of the target object and the second two-dimensional normal map.

15. The computer-readable non-transitory storage media of claim 14 , wherein reconstructing the three-dimensional shape of the target object comprises performing a Non-rigid Signed Distance Function (SDF) fusion of a stream of image data for the target object.

16. The computer-readable non-transitory storage media of claim 14 , wherein creating the second two-dimensional normal map comprises minimizing an energy function of a data term and a smooth term using gradient descent.

17. The computer-readable non-transitory storage media of claim 16 , wherein the data term measures differences between a normal on the first two-dimensional normal map and an observed normal in the first normal data generated using the image and depth data corresponding to the first frame.

18. The computer-readable non-transitory storage media of claim 16 , wherein the smooth term is for ensuring consistency between neighboring pixels.

19. The computer-readable non-transitory storage media of claim 14 , wherein rendering the second output image comprises rendering a surface of the reconstructed three-dimensional shape of the target object based on the second two-dimensional normal map.

20. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

reconstruct a three-dimensional shape of a target object that is moving when captured in a stream of image and depth data;

create a first two-dimensional normal map for the three-dimensional shape of the target object based on data corresponding to one or more frames, wherein the first two-dimensional normal map comprises first pixel elements that correspond to surface locations on the target object, and each of the first pixel elements specifying a first surface normal at the corresponding surface location;

render a first output image based on the three-dimensional shape of the target object and the first two-dimensional normal map;

access image and depth data associated with the target object corresponding to a first frame after the one or more frames;

generate a first normal data associated with the target object using the image and depth data corresponding to the first frame;

create a second two-dimensional normal map for the three-dimensional shape of the target object based on the first normal data and the first surface normals corresponding to the first pixel elements in the first two-dimensional normal map created based on the data corresponding to the one or more frames preceding the first frame, wherein the second two-dimensional normal map comprises second pixel elements that correspond to the surface locations on the target object, and each of the second pixel elements specifying a second surface normal at the corresponding surface location; and

render a second output image based on the three-dimensional shape of the target object and the second two-dimensional normal map.