Object reconstruction with texture parsing

1. A method of generating one or more three-dimensional (3D) models, the method comprising:

obtaining an image of an object;

generating a 3D model of the object in the image, the 3 D model including geometry information;

determining, using an object parsing mask, color information for the 3D model, the object parsing mask defining one or more regions of the 3D model for color adjustment;

generating a fitted 3D model of the object at least in part by modifying the geometry information for each vertex of the 3D model and adjusting color for each vertex of the 3D model using the color information determined using the object parsing mask; and

performing a pixel-level refinement of the fitted 3D model to generate a refined 3D model of the object, the pixel-level refinement using depth information associated with the fitted 3D model to apply a pixel-level depth displacement to a portion of the fitted 3D model defined by a refinement mask, wherein the refinement mask indicates a position of the portion of the fitted 3D model for the pixel-level depth displacement.

2. The method of claim 1 , wherein the color information includes albedo color.

3. The method of claim 1 , wherein generating the 3D model of the object includes:

obtaining a generic object model;

identifying landmarks of the object in the image; and

generating the 3D model by fitting the generic object model to the object in the image using the identified landmarks of the object.

4. The method of claim 1 , wherein the fitted 3D model of the object is generated based on performing at least one vertex-level fitting of the 3D model.

5. The method of claim 4 , wherein performing the at least one vertex-level fitting of the 3D model includes performing a joint optimization of the geometry information and the color information.

6. The method of claim 4 , wherein performing the at least one vertex-level fitting of the 3D model includes:

identifying the one or more regions defined by the object parsing mask;

performing a first vertex-level fitting of the 3D model to the object, the first vertex-level fitting modifying the geometry information for each vertex of the 3D model by shifting each vertex of the 3D model to fit the object in the image, the first vertex-level fitting further determining the color information for each vertex of the 3D model; and

performing a second vertex-level fitting of the 3D model to the object, the second vertex-level fitting adjusting the color for each vertex of the 3D model included in the one or more regions identified by the object parsing mask.

7. The method of claim 6 , wherein determining the color information for each vertex of the 3D model includes:

determining, using a clustering algorithm, a plurality of candidate colors for each vertex of the 3D model included in the one or more regions identified by the object parsing mask; and

selecting, for each vertex included in the one or more regions identified by the object parsing mask, a brightest color from the plurality of candidate colors determined for each vertex.

8. The method of claim 7 , wherein the clustering algorithm includes a K-means algorithm.

9. The method of claim 1 , further comprising:

determining, using the refinement mask, the portion of the fitted 3D model for applying the pixel-level depth displacement.

10. The method of claim 9 , wherein the object includes a face, and wherein the refinement mask includes an eye mask indicating that the portion includes a region of the face outside of eye regions of the face.

11. The method of claim 1 , wherein performing the pixel-level refinement of the fitted 3D model includes:

generating an albedo image and a depth image for the fitted 3D model; and

generating the refined 3D model using the albedo image and the depth image.

12. The method of claim 1 , further comprising:

applying a smoothing function to the fitted 3D model.

13. The method of claim 12 , wherein the smoothing function includes a Discrete Laplacian Smoothing function.

14. The method of claim 1 , wherein the object is a face.

15. The method of claim 1 , further comprising:

outputting the refined 3D model of the object.

16. An apparatus for generating one or more three-dimensional (3D) models, comprising:

a memory; and

one or more processors coupled to the memory, the one or more processors being configured to:

obtain an image of an object;

generate a 3D model of the object in the image, the 3D model including geometry information;

determine, using an object parsing mask, color information for the 3D model, the object parsing mask defining one or more regions of the 3D model for color adjustment;

generate a fitted 3D model of the object at least in part by modifying the geometry information for each vertex of the 3D model and adjusting color for each vertex of the 3D model using the color information determined using the object parsing mask; and

perform a pixel-level refinement of the fitted 3D model to generate a refined 3D model of the object, the pixel-level refinement using depth information associated with the fitted 3D model to apply a pixel-level depth displacement to a portion of the fitted 3D model defined by a refinement mask, wherein the refinement mask indicates a position of the portion of the fitted 3D model for the pixel-level depth displacement.

17. The apparatus of claim 16 , wherein the color information includes albedo color.

18. The apparatus of claim 16 , wherein the one or more processors are configured to generate the fitted 3D model of the object based on performing at least one vertex-level fitting of the 3D model.

19. The apparatus of claim 18 , wherein the one or more processors are configured to perform the at least one vertex-level fitting of the 3D model at least in part by performing a joint optimization of the geometry information and the color information.

20. The apparatus of claim 18 , wherein, to perform the at least one vertex-level fitting of the 3D model, the one or more processors are configured to:

identify the one or more regions defined by the object parsing mask;

perform a first vertex-level fitting of the 3D model to the object, the first vertex-level fitting modifying the geometry information for each vertex of the 3D model by shifting each vertex of the 3D model to fit the object in the image, the first vertex-level fitting further determining the color information for each vertex of the 3D model; and

perform a second vertex-level fitting of the 3D model to the object, the second vertex-level fitting adjusting the color for each vertex of the 3D model included in the one or more regions identified by the object parsing mask.

21. The apparatus of claim 20 , wherein determining the color information for each vertex of the 3D model includes:

determining, using a clustering algorithm, a plurality of candidate colors for each vertex of the 3D model included in the one or more regions identified by the object parsing mask; and

selecting, for each vertex included in the one or more regions identified by the object parsing mask, a brightest color from the plurality of candidate colors determined for each vertex.

22. The apparatus of claim 21 , wherein the clustering algorithm includes a K-means algorithm.

23. The apparatus of claim 16 , where the one or more processors are configured to:

determine, using a refinement mask, the portion of the fitted 3D model for applying the pixel-level depth displacement.

24. The apparatus of claim 16 , wherein performing the pixel-level refinement of the fitted 3D model includes:

generating an albedo image and a depth image for the fitted 3D model; and

generating the refined 3D model using the albedo image and the depth image.

25. The apparatus of claim 16 , wherein the object is a face.

26. The apparatus of claim 16 , wherein the apparatus includes a mobile device.

27. The apparatus of claim 16 , wherein the apparatus includes a display configured to display the refined 3D model of the object.

28. The apparatus of claim 16 , wherein the apparatus includes a camera configured to capture the image.

29. The apparatus of claim 16 , wherein, to generate the 3D model of the object, the one or more processors are configured to:

obtain a generic object model;

identify landmarks of the object in the image; and

generate the 3D model by fitting the generic object model to the object in the image using the identified landmarks of the object.

30. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

obtain an image of an object;

generate a 3D model of the object in the image, the 3D model including geometry information;

determine, using an object parsing mask, color information for the 3D model, the object parsing mask defining one or more regions of the 3D model for color adjustment;

generate a fitted 3D model of the object at least in part by modifying the geometry information for each vertex of the 3D model and adjusting color for each vertex of the 3D model using the color information determined using the object parsing mask; and

perform a pixel-level refinement of the fitted 3D model to generate a refined 3D model of the object, the pixel-level refinement using depth information associated with the fitted 3D model to apply a pixel-level depth displacement to a portion of the fitted 3D model defined by a refinement mask, wherein the refinement mask indicates a position of the portion of the fitted 3D model for the pixel-level depth displacement.