Three-dimensional mesh generator based on two-dimensional image

1 . A non-transitory medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

obtaining a two-dimensional image of a person that is generated by a camera;

generating, based on the two-dimensional image, a coarse segmentation map that represents an outline of the person and a plurality of joint heatmaps, each of which represents a different one of a plurality of joints of the person;

applying a neural network to the two-dimensional image, the coarse segmentation map, and/or the plurality of joint heatmaps, so as to produce multiple two-dimensional maps,

wherein the multiple two-dimensional maps include

(i) a plurality of normal maps,

(ii) a distance map that provides, for each pixel in the two-dimensional image, distance to a reference plane, and

(iii) a thickness map that provides, for each pixel in the two-dimensional image, distance to a corresponding point along a back surface of a three-dimensional mesh; and

generating the three-dimensional mesh for the person by—

(a) isolating pixels in the two-dimensional image that form part of the person,

(b) forming a front surface of the three-dimensional mesh using at least one of the plurality of normal maps and the distance map, and

(c) forming the back surface of the three-dimensional mesh using at least one of the plurality of normal maps and the thickness map with reference to the front surface of the three-dimensional mesh.

2 . The non-transitory medium of claim 1 , wherein the multiple two-dimensional maps further include:

a first intensity map that represents, on a per-pixel basis, intensity of red in the two-dimensional image,

a second intensity map that represents, on a per-pixel basis, intensity of green in the two-dimensional image, and

a third intensity map that represents, on a per-pixel basis, intensity of blue in the two-dimensional image.

3 . The non-transitory medium of claim 1 , wherein said obtaining comprises:

receiving the two-dimensional image from a source external to a computing device of which the processor is a part via a communications interface.

4 . The non-transitory medium of claim 1 , wherein the operations further comprise:

processing the two-dimensional image so as to remove shadows and/or additional source lighting in the two-dimensional image.

5 . The non-transitory medium of claim 1 , wherein the operations further comprise:

setting the reference plane to be a plane other than a camera plane that corresponds to the camera.

6 . The non-transitory medium of claim 1 , wherein the camera is part of a same computing device as the processor.

7 . The non-transitory medium of claim 1 ,

wherein the plurality of normal maps includes:

a front red map that represents intensity of red across a front surface of the person,

a front green map that represents intensity of green across the front surface of the person,

a front blue map that represents intensity of blue across the front surface of the person,

a back red map that represents intensity of red across a back surface of the person,

a back green map that represents intensity of green across the back surface of the person, and

a back blue map that represents intensity of blue across the back surface of the person.

8 . The non-transitory medium of claim 1 , wherein the multiple two-dimensional maps further include a fine segmentation map that represents the outline of the person with greater accuracy than the coarse segmentation map, and wherein said isolating is based on the fine segmentation map such that pixels outside of a segmentation are discarded.

9 . The non-transitory medium of claim 1 , wherein the neural network is a fully convolutional network that includes two stacked U-nets.

10 . The non-transitory medium of claim 1

wherein the multiple two-dimensional maps further include:

a first intensity map that represents, on a per-pixel basis, intensity of red in the two-dimensional image,

a second intensity map that represents, on a per-pixel basis, intensity of green in the two-dimensional image, and

a third intensity map that represents, on a per-pixel basis, intensity of blue in the two-dimensional image, and

wherein the neural network is also applied to the first, second, and third intensity maps to produce the multiple two-dimensional maps.

11 . The non-transitory medium of claim 10 , further comprising:

using at least one of the multiple two-dimensional maps to improve a training process of the neural network.

12 . The non-transitory medium of claim 1 , wherein the plurality of joint heatmaps include separate heatmaps for at least two of a left eye, a right eye, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle, a right ankle, a left toe, and a right toe.

13 . The non-transitory medium of claim 1 , wherein the plurality of normal maps includes:

for a first surface of the person,

a first red map that indicates intensity of red colors across the first surface,

a first green map that indicates intensity of green colors across the first surface, and

a first blue map that indicates intensity of blue colors across the first surface, and

for a second surface of the person,

a second red map that indicates intensity of red colors across the second surface,

a second green map that indicates intensity of green colors across the second surface, and

a second blue map that indicates intensity of blue colors across the second surface.

14 . The non-transitory medium of claim 13 , wherein the operations further comprise:

adding color to the three-dimensional mesh based on the first red map, the first green map, and the first blue map produced for the first surface of the person and the second red map, the second green map, and the second blue map produced for the second surface of the person.