Hand surface normal estimation
An system for augmenting images using hand surface normal estimation is provided. In a model training phase, 3D models of hands are generated using 3D data of hands in a variety of positions. Target normal training data is generated that includes normals of surfaces of the 3D models and synthetic 2D image training data corresponding to the 3D models and the normals. The target normal training data and the synthetic image training data are used to train a normal estimation model. The normal estimation is used by an interactive application to generate augmentations that are applied to hand image data.
1 . A computer-implemented method comprising:
receiving, by a set of processors, 3D data of a hand;
generating, by the set of processors, a 3D model of the hand using the 3D data, the 3D model comprising a set of surfaces;
generating, by the set of processors, target normal training data comprising a set of normals of the set of surfaces of the 3D model;
generating, by the set of processors, synthetic 2D image training data comprising a set of synthetic 2D images using the 3D model, the set of normals, and combinations of lighting levels, lighting angles, camera angles, and camera distances; and
training, by the set of processors, a normal estimation model using the synthetic 2D image training data and the target normal training data.
2 . The computer-implemented method of claim 1 , wherein generating the synthetic 2D image training data is further using a set of camera and lighting parameters.
3 . The computer-implemented method of claim 2 , wherein the set of camera and lighting parameters comprise randomized values.
4 . The computer-implemented method of claim 1 , wherein training the normal estimation model comprises:
determining a set of cropping boundaries using the synthetic 2D image training data and a detection model; and
cropping the set of synthetic 2D images using the set of cropping boundaries.
5 . The computer-implemented method of claim 1 ,
wherein the synthetic 2D image training data comprises a set of pixels, and
wherein the set of normals comprises a respective normal for each pixel of the set of pixels.
6 . The computer-implemented method of claim 1 , further comprising:
capturing, by a second set of processors, image data of a second hand;
generating, by the second set of processors, a set of estimated normals using the image data and the normal estimation model;
generating, by the second set of processors, augmented image data using the set of estimated normals and the image data; and
providing, by the second set of processors, an augmented image to a user in a user interface using the augmented image data.
7 . The computer-implemented method of claim 6 , wherein generating the set of estimated normals comprises:
determining, by the second set of processors, a set of cropping boundaries using the image data and a detection model; and
cropping, by the second set of processors, the image data using the set of cropping boundaries.
8 . The computer-implemented method of claim 6 ,
wherein the image data of the second hand comprises a set of pixels, and
wherein the set of estimated normals comprises a respective normal for each pixel of the set of pixels.
9 . A machine comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving 3D data of a hand;
generating a 3D model of the hand using the 3D data, the 3D model comprising a set of surfaces;
generating target normal training data comprising a set of normals of the set of surfaces of the 3D model;
generating synthetic 2D image training data comprising a set of synthetic 2D images using the 3D model, the set of normals, and combinations of lighting levels, lighting angles, camera angles, and camera distances; and
training a normal estimation model using the synthetic 2D image training data and the target normal training data.
10 . The machine of claim 9 , wherein generating the synthetic 2D image training data is further using a set of camera and lighting parameters.
11 . The machine of claim 10 , wherein the set of camera and lighting parameters comprise randomized values.
12 . The machine of claim 9 , wherein training the normal estimation model comprises:
determining a set of cropping boundaries using the synthetic 2D image training data and a detection model; and
cropping the set of synthetic 2D images using the set of cropping boundaries.
13 . The machine of claim 9 ,
wherein the synthetic 2D image training data comprises a set of pixels, and
wherein the set of normals comprises a respective normal for each pixel of the set of pixels.
14 . An interactive system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the interactive system to perform operations comprising:
capturing, using a camera of the interactive system, image data of a first hand;
generating a set of estimated normals using the image data and a normal estimation model trained using operations comprising:
generating target normal training data comprising a set of normals of a set of surfaces of a 3D model of a second hand;
generating synthetic 2D image training data comprising a set of synthetic 2D images using the 3D model, the set of normals, and combinations of lighting levels, lighting angles, camera angles, and camera distances; and
training the normal estimation model using the synthetic 2D image training data and the target normal training data;
generating augmented image data using the set of estimated normals and the image data; and
providing an augmented image to a user in a user interface using the augmented image data.
15 . The interactive system of claim 14 , wherein generating the set of estimated normals comprises:
determining a set of cropping boundaries using the image data and a detection model; and
cropping the image data using the set of cropping boundaries.
16 . The interactive system of claim 14 ,
wherein the image data of the hand comprises a set of pixels, and
wherein the set of estimated normals comprises a respective normal for each pixel of the set of pixels.
17 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
receiving 3D data of a hand;
generating a 3D model of the hand using the 3D data, the 3D model comprising a set of surfaces;
generating target normal training data comprising a set of normals of the set of surfaces of the 3D model;
generating synthetic 2D image training data comprising a set of synthetic 2D images using the 3D model, the set of normals, and combinations of lighting levels, lighting angles, camera angles, and camera distances; and
training a normal estimation model using the synthetic 2D image training data and the target normal training data.
18 . The computer-readable storage medium of claim 17 , wherein generating the synthetic 2D image training data is further using a set of camera and lighting parameters.
19 . The computer-readable storage medium of claim 18 , wherein the set of camera and lighting parameters comprise randomized values.
20 . The computer-readable storage medium of claim 17 , wherein training the normal estimation model comprises:
determining a set of cropping boundaries using the synthetic 2D image training data and a detection model; and
cropping the set of synthetic 2D images using the set of cropping boundaries.