Facial model mapping with a neural network trained on varying levels of detail of facial scans
In an image processing system, a scan of an actor is converted to a high-resolution two-dimensional map, which is converted to low-resolution map and to a facial rig model. Manipulations of the facial rig create a modified facial rig. A new low-resolution two-dimensional map can be obtained of the modified facial rig and a neural network can be used to generate a new high-resolution two-dimensional map that can be used to generate a mesh that is a mesh of the scan, modified by the manipulations of the facial rig.
1. A computer-implemented method for processing data derived from scans of live actors, the method comprising:
obtaining pose data corresponding to a pose of an object in a first pose, wherein the object corresponds to a part of an actor and the pose data corresponds to a scan of the part of the actor;
determining a set of feature values for positions or characteristics of features of the object;
generating a first two-dimensional mapping of the set of feature values from the pose data to a high-resolution two-dimensional map;
generating a second two-dimensional mapping from the first two-dimensional mapping at a lower resolution than the first two-dimensional mapping;
generating, from the second two-dimensional mapping, an object rig having a first object rig state corresponding to the first pose for the object;
obtaining artist modifications, in an object rig space, to the object rig to form a second object rig state;
generating a third two-dimensional mapping representing the artist modifications to the object rig;
applying the third two-dimensional mapping to a neural network trained on differing resolution levels of a set of two-dimensional maps, to form a fourth two-dimensional mapping having a higher resolution than the third two-dimensional mapping; and
generating a mesh, defined in a three-dimensional space, corresponding to the pose data modified according to the artist modifications made in the object rig space, wherein the three-dimensional space in which the mesh is defined is a space distinct from the object rig space.
2. The computer-implemented method of claim 1 , wherein the pose data comprises scan data corresponding to the scan of the part of the actor in the first pose.
3. The computer-implemented method of claim 1 , wherein the object rig comprises a facial rig, the first object rig state represents a neutral pose for the facial rig, and the second object rig state represents a second pose for the facial rig distinct from the neutral pose.
4. The computer-implemented method of claim 1 , wherein the object comprises a face of the actor, and wherein the pose data comprises a facial scan, wherein the object rig is a facial rig.
5. The computer-implemented method of claim 4 , further comprising obtaining scan data from a plurality of poses of the actor and generating a plurality of facial rig states from the plurality of poses.
6. The computer-implemented method of claim 1 , wherein the set of feature values comprises positions of a plurality of vertices of the pose data.
7. The computer-implemented method of claim 1 , wherein the neural network is a convolutional neural network (CNN), the method further comprising training the CNN by:
obtaining a plurality of scan datasets, one scan dataset of which comprises the scan of the part of the actor;
generating a plurality of low-resolution images, wherein each of the plurality of low-resolution images comprises an image of a scan represented in the plurality of scan datasets;
obtaining a two-dimensional map of a face of the actor;
generating a plurality of vertex maps, wherein each of the plurality of vertex maps comprises represents a mapping from a low-resolution image of the plurality of vertex maps to the two-dimensional map; and
applying, in a training process, pairs of CNN inputs, wherein a pair of CNN inputs comprises a vertex map from the plurality of vertex maps and a ground truth input comprising a corresponding scan of the plurality of scan datasets wherein the vertex map is derived from the corresponding scan, whereby the CNN is trained to output an estimate of a scan based on an input vertex map.
8. The computer-implemented method of claim 7 , wherein the input vertex map corresponds to vertices in a two-dimensional space that map to vertices in the corresponding scan.
9. The computer-implemented method of claim 1 , wherein a vector represents a movement of a point on a facial scan in a two-dimensional space.
10. The computer-implemented method of claim 1 , wherein two-dimensional mappings arrays are stored as UV maps in image files.
11. The computer-implemented method of claim 1 , further comprising generating a facial model from a facial scan as the pose data, to be used in generating a facial expression procedurally and independent of scans of the actor.
12. A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor of a computer system, causes the computer system to carry out the method of claim 1 .
13. A computer system comprising:
one or more processors; and
a storage medium storing instructions, which when executed by the one or more processors, cause the computer system to implement the method of claim 1 .
14. A non-transitory carrier medium carrying image data that includes pixel information generated according to the method of claim 1 .