Creating images, meshes, and talking animations from mouth shape data
Creating images and animations of lip motion from mouth shape data includes providing, as one or more input features to a neural network model, a vector of a plurality of coefficients. Each vector of the plurality of coefficients corresponds to a different mouth shape. Using the neural network model, a data structure output specifying a visual representation of a mouth including lips having a shape corresponding to the vector is generated.
1. A computer-implemented method, comprising:
providing, as an input feature to a neural network model, a vector of a plurality of coefficients, wherein each vector of the plurality of coefficients corresponds to a different mouth shape;
providing, as a further input feature to the neural network model, head pose information; and
generating, using the neural network model operating only on 3-dimensional data consisting of the vector of the plurality of coefficients and the head pose information, a data structure output specifying a visual representation of a mouth including lips having a shape corresponding to the vector.
2. The computer-implemented method of claim 1 , wherein the plurality of coefficients of each vector are viseme coefficients.
3. The computer-implemented method of claim 2 , further comprising:
generating the viseme coefficients from audio data of speech.
4. The computer-implemented method of claim 2 , further comprising:
encoding the viseme coefficients into the vector provided to the neural network model.
5. The computer-implemented method of claim 1 , wherein the data structure output is generated without using any facial landmark data.
6. The computer-implemented method of claim 1 , wherein the data structure output is a Red-Green-Blue (RGB) 2-dimensional image.
7. The computer-implemented method of claim 1 , wherein the data structure output is a mesh.
8. The computer-implemented method of claim 1 , wherein the generating the data structure output includes:
generating, using the neural network model, a plurality of images, wherein each image is generated from a different vector having a plurality of coefficients, and wherein the plurality of images collectively represent an animation of lip motion.
9. The computer-implemented method of claim 1 , further comprising:
providing to the neural network model, with the vector of the plurality of coefficients, a first mesh of a face;
wherein the data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face, wherein the second mesh is a version of the first mesh of the face that specifies the mouth including the lips having the shape corresponding to the vector.
10. A system, comprising:
one or more processors configured to initiate operations including:
providing, as an input feature to a neural network model, a vector of a plurality of coefficients, wherein each vector of the plurality of coefficients corresponds to a different mouth shape;
providing, as a further input feature to the neural network model, head pose information; and
generating, using the neural network model operating only on 3-dimensional data consisting of the vector of the plurality of coefficients and the head pose information, a data structure output specifying a visual representation of a mouth including lips having a shape corresponding to the vector.
11. The system of claim 10 , wherein the plurality of coefficients of each vector are viseme coefficients.
12. The system of claim 11 , wherein the one or more processors are configured to initiate operations including:
generating the viseme coefficients from audio data of speech.
13. The system of claim 11 , wherein the one or more processors are configured to initiate operations including:
encoding the viseme coefficients into the vector provided to the neural network model.
14. The system of claim 10 , wherein the data structure output is generated without using any facial landmark data.
15. The system of claim 10 , wherein the data structure output is a Red-Green-Blue (RGB) 2-dimensional image.
16. The system of claim 10 , wherein the data structure output is a mesh.
17. The system of claim 10 , wherein the generating the data structure output includes:
generating, using the neural network model, a plurality of images, wherein each image is generated from a different vector having a plurality of coefficients, and wherein the plurality of images collectively represent an animation of lip motion.
18. The system of claim 10 , wherein the one or more processors are configured to initiate operations including:
providing to the neural network model, with the vector of the one or more coefficients, a first mesh of a face;
wherein the data structure output is a second mesh generated by the neural network model based on the vector and the first mesh of the face, wherein the second mesh is a version of the first mesh of the face that specifies the mouth including the lips having the shape corresponding to the vector.
19. A computer program product, comprising:
a computer readable storage medium, and program instructions collectively stored on the computer readable storage medium, wherein the program instructions are executable by one or more processors to initiate operations including:
providing, as an input feature to a neural network model, a vector of a plurality of coefficients, wherein each vector of the plurality of coefficients corresponds to a different mouth shape;
providing, as a further input feature to the neural network model, a first mesh of a head and face having a mouth region of the first mesh masked so as to specify no mouth or lip shape information;
wherein the vector of the plurality of coefficients and the first mesh information specify 3-dimensional data; and
generating, using the neural network model operating only on the 3-dimensional data, a data structure output specifying a second mesh adapted from the first mesh, wherein the second mesh includes a mouth region including lips having a shape corresponding to the vector.
20. The computer program product of claim 19 , wherein the plurality of coefficients of each vector are viseme coefficients.