THREE-DIMENSIONAL FACE ANIMATION FROM SPEECH
A method for training a three-dimensional model face animation model from speech, is provided. The method includes determining a first correlation value for a facial feature based on an audio waveform from a first subject, generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value, updating the first correlation value when a difference between the first mesh and a ground truth image of the first subject is greater than a pre-selected threshold, and providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first subject. A non-transitory, computer-readable medium storing instructions to cause a system to perform the above method, and the system, are also provided.
1 . (canceled)
2 . A computer-implemented method, comprising:
identifying, based at least in part upon an audio capture of a voice of a subject, an audio-correlated facial feature;
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject;
generating a three-dimensional model of the face of the subject, wherein the three-dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature; and
providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.
3 . The computer-implemented method of claim 2 , wherein identifying the audio-correlated facial feature comprises correlating the audio capture with a geometry of a lower portion of the face of the subject.
4 . The computer-implemented method of claim 2 , wherein identifying the expression-like facial feature of the subject comprises correlating a facial feature with a speech feature from the audio capture of the subject.
5 . The computer-implemented method of claim 2 , wherein identifying the expression-like facial feature of the subject comprises selecting the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject.
6 . The computer-implemented method of claim 2 , wherein identifying the expression-like facial feature of the subject comprises using a sampling of multiple facial expressions collected during a training session of a second subject speaking words.
7 . The computer-implemented method of claim 2 , wherein the expression-like facial feature is associated with an upper facial feature of the subject.
8 . The computer-implemented method of claim 7 , wherein the upper facial feature is related to an eyebrow of the subject or an eye of the subject.
9 . The computer-implemented method of claim 2 , wherein the audio-correlated facial feature is associated with a shape of a lip of the subject.
10 . The computer-implemented method of claim 2 , wherein the three-dimensional model of the face of the subject is based at least in part upon a mesh, wherein the mesh comprises a lower portion of the face of the subject and an upper portion of the face of the subject.
11 . The computer-implemented method of claim 10 , wherein the lower portion of the face of the subject in the mesh is based at least in part upon the audio-correlated facial feature.
12 . The computer-implemented method of claim 10 , wherein the upper portion of the face of the subject in the mesh is based at least in part upon the expression-like facial feature of the subject.
13 . The computer-implemented method of claim 2 , wherein the three-dimensional model of the face of the subject is based at least in part upon information associated with the face of the subject with a neutral expression.
14 . A system, comprising:
one or more processors; and
a memory storing instructions which, when executed by the one or more processors, cause the system to:
identify, based at least in part upon an audio capture of a voice of a subject, an audio-correlated facial feature;
identify, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject;
generate a three-dimensional model of the face of the subject, wherein the three-dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature; and
provide the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.
15 . The system of claim 14 , wherein the one or more processors further execute instructions to correlate the audio capture with a geometry of a lower portion of the face of the subject.
16 . The system of claim 14 , wherein the one or more processors further execute instructions to correlate a facial feature with a speech feature from the audio capture of the subject.
17 . The system of claim 14 , wherein the one or more processors further execute instructions to select the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject.
18 . A non-transitory computer-readable media storing computer-readable instructions that, when executed by at least one processor, cause the at least one processor to execute operations comprising:
identifying, based at least in part upon an audio capture of a voice of a subject, an audio-correlated facial feature;
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject;
generating a three-dimensional model of the face of the subject, wherein the model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature; and
providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.
19 . The non-transitory computer-readable media storing computer-readable instructions of claim 18 that, when executed by the at least one processor, cause the processor to execute operations further comprising:
correlating the audio capture with a geometry of a lower portion of the face of the subject.
20 . The non-transitory computer-readable media storing computer-readable instructions of claim 18 that, when executed by the at least one processor, cause the processor to execute operations further comprising:
correlating a facial feature with a speech feature from the audio capture of the subject.
21 . The non-transitory computer-readable media storing computer-readable instructions of claim 18 that, when executed by the at least one processor, cause the processor to execute operations further comprising:
selecting the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject.