Remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice

1. A system for remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice, the system comprising:

a sensor device that (i) captures one or more frames of a face of a person, each frame comprising one or more color images of the person's face, one or more depth maps of the person's face, voice stream data associated with the person, and a timestamp, and (ii) generates a 3D face model of the person using the one or more depth maps;

a computing device coupled to the sensor device, the computing device comprising a memory that stores computer-executable instructions and a processor that executes the instructions to:

receive the one or more frames of the person's face and the 3D face model from the sensor device;

preprocess the 3D face model;

for each received frame:

detect facial landmarks using the one or more color images;

match the 3D face model to the one or more depth maps using non-rigid registration;

update a texture on a portion of the 3D face model corresponding to the person's face using the one or more color images, wherein the texture comprises a cropped portion of one or more of the color images, comprising:

keeping the texture constant on remaining portions of the 3D face model, and

after updating the texture on the portion of the 3D face model corresponding to the person's face, blending a portion of the texture on the portion of the 3D face model corresponding to the person's face to match the texture on adjacent portions of the 3D face model;

synchronize the 3D face model with a segment of the voice stream data using the timestamp; and

transmit the synchronized 3D face model and voice stream data to a remote device for display.

2. The system of claim 1 , wherein the 3D face model comprises one or more of: a face of the person, a chest of the person, one or more shoulders of the person, and a back of a head of the person.

3. The system of claim 2 , wherein preprocessing the 3D face model comprises:

loading the 3D face model into memory; and

separating a part of the 3D face model corresponding to the face of the person from one or more other parts of the 3D face model.

4. The system of claim 1 , wherein the computing device preprocesses the 3D face model once at a beginning of a streaming session.

5. The system of claim 1 , wherein detecting facial landmarks using the one or more color images comprises executing a pre-trained neural network model on the one or more color images to detect the facial landmarks.

6. The system of claim 1 , wherein synchronizing the 3D face model with a segment of the voice stream data using the timestamp comprises:

determining that the timestamp associated with the frame is delayed based upon a current time; and

discarding one or more of (i) the segment of the voice segment data or (ii) the frame.

7. The system of claim 1 , wherein detecting facial landmarks using the one or more color images comprises:

generating a control point animation map for the 3D face model using the detected facial landmarks by locating the detected facial landmarks in 3D space using the 3D face model, including triangulating each of the detected facial landmarks using a triangulation function, and extracting a 3D deformation for each point in the 3D face model using the located facial landmarks; and

mapping the control point animation map to the 3D face model to generate a mapped 3D face model.

8. A computerized method of remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice, the method comprising:

capturing, by a sensor device, one or more frames of a face of a person, each frame comprising one or more color images of the person's face, one or more depth maps of the person's face, voice stream data associated with the person, and a timestamp;

generating, by the sensor device, a 3D face model of the person using the one or more depth maps;

receiving, by a computing device, the one or more frames of the person's face and the 3D face model from the sensor device;

preprocessing, by the computing device, the 3D face model;

for each received frame:

detecting, by the computing device, facial landmarks using the one or more color images;

matching, by the computing device, the 3D face model to the one or more depth maps using non-rigid registration;

updating, by the computing device, a texture on a portion of the 3D face model corresponding to the person's face using the one or more color images, wherein the texture comprises a cropped portion of one or more of the color images, comprising:

keeping the texture constant on remaining portions of the 3D face model, and

after updating the texture on the portion of the 3D face model corresponding to the person's face, blending a portion of the texture on the portion of the 3D face model corresponding to the person's face to match the texture on adjacent portions of the 3D face model;

synchronizing, by the computing device, the 3D face model with a segment of the voice stream data using the timestamp; and

transmitting, by the computing device, the synchronized 3D face model and voice stream data to a remote device for display.

9. The method of claim 8 , wherein the 3D face model comprises one or more of: a face of the person, a chest of the person, one or more shoulders of the person, and a back of a head of the person.

10. The method of claim 9 , wherein preprocessing the 3D face model comprises:

loading the 3D face model into memory; and

separating a part of the 3D face model corresponding to the face of the person from one or more other parts of the 3D face model.

11. The method of claim 8 , wherein the computing device preprocesses the 3D face model once at a beginning of a streaming session.

12. The method of claim 8 , wherein detecting facial landmarks using the one or more color images comprises executing a pre-trained neural network model on the one or more color images to detect the facial landmarks.

13. The method of claim 8 , wherein synchronizing the 3D face model with a segment of the voice stream data using the timestamp comprises:

determining that the timestamp associated with the frame is delayed based upon a current time; and

discarding one or more of (i) the segment of the voice segment data or (ii) the frame.

14. The method of claim 8 , wherein detecting facial landmarks using the one or more color images comprises:

generating a control point animation map for the 3D face model using the detected facial landmarks by locating the detected facial landmarks in 3D space using the 3D face model, including triangulating each of the detected facial landmarks using a triangulation function, and extracting a 3D deformation for each point in the 3D face model using the located facial landmarks; and

mapping the control point animation map to the 3D face model to generate a mapped 3D face model.

15. A system for remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice, the system comprising:

a sensor device that (i) captures one or more frames of a face of a person, each frame comprising one or more color images of the person's face, one or more depth maps of the person's face, voice stream data associated with the person, and a timestamp, and (ii) generates a 3D face model of the person using the one or more depth maps;

a computing device coupled to the sensor device, the computing device comprising a memory that stores computer-executable instructions and a processor that executes the instructions to:

receive the one or more frames of the person's face and the 3D face model from the sensor device;

preprocess the 3D face model;

for each received frame:

detect facial landmarks using the one or more color images, comprising:

generating a control point animation map for the 3D face model using the detected facial landmarks by locating the detected facial landmarks in 3D space using the 3D face model, including triangulating each of the detected facial landmarks using a triangulation function, and extracting a 3D deformation for each point in the 3D face model using the located facial landmarks, and

mapping the control point animation map to the 3D face model to generate a mapped 3D face model;

match the 3D face model to the one or more depth maps using non-rigid registration;

update a texture on a portion of the 3D face model corresponding to the person's face using the one or more color images, wherein the texture comprises a cropped portion of one or more of the color images;

synchronize the 3D face model with a segment of the voice stream data using the timestamp; and

transmit the synchronized 3D face model and voice stream data to a remote device for display.

16. A computerized method of remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice, the method comprising:

capturing, by a sensor device, one or more frames of a face of a person, each frame comprising one or more color images of the person's face, one or more depth maps of the person's face, voice stream data associated with the person, and a timestamp;

generating, by the sensor device, a 3D face model of the person using the one or more depth maps;

receiving, by a computing device, the one or more frames of the person's face and the 3D face model from the sensor device;

preprocessing, by the computing device, the 3D face model;

for each received frame:

detecting, by the computing device, facial landmarks using the one or more color images, comprising:

generating a control point animation map for the 3D face model using the detected facial landmarks by locating the detected facial landmarks in 3D space using the 3D face model, including triangulating each of the detected facial landmarks using a triangulation function, and extracting a 3D deformation for each point in the 3D face model using the located facial landmarks, and

mapping the control point animation map to the 3D face model to generate a mapped 3D face model;

matching, by the computing device, the 3D face model to the one or more depth maps using non-rigid registration;

updating, by the computing device, a texture on a portion of the 3D face model corresponding to the person's face using the one or more color images, wherein the texture comprises a cropped portion of one or more of the color images;

synchronizing, by the computing device, the 3D face model with a segment of the voice stream data using the timestamp; and

transmitting, by the computing device, the synchronized 3D face model and voice stream data to a remote device for display.