IP Library › Granted Patent US 12,470,746
Granted Patent B2
US 12,470,746 · App. 18/484,099 · Granted Nov 11, 2025

Method and apparatus for talking face video compression

Inventors: Bolin Chen (Hong Kong, HK); Zhao Wang (Beijing, CN); Yan Ye (San Diego, CA); Shiqi Wang (Hong Kong, HK)
Assignee: Alibaba Damo (Hangzhou) Technology Co., Ltd.
H04N19/597H04N19/137H04N19/172
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,470,746
App. No.
18/484,099
Granted
Nov 11, 2025
Kind
B2
Abstract

Methods and apparatuses are provided for processing video data. An exemplary method includes: decompressing a compressed frame to generate a key frame representing a face; generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face; reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to compressed inter-predicted residuals of the second set of parameters; and generating a video comprising the face based on the key frame, the first set of parameters, and the second set of parameters.

Claims (71)

1 . A method for decoding a bitstream, comprising:

receiving a bitstream; and

decoding the bitstream to generate a video, the decoding comprising:

decompressing a compressed frame to generate a key frame representing a face;

generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face;

reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to a set of compressed parameters, the set of compressed parameter being generated by compressing inter-predicted residuals of the second set of parameters; and

generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters;

wherein generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters comprises:

generating a key-frame mesh according to the first set of parameters by a 3D Morphable Model (3DMM);

generating, for each of one or more inter frames, an inter-frame mesh according to the second set of parameters by the 3DMM;

determining, for each of the key frame and the one or more inter frames, a dense motion map and an occlusion map according to the key-frame mesh and the inter-frame mesh; and

generating the video according to the key frame, the dense motion map, and the occlusion map.

2 . The method according to claim 1 , wherein the compressed frame is encoded according to any of following standards: Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC).

3 . The method according to claim 1 , wherein the first and second set of parameters comprise at least one of expression, translation, or angle.

4 . The method according to claim 3 , further comprising:

identifying an identity of the face in the key frame, wherein the first set of parameters further comprise the identity of the face, and the second set of parameters comprise the identity of the face by inheriting from the first set of parameters.

5 . The method according to claim 1 , wherein determining, for each of the key frame and the one or more inter frames, the dense motion map and the occlusion map according to the key-frame mesh and the inter-frame mesh comprises:

obtaining a coarse motion filed of the face according to the key-frame mesh and the inter-frame mesh;

generating a coarse deformed frame by inputting the key-frame and the coarse motion filed to a deep neural network; and

estimating the dense motion map and the occlusion map by concatenating the coarse motion frame, the key frame, and the coarse motion filed to the deep neural network.

6 . The method according to claim 5 , wherein generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters comprises:

warping the key frame according to the dense motion map; and

generating the video by calculating a Hadamard product of the occlusion map and the warping result.

7 . The method according to claim 5 , wherein the video is generated by a generation module, and the generated video comprises a corresponding reconstructed frame for each of one or more inter frames, and

the deep neural network or the generation module is updated with at least one of: a perceptual loss, an adversarial loss, an identity preserving loss, and a reconstruction texture loss between each of the one or more inter frames and its corresponding reconstructed frame.

8 . The method according to claim 1 , wherein the 3DMM is a weakly-supervised multi-face 3D reconstruction (WM3DR) model.

9 . The method according to claim 1 , further comprising:

receiving a bitstream comprising the compressed frame and the inter-predicted residuals for each of the one or more inter frames.

10 . The method according to claim 1 , wherein the video is generated based on dense motion maps and occlusion maps corresponding to the key-frame and the one or more inter frames.

11 . An apparatus for information processing, comprising:

a memory configured to store instructions; and

one or more processors configured to execute the instructions to cause the apparatus to perform operations comprising:

receiving a bitstream; and

decoding the bitstream to generate a video, the decoding comprising:

decompressing a compressed frame to generate a key frame representing a face;

generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face;

reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to a set of compressed parameters, the set of compressed parameter being generated by compressing inter-predicted residuals of the second set of parameters; and

generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters;

wherein generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters comprises:

generating a key-frame mesh according to the first set of parameters by a 3D Morphable Model (3DMM);

generating, for each of one or more inter frames, an inter-frame mesh according to the second set of parameters by the 3DMM;

determining, for each of the key frame and the one or more inter frames, a dense motion map and an occlusion map according to the key-frame mesh and the inter-frame mesh; and

generating the video according to the key frame, the dense motion map, and the occlusion map.

12 . The apparatus according to claim 11 , wherein the compressed frame is encoded according to any of following standards: Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC).

13 . The apparatus according to claim 11 , wherein the first and second set of parameters comprise at least one of expression, translation, or angle.

14 . The apparatus according to claim 13 , wherein the operations further comprise:

identifying an identity of the face in the key frame, wherein the first set of parameters further comprise the identity of the face, and the second set of parameters comprise the identity of the face by inheriting from the first set of parameters.

15 . The apparatus according to claim 11 , wherein determining, for each of the key frame and the one or more inter frames, the dense motion map and the occlusion map according to the key-frame mesh and the inter-frame mesh comprises:

obtaining a coarse motion filed of the face according to the key-frame mesh and the inter-frame mesh;

generating a coarse deformed frame by inputting the key-frame and the coarse motion filed to a deep neural network; and

estimating the dense motion map and the occlusion map by concatenating the coarse motion frame, the key frame, and the coarse motion filed to the deep neural network.

16 . The apparatus according to claim 15 , wherein generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters comprises:

warping the key frame according to the dense motion map; and

generating the video by calculating a Hadamard product of the occlusion map and the warping result.

17 . The apparatus according to claim 15 , wherein the video is generated by a generation module, and the generated video comprises a corresponding reconstructed frame for each of one or more inter frames, and

the deep neural network or the generation module is updated with at least one of: a perceptual loss, an adversarial loss, an identity preserving loss, and a reconstruction texture loss between each of the one or more inter frames and its corresponding reconstructed frame.

18 . The apparatus according to claim 11 , wherein the 3DMM is a weakly-supervised multi-face 3D reconstruction (WM3DR) model.

19 . The apparatus according to claim 11 , wherein the operations further comprise:

receiving a bitstream comprising the compressed frame and the inter-predicted residuals for each of the one or more inter frames.

20 . A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to perform operations comprising:

receiving a bitstream; and

decoding the bitstream to generate a video, the decoding comprising:

decompressing a compressed frame to generate a key frame representing a face;

generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face;

reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to a set of compressed parameters, the set of compressed parameter being generated by compressing inter-predicted residuals of the second set of parameters; and

generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters;

wherein generating the video comprising the face based on the key frame, the first set of parameters, and the second set of parameters comprises:

generating a key-frame mesh according to the first set of parameters by a 3D Morphable Model (3DMM);

generating, for each of one or more inter frames, an inter-frame mesh according to the second set of parameters by the 3DMM;

determining, for each of the key frame and the one or more inter frames, a dense motion map and an occlusion map according to the key-frame mesh and the inter-frame mesh; and

generating the video according to the key frame, the dense motion map, and the occlusion map.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 5, 2026
From: ALIBABA DAMO (HANGZHOU) TECHNOLOGY CO., LTD.
To: ALIBABA INNOVATION PRIVATE LIMITED
Reel/Frame 075529/0583 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 5, 2026
From: ALIBABA INNOVATION PRIVATE LIMITED
To: SIM IP 5 LLC
Reel/Frame 075529/0713 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 16, 2024
From: CHEN, BOLIN; WANG, ZHAO; YE, YAN; WANG, SHIQI
To: ALIBABA DAMO (HANGZHOU) TECHNOLOGY CO., LTD.
Reel/Frame 066137/0985 →
Continuity (2)
Provisional Application 63379779 · Oct 17, 2022
Related Publication 20240146963A1 · May 2, 2024
References Cited (60)
US 20190200046A1 · Lucas · 2019 [cited by examiner]
US 20200234480A1 · Volkov · 2020 [cited by examiner]
US 20220217371A1 · Jiang et al. · 2022 [cited by applicant]
US 20220392133A1 · Volkov · 2022 [cited by examiner]
CN 1384469A · 2002 [cited by examiner]
CN 109271933A · 2019 [cited by applicant]
CN 109417635A · 2019 [cited by examiner]
CN 110415341A · 2019 [cited by applicant]
CN 114363623A · 2022 [cited by applicant]
CN 114882545A · 2022 [cited by examiner]
TW 202301277A · 2023 [cited by examiner]
WO 2013148002A2 · 2013 [cited by applicant]
Agrawal, Vasu translation TW 202301277 A Mar. 30, 2022 (Year: 2022). [cited by examiner]
Zhang, Hui translation CN 114882545 A Mar. 31, 2022 (Year: 2022). [cited by examiner]
Kim, Seong-Jin translation CN 1384469 A Feb. 28, 2002 (Year: 2002). [cited by examiner]
Zhao, Zhi-jie translation CN 109417635 A Jun. 17, 2016 (Year: 2016). [cited by examiner]
Ahmed et al., “Discrete cosine transform,” IEEE transactions on Computers, vol. 100, No. 1, pp. 90-93, 1974. [cited by applicant]
Balle et al., End-to-End Optimized Image Compression, ICLR, 2017, 27 pages. [cited by applicant]
Balle et al., “Density modeling of images using a generalized normalization transformation,” ICLR, 14 pages, 2016. [cited by applicant]
Blanz et al., “A morphable model for the synthesis of 3d faces,” in Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999, pp. 187-194. [cited by applicant]
Bross et al., “Overview of the Versatile Video Coding (VVC) Standard and Its Applications,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, No. 10, pp. 3736-3764, 2023. [cited by applicant]
Chen et al., “Beyond key-point coding: Temporal evolution inference with compact feature representation for talking face video compression,” in Proceedings of the IEEE Data Compression Conference, 10 pages, 2022. [cited by applicant]
Cleary et al., “Data compression using adaptive coding and partial string matching,” IEEE Transactions on Communications, vol. 32, No. 4, pp. 396-402, 1984. [cited by applicant]
Deng et al., “ArcFace: Additive Angular Margin Loss for Deep Face Recognition,” CVPR, 2019, 4690-4699. [cited by applicant]
Ding et al., “Image Quality Assessment: Unifying Structure and Texture Similarity,” IEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, No. 5, May 2022. [cited by applicant]
Doukas et al., ,“HeadGAN: One-shot Neural Head Synthesis and Editing,” Proceedings of 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 14398-14407, 2021. [cited by applicant]
Feng et al., “A generative compression framework for low bandwidth video conference,” in ICME Workshop, 6 pages 2021. [cited by applicant]
Gondal et al., “The unreasonable effectiveness of texture transfer for single image super-resolution,” ECCV, 2018, 17 pages. [cited by applicant]
Goodfellow et al., “Generative adversarial nets,” Advances in neural information processing systems, vol. 27, 9 pages, 2014. [cited by applicant]
Hu et al., “FVC: A New Framework towards Deep Video Compression in Feature Space,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1502-1511. [cited by applicant]
Jia et al., “Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding,” IEEE Transactions on Image Processing, vol. 28, No. 7, pp. 3343-3356 (2019). [cited by applicant]
Kingma et al., “Auto-encoding variational bayes,” in Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014, p. 14. [cited by applicant]
Koufakis et al., “Very low bit rate face video compression using linear combination of 2d face views and principal components analysis,” Image and Vision computing, vol. 17, No. 14, pp. 1031-1051, 1999. [cited by applicant]
Li et al., “Efficient, Multiple-Line-Based Intra Prediction for HEVC, ”IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, No. 4, pp. 947-957, 2016. [cited by applicant]
Lin et al., “M-LVC: multiple frames prediction for learned video compression,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3546-3554. [cited by applicant]
Lopez et al., “Head pose computation for very low bitrate video coding,” in International Conference on Computer Analysis of Images and Patterns. Springer, 1995, pp. 440-447. [cited by applicant]
Lu et al. “DVC: An end-to-end deep video compression framework,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11006-11015, 2019. [cited by applicant]
Lu et al., “Content adaptive and error propagation aware deep video compression,” European Conference on Computer Vision. Springer, 2020, pp. 456-472. [cited by applicant]
Ma et al. “Convolutional neural network-based arithmetic coding for hevc intra-predicted residues,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, No. 7, pp. 1901-1916, 2019. [cited by applicant]
Oquab et al., “Low bandwidth video-chat compression using deep generative models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2388-2397. [cited by applicant]
Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention, 8 pages, 2015. [cited by applicant]
Siarohin et al., “First Order Motion Model for Image Animation,” Advances in Neural Information Processing Systems, vol. 32, pp. 7137-7147, 2019. [cited by applicant]
Sullivan et al., “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEE Transactions on Circuits and Systems for Video Technology, vol. 22, No. 12, pp. 1649-1668 (2012). [cited by applicant]
Teuhola, Jukka, “A Compression Method for Clustered Bit-Vectors,” Information Processing Letters, vol. 7, No. 6, pp. 308-311, 1978. [cited by applicant]
Torres et al., “A proposal for high compression of faces in video sequences using adaptive eigenspaces,” in International Conference on image Processing. IEEE, 2002, vol. 1. [cited by applicant]
Tripathy et al., “FACEGAN: Facial Attribute Controllable rEnactment GAN,” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 1329-1338. [cited by applicant]
Wang et al., “One-shot free-view neural talking-head synthesis for video conferencing,” in Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, 2021, pp. 10039-10049. [cited by applicant]
Wang et al., “Few-shot video-to-video synthesis.,” in NeurIPS, 2019, 14 pages. [cited by applicant]
Wiegand et al., “Overview of the H.264/AVC Video Coding Standard,” IEEE Transactions on Circuitds and Systems for Video Technology, vol. 13, No. 7, Jul. 2003, pp. 560-576. [cited by applicant]
Wiles et al., “X2face: A network for controlling face generation using images, audio, and pose codes,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 670-686. [cited by applicant]
Yang et al., “Learning for video compression with hierarchical quality and recurrent enhancement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6628-6637. [cited by applicant]
Yang et al., “Learning for video compression with recurrent auto-encoder and recurrent probability model,” IEEE Journal of Selected Topics in Signal Processing, vol. 15, No. 2, pp. 388-401, 2020. [cited by applicant]
Yang et al., “Face2Face : Real-Time High-Resolution One-Shot Face Reenactment,” Proceedings of 2022 European Conference on Computer Vision, 17 pages. [cited by applicant]
Yao et al., “Mesh Guided One-shot Face Reenactment Using Graph Convolutional Networks.” In Proceedings of the 28th ACM International Conference on Multimedia, 17 pages, 2020. [cited by applicant]
Zakharov et al., “Few-shot adversarial learning of realistic neural talking head models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019,pp. 9459-9468. [cited by applicant]
Zhang et al., “Weakly-Supervised Multi-Face 3D Reconstruction,” arXiv: Computer Vision and Pattern Recognition, 2021, 9 pages. [cited by applicant]
Zhao et al., Enhanced Bi-Prediction with convolutional neural network for high-efficiency video coding, IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, No. 11, pp. 3291-3301, 2018. [cited by applicant]
Zhu et al., “Generative adversarial network-based intra prediction for video coding,” IEEE transactions on multimedia, vol. 22, No. 1, pp. 45-58, 2019. [cited by applicant]
PCT International Search Report and Written Opinion mailed Dec. 20, 2023, issued in corresponding International Application No. PCT/CN2023/124879 (7 pages). [cited by applicant]
Li et al., “Detail 3D Face Reconstruction Based on 3DMM and Displacement Map, ”Journal of Sensors, 13 pages, 2021. [cited by applicant]