IP Library › Granted Patent US 12,307,616
Granted Patent B2
US 12,307,616 · App. 17/473,232 · Granted May 20, 2025

Techniques for re-aging faces in images and video frames

Inventors: Gaspard Zoss (Zurich, CH); Derek Edward Bradley (Zurich, CH); Prashanth Chandran (Zurich, CH); Paulo Fabiano Urnau Gotardo (Zurich, CH); Eftychios Sifakis (Verona, WI)
Assignees: Disney Enterprises, INC.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
G06T19/20G06N3/08G06T7/149G06T17/20G06T2207/20081G06T2207/20084G06T2207/20212G06T2207/30201G06T2219/2021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,616
App. No.
17/473,232
Granted
May 20, 2025
Kind
B2
Abstract

Techniques are disclosed for re-aging images of faces and three-dimensional (3D) geometry representing faces. In some embodiments, an image of a face, an input age, and a target age, are input into a re-aging model, which outputs a re-aging delta image that can be combined with the input image to generate a re-aged image of the face. In some embodiments, 3D geometry representing a face is re-aged using local 3D re-aging models that each include a blendshape model for finding a linear combination of sample patches from geometries of different facial identities and generating a new shape for the patch at a target age based on the linear combination. In some embodiments, 3D geometry representing a face is re-aged by performing a shape-from-shading technique using re-aged images of the face captured from different viewpoints, which can optionally be constrained to linear combinations of sample patches from local blendshape models.

Claims (29)

1. A computer-implemented method for re-aging a face included in a first image, the method comprising:

generating an input age image corresponding to an input age associated with the face in the first image and a target age image associated with a target age of the face in the first image, wherein the input age image includes a first plurality of pixels having age values indicating the input age and the target age image includes a second plurality of pixels having age values indicating the target age, and wherein the first image, the input age image, and the target age image have the same spatial resolution;

generating, via a machine learning model, a second image based on a multi-channel tensor provided to the machine learning model at execution, wherein the multi-channel tensor comprises (i) the first image that includes the face, (ii) an input age image, and (iii) a target age image, wherein the second image includes one or more differences from the first image; and

combining the first image and the second image into a third image.

2. The computer-implemented method of claim 1 , further comprising:

training the machine learning model based on a first set of images of a plurality of facial identities at a plurality of ages.

3. The computer-implemented method of claim 2 , further comprising generating the first set of images by:

generating, via a first pre-trained machine learning model, a second set of images of the plurality of facial identities; and

generating, via a second pre-trained machine learning model, the first set of images based on the second set of images.

4. The computer-implemented method of claim 2 , wherein training the machine learning model comprises minimizing a loss function that comprises a L1 loss, a perceptual loss, and an adversarial loss.

5. The computer-implemented method of claim 1 , wherein the machine learning model comprises a U-Net architecture.

6. The computer-implemented method of claim 1 , further comprising:

generating a segmented image indicating a plurality of pixels included in the first image that correspond to skin of the face,

wherein the one or more pixels indicating the input age in the fourth image and the one or more pixels indicating the target age in the fifth image correspond to one or more pixels in the plurality of pixels included in the first image.

7. The computer-implemented method of claim 1 , wherein the fifth image further comprises one or more pixels indicating at least one other target age.

8. The computer-implemented method of claim 1 , wherein the target age is injected via a layer modulation technique.

9. The computer-implemented method of claim 1 , further comprising:

determining a deformation of three-dimensional geometry representing the face based on the third image.

10. The computer-implemented method of claim 1 , further comprising training a second machine learning model based on the third image, wherein, subsequent to training, the second machine learning model generates an output image including a given face at a given target age based on an input image of the given face at a given input age.

11. The computer-implemented method of claim 10 , wherein a 3D geometry representing the given face at the given input age is deformed based on the output image to generate a re-aged 3D geometry.

12. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for re-aging a face included in a first image, the steps comprising:

generating an input age image corresponding to an input age associated with the face in the first image and a target age image associated with a target age of the face in the first image, wherein the input age image includes a first plurality of pixels having age values indicating the input age and the target age image includes a second plurality of pixels having age values indicating the target age, and wherein the first image, the input age image, and the target age image have the same spatial resolution;

generating, via a machine learning model, a second image based on a multi-channel tensor provided to the machine learning model at execution, wherein the multi-channel tensor comprises (i) the first image that includes the face, (ii) an input age image, and (iii) a target age image, wherein the second image includes one or more differences from the first image; and

combining the first image and the second image into a third image.

13. The one or more non-transitory computer-readable storage media of claim 12 , wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform steps comprising:

generating a first set of images of a plurality of facial identities at a plurality of ages by:

generating, via a first pre-trained machine learning model, a second set of images of the plurality of facial identities, and

generating, via a second pre-trained machine learning model, the first set of images based on the second set of images; and

training the machine learning model based on the first set of images.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA TO OMISSION THE SECOND ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 057463 FRAME: 0262. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Sep 21, 2021
From: ZOSS, GASPARD; BRADLEY, DEREK EDWARD; CHANDRAN, PRASHANTH; URNAU GOTARDO, PAULO FABIANO; SIFAKIS, EFTYCHIOS
To: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH; ETH ZÜRICH (EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE ZÜRICH)
Reel/Frame 057553/0113 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 16, 2021
From: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
To: DISNEY ENTERPRISES, INC.
Reel/Frame 057524/0193 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 13, 2021
From: ZOSS, GASPARD; BRADLEY, DEREK EDWARD; CHANDRAN, PRASHANTH; URNAU GOTARDO, PAULO FABIANO; SIFAKIS, EFTYCHIOS
To: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
Reel/Frame 057463/0262 →
Continuity (1)
Related Publication 20230080639A1 · Mar 16, 2023
References Cited (83)
US 10504267B2 · Simons · 2019 [cited by examiner]
US 10614623B2 · D'Alessandro · 2020 [cited by examiner]
US 10621771B2 · Matts et al. · 2020 [cited by applicant]
US 20060034542A1 · Aoyama · 2006 [cited by applicant]
US 20080212894A1 · Demirli et al. · 2008 [cited by applicant]
US 20080240489A1 · Marugame · 2008 [cited by applicant]
US 20090028380A1 · Hillebrand et al. · 2009 [cited by applicant]
US 20180018819A1 · Hwang et al. · 2018 [cited by applicant]
US 20180365874A1 · Hadap · 2018 [cited by examiner]
US 20190034704A1 · Qiu · 2019 [cited by examiner]
US 20210407153A1 · Despois · 2021 [cited by examiner]
US 20220084173A1 · Liang · 2022 [cited by examiner]
US 20220172431A1 · Abel · 2022 [cited by examiner]
US 20220198830A1 · Li · 2022 [cited by examiner]
US 20220309633A1 · Davies · 2022 [cited by examiner]
US 20230061875A1 · Zhao · 2023 [cited by examiner]
CN 101556699A · 2009 [cited by applicant]
CN 105787974A · 2016 [cited by applicant]
CN 109523463A · 2019 [cited by applicant]
CN 109584327A · 2019 [cited by applicant]
CN 110322394A · 2019 [cited by examiner]
CN 110348352A · 2019 [cited by applicant]
CN 110852935A · 2020 [cited by applicant]
CN 111652024A · 2020 [cited by applicant]
CN 111798369A · 2020 [cited by applicant]
CN 112002009A · 2020 [cited by applicant]
CN 112883756A · 2021 [cited by applicant]
GB 2602862A · 2022 [cited by applicant]
KR 102266165B1 · 2021 [cited by applicant]
WO 2021171118A1 · 2021 [cited by applicant]
Markus Gross et al. “Production-Ready Face Re-Aging for Visual-Effects”; ACM Trans. Graph., vol. 41, No. 6, Article 237. Publication date: Dec. 2022 (Year: 2022). [cited by examiner]
Wang et al. “U-Net Conditional GANs for Photo-Realistic and Identity-Preserving Facial Expression Synthesis”; ACM Transactions on Multimedia Computing, Communications, and Applications vol. 15 Issue 3s Article No. 88pp.… [cited by examiner]
GB Combined Search and Examination Report for application No. 2213089.2 dated Mar. 16, 2023. [cited by applicant]
GB Combined Search and Examination Report for Application No. 2213089.2 dated Nov. 27, 2023. [cited by applicant]
Alaluf et al., “Only a Matter of Style: Age Transformation Using a Style-Based Regression Model”, http://arxiv.org/abs/2102.02754, ACM Trans. Graph., vol. 40, No. 4, Article 45, https://doi.org/10.1145/3450626.3459805, … [cited by applicant]
Amberg et al., “Weight, Sex, and Facial Expressions: On the Manipulation of Attributes in Generative 3D Face Models”, 2009, pp. 1-11. [cited by applicant]
Zhang et al., “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”, http://arxiv.org/abs/1801.03924, 2018, pp. 1-14. [cited by applicant]
Despois et al., “AgingMapGAN (AMGAN): High-Resolution Controllable Face Aging with Spatially-Aware Conditional GANs”, http://arxiv.org/abs/2008.10960, 2020, pp. 1-16. [cited by applicant]
Yao et al., “High Resolution Face Age Editing”, http://arxiv.org/abs/2005.04410, 2020, pp. 1-26. [cited by applicant]
Or-El et al., “Lifespan Age Transformation Synthesis”, http://arxiv.org/abs/2003.09764, 2020, pp. 1-27. [cited by applicant]
Tazoe et al., “Facial aging simulator considering geometry and patch-tiled texture”, ACM SIGGRAPH 2012 Posters on—SIGGRAPH '12, http://dx.doi.org/10.1145/2342896.2343002, Aug. 5-9, 2012, 1 page. [cited by applicant]
Lin et al., “De-aging High-Resolution 3D Facial Models by Example-Driven Mesh Deformation”, ACM SIGGRAPH 2012 Posters, Association for Computing Machinery, Aug. 5-9, 2012, 1 page. [cited by applicant]
Riaz et al., “Age Progression by Gender-Specific 3D Aging Model”, Machine Vision and Applications, https://doi.org/10.1007/s00138-018-0975-2, 2017, 19 pages. [cited by applicant]
Majid et al., “Adult-child 3D backward face aging model (3D B-Fam)”, J. Vis. Commun., vol. 72, https://doi.org/10.1016/j.jvcir.2020.102803, 2020, pp. 1-11. [cited by applicant]
Pham et al., “Semi-Supervised FaceGAN for Face-Age Progression and Regression with Synthesized Paired Images”, Electronics 2020, vol. 9, No. 603, doi: 10.3390/electronics9040603, 2020, pp. 1-16. [cited by applicant]
Viazovetskyi et al., “StyleGAN2 Distillation for Feed-forward Image Manipulation”, https://arxiv.org/abs/2003.03581, Oct. 22, 2020, pp. 1-18. [cited by applicant]
Digital Human, “Machine Learning for De-Aging”, Retrieved from https://motuslab.org/f/machine-learning-for-de-aging, on Sep. 1, 2021, Jul. 2, 2020, 14 pages. [cited by applicant]
Duong et al., “Automatic Face Aging in Videos via Deep Reinforcement Learning”, https://arxiv.org/pdf/1811.11082.pdf, Apr. 24, 2019, 10 pages. [cited by applicant]
Abdal et al., “StyleFlow: Attribute-Conditioned Exploration of StyleGAN-Generated Images Using Conditional Continuous Normalizing Flows”, ACM Trans. Graph., vol. 40, No. 3, Article 21, https://doi.org/10.1145/3447648, a… [cited by applicant]
Angulu et al., “Age estimation via face images: a survey”, EURASIP Journal on Image and Video Processing, vol. 12, https://doi.org/10.1186/s13640-018-0278-6, 2018, pp. 1-35. [cited by applicant]
Antipov et al., “Face aging with conditional generative adversarial networks”, In 2017 IEEE International Conference on Image Processing (ICIP), https://doi.org/10.1109/ICIP.2017.8296650, 2017, pp. 2089-2093. [cited by applicant]
Duong et al., “Automatic Face Aging in Videos via Deep Reinforcement Learning”, In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1811.11082, 2019, pp. 10005-10014. [cited by applicant]
Fang et al., “Triple-GAN: Progressive Face Aging with Triple Translation Loss”, In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), DOI 10.1109/CVPRW50498.2020.00410, 2020, pp. 3500… [cited by applicant]
Gal et al., “SWAGAN:A Style-based Wavelet-driven Generative Model”, https://arxiv.org/abs/2102.06108, 2021, pp. 1-13. [cited by applicant]
Georgopoulos et al., “Modeling of Facial Aging and Kinship: A Survey”, Image Vis. Comput., vol. 80, Dec. 2018, pp. 58-79. [cited by applicant]
Goodfellow et al., “Generative Adversarial Nets”, Advances in neural information processing systems, vol. 27, arXiv:1406.2661, 2014, pp. 2672-2680. [cited by applicant]
Hsu et al., “Wasserstein Divergence GAN With Cross-Age Identity Expert and Attribute Retainer for Facial Age Transformation”, IEEE Access, vol. 9, https://doi.org/10.1109/ACCESS.2021, 2021, p. 39695-39706. [cited by applicant]
Huang et al., “When Age-Invariant Face Recognition Meets Face Age Synthesis:A Multi-Task Learning Framework”, CVPR, arXiv:2103.01520, 2021, pp. 1-12. [cited by applicant]
Härkönen et al., “GANSpace:Discovering Interpretable GAN Controls”, In Proc. NeurIPS, arXiv:2004.02546, 2020, 29 pages. [cited by applicant]
Karras et al., “Progressive Growing of GANs for Improved Quality, Stability, and Variation”, In International Conference on Learning Representations, 2018, pp. 1-26. [cited by applicant]
Karras et al., “Training Generative Adversarial Networks with Limited Data”, In Proc. NeurIPS, arXiv:2006.06676, 2020, 37 pages. [cited by applicant]
Karras et al., “A Style-Based Generator Architecture for Generative Adversarial Networks”, In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4401-4410. [cited by applicant]
Karras et al., “Analyzing and Improving the Image Quality of StyleGAN”, In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 8110-8119. [cited by applicant]
Kemelmacher-Shlizerman et al., “Illumination-Aware Age Progression”, In 2014 IEEE Conference on Computer Vision and Pattern Recognition, https://doi.org/10.1109/CVPR.2014.426, 2014, pp. 3334-3341. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization”, In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.), http://arxiv.org/ab… [cited by applicant]
Li et al., “Continuous Face Aging via Self-estimated Residual Age Embedding”, arXiv:2105.00020, 2021, pp. 1-14. [cited by applicant]
Liu et al., “A3GAN: An Attribute-Aware Attentive Generative Adversarial Network for Face Aging”, IEEE Transactions on Information Forensics and Security, vol. 16, https://doi.org/10.1109/TIFS.2021.3065499, 2019, pp. 277… [cited by applicant]
Pandey et al., “Total Relighting: Learning to Relight Portraits for Background Replacement”, ACM Transactions on Graphics (Proceedings SIGGRAPH), vol. 40, No. 4, Article 1, https://doi.org/10.1145/3450626.3459872, Aug. … [cited by applicant]
Park et al., “Semantic Image Synthesis with Spatially-Adaptive Normalization”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, arXiv:1903.07291, 2019, pp. 1-19. [cited by applicant]
Ramanathan et al., “Computational Methods for Modeling Facial Aging: A Survey”, J. Vis. Lang. Comput., vol. 20, No. 3, doi:10.1016/j.jvlc.2009.01.011, Jun. 2009, pp. 131-144. [cited by applicant]
Schroff et al., “FaceNet: A Unified Embedding for Face Recognition and Clustering”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815-823. [cited by applicant]
Shen et al., “Interpreting the Latent Space of GANs for Semantic Face Editing”, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1907.10786, 2020, pp. 1-12. [cited by applicant]
Shu et al., “Age progression: Current technologies and applications”, Neurocomputing, vol. 208, https://doi.org/10.1016/j.neucom.2016.01.101, 2016, pp. 249-261. [cited by applicant]
Simonyan et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition”, In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (… [cited by applicant]
Tov et al., “Designing an Encoder for StyleGAN Image Manipulation”, arXiv:2102.02766, 2021, pp. 1-33. [cited by applicant]
Yang et al., “Learning Continuous Face Age Progression: A Pyramid of GANs”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, No. 2, https://doi.org/10.1109/TPAMI.2019.2930985, Feb. 2021, pp. 499-515. [cited by applicant]
Yao et al., “High Resolution Face Age Editing”, arXiv:2005.04410, 2020, pp. 1-26. [cited by applicant]
Yu et al., “BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation”, arXiv:2004.02147, 2020, pp. 1-16. [cited by applicant]
Zhang, Richard, “Making Convolutional Networks Shift-Invariant Again”, In ICML, arXiv:1904.11486, 2019, 17 pages. [cited by applicant]
Zhang et al., “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”, In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1801.03924, 2019, pp. 1-14. [cited by applicant]
Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”, In Computer Vision (ICCV), 2017 IEEE International Conference, arXiv:1703.10593, 2017, 18 pages. [cited by applicant]
Examination Report dated Mar. 19, 2024, for New Zealand Patent Application No. 792263, 4 pages. [cited by applicant]
GB Combined Search and Examination Report for Application No. 2411365.6 dated Dec. 11, 2024. [cited by applicant]