IP Library Granted Patent US 11,393,149
Granted Patent B2
US 11,393,149 · App. 17/035,550 · Granted Jul 19, 2022

Generating an animation rig for use in animating a computer-generated character based on facial scans of an actor and a muscle model

Inventors: Byung Kuk Choi (Wellington, NZ); Gergely Klár (Wellington, NZ)
Assignee: Unity Technologies SF
G06T13/40G06T17/20G06V40/166
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,393,149
App. No.
17/035,550
Granted
Jul 19, 2022
Kind
B2
Abstract

An animation system wherein scanned facial expressions are processed to form muscle models that can be used to generate expressions based on specification of a strain vector and a control vector of the muscle model.

Claims (41)

1. A computer-implemented method for generating a facial expression in an animation system, the method comprising:

receiving data associated with a plurality of facial scans of a face of an actor over a plurality of facial expression poses;

training a system to match the plurality of facial scans to a muscle model that could, at least approximately, express some of the plurality of facial expression poses;

extracting from the facial expression poses, a representation of physical deformations of a facial surface of the face;

determining, from the representation of physical deformations of the facial surface and the muscle model, a strain vector of numerical strain vector values representing strains of a set of facial muscles of the muscle model based on current lengths and rest lengths corresponding to the set of facial muscles,

wherein the numerical strain vector values evolve to different values as the face of the actor changes from a first facial expression pose to a second facial expression pose;

generating a mesh representing the facial surface of the face that is formed into expressions based on the muscle model and the numerical strain vector values and their corresponding deformations; and

determining a set of updated numerical strain vector values for the strain vector that adjust the mesh to form an animated facial expression that corresponds to a plausible facial expression from the actor.

2. The method of claim 1 , wherein the plurality of facial scans comprise data pertaining to the physical surface deformations, wherein the deformations are associated with the facial expression poses.

3. The method of claim 1 , wherein the actor is a human being.

4. The method of claim 1 , wherein determining the control vectors further comprises determining strain values associated with expansion and contraction of muscles of the set of facial muscles.

5. The method of claim 1 further comprising generating an animated image using a facial expression derived from a blending of strain vector values.

6. A non-transitory storage medium carrying image data that includes pixel data or mesh data generated according to the method of claim 1 .

7. The method of claim 1 , wherein the numerical strain vector values contains a first numerical strain value corresponding to a first facial muscle from the set of facial muscles, and the first numerical strain value is computed as a difference between a first current length of the first facial muscle and a rest length of the first facial muscle divided by the rest length.

8. An animation system comprising:

at least one processor; and

medium storing instructions, which when executed by the at least one processor, cause the animation system to:

receive data associated with a plurality of facial scans of a face of an actor over a plurality of facial expression poses;

train a system to match the plurality of facial scans to a muscle model that could, at least approximately, express some of the plurality of facial expression poses;

extract, from the facial expression poses, a representation of physical deformations of a facial surface of the face;

determine, from the representation of physical deformations of the facial surface and the muscle model, a strain vector of numerical strain vector values representing strains of a set of facial muscles of the muscle model based on current lengths and rest lengths corresponding to the set of facial muscles,

wherein the numerical strain vector values evolve to different values as the face of the actor changes from a first facial expression pose to a second facial expression pose; generate a mesh representing the facial surface of the face that is formed into expressions based on the muscle model and the numerical strain vector values and their corresponding deformations; and

determine a set of updated numerical strain vector values for the strain vector that adjust the mesh to form an animated facial expression that corresponds to a plausible facial expression from the actor.

9. The animation system of claim 8 , wherein the plurality of facial scans comprise data pertaining to the physical surface deformations, wherein the deformations are associated with the facial expression poses.

10. The animation system of claim 8 , wherein the actor is a human being.

11. The animation system of claim 8 , the control vectors are strain values associated with expansion and contraction of muscles of the set of facial muscles.

12. The animation system of claim 8 , wherein an animated image is generated using a facial expression derived from a blending of strain vector values.

13. The system of claim 8 , wherein the numerical strain vector values contains a first numerical strain value corresponding to a first facial muscle from the set of facial muscles, and the first numerical strain value is computed as a difference between a first current length of the first facial muscle and a rest length of the first facial muscle divided by the rest length.

14. A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor of a computer system, causes the computer system to:

receive data associated with a plurality of facial scans of a face of an actor over a plurality of facial expression poses;

train a system to match the plurality of facial scans to a muscle model that could, at least approximately, express some of the plurality of facial expression poses;

extract, from the facial expression poses, a representation of physical deformations of a facial surface of the face;

determine, from the representation of physical deformations of the facial surface and the muscle model, a strain vector of numerical strain vector values representing strains of a set of facial muscles of the muscle model based on current lengths and rest lengths corresponding to the set of facial muscles,

wherein the numerical strain vector values evolve to different values as the face of the actor changes from a first facial expression pose to a second facial expression pose; generate a mesh representing the facial surface of the face that is formed into expressions based on the muscle model and the numerical strain vector values and their corresponding deformations; and

determine a set of updated numerical strain vector values for the strain vector that adjust the mesh to form an animated facial expression that corresponds to a plausible facial expression from the actor.

15. The non-transitory computer-readable storage medium of claim 14 , further storing instructions, which when executed by at least one processor of a computer system, causes the computer system to process the plurality of facial scans as data pertaining to the physical surface deformations, wherein the deformations are associated with the facial expression poses.

16. The non-transitory computer-readable storage medium of claim 14 , further storing instructions, which when executed by at least one processor of a computer system, causes the computer system to process the plurality of facial scans based on an anatomical model that corresponds to human anatomy.

17. The non-transitory computer-readable storage medium of claim 16 , further storing instructions, which when executed by at least one processor of a computer system, causes the computer system to process the animated facial expression using an alternative anatomical model that is distinct from an anatomy of the actor.

18. The non-transitory computer-readable storage medium of claim 16 , further storing instructions, which when executed by at least one processor of a computer system, causes the computer system to process the control vectors by determining strain values associated with expansion and contraction of muscles of the set of facial muscles.

19. The non-transitory computer-readable storage medium of claim 16 , further storing instructions, which when executed by at least one processor of a computer system, causes the computer system to process a facial expression derived from a blending of strain vector values.

20. The non-transitory computer-readable storage medium of claim 14 , wherein the numerical strain vector values contains a first numerical strain value corresponding to a first facial muscle from the set of facial muscles, and the first numerical strain value is computed as a difference between a first current length of the first facial muscle and a rest length of the first facial muscle divided by the rest length.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 14, 2022
From: CHOI, BYUNG KUK; KLÁR, GERGELY
To: WETA DIGITAL LIMITED
Reel/Frame 059260/0699 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2022
From: UNITY SOFTWARE INC.
To: UNITY TECHNOLOGIES SF
Reel/Frame 058980/0369 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 8, 2022
From: WETA DIGITAL LIMITED
To: UNITY SOFTWARE INC.
Reel/Frame 058978/0905 →
Continuity (2)
Provisional Application 63047847 · Jul 2, 2020
Related Publication 20220005248A1 · Jan 6, 2022