IP Library Granted Patent US 11,270,487
Granted Patent B1
US 11,270,487 · App. 16/574,014 · Granted Mar 8, 2022

Systems and methods for improving animation of computer-generated avatars

Inventors: William Arthur Hugh Steptoe (London, GB); Michael Andrew Howard (Redwood City, CA); Melinda Ozel (San Francisco, CA); Giovanni F. Nakpil (San Francisco, CA); Timothy Naylor (Menlo Park, CA)
Assignee: Facebook Technologies, LLC
G06T13/40G06K9/00221G06K9/00362G06T7/20G06T7/97G06T13/20G06T13/80G06T2207/20081G06T2207/30196
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,270,487
App. No.
16/574,014
Granted
Mar 8, 2022
Kind
B1
Abstract

The disclosed computer-implemented method may include identifying a muscle group engaged by a user to execute a predefined body action by (1) capturing a set of images of the user while the user executes the predefined body action, and (2) associating a feature of a body of the user with the muscle group based on the predefined body action and the set of images. The method may also include determining, based on the set of images, a set of parameters associated with the user, the muscle group, and the predefined body action. The method may also include directing a computer-generated avatar that represents the body of the user to produce the predefined body action in accordance with the set of parameters. Various other methods, systems, and computer-readable media are also disclosed.

Claims (91)

1. A computer-implemented method comprising:

identifying a muscle group engaged by a user to execute a predefined body action by:

capturing a set of images of the user while the user executes the predefined body action; and

associating a feature of a body of the user with the muscle group based on the predefined body action and the set of images;

training an artificial neural network to determine parameters associated with the user, the muscle group, and the predefined body action, the parameters comprising:

an onset curve that describes a rate of change over time of an intensity level of the muscle group as the user executes the predefined body action; and

a falloff curve that describes a rate of change over time of an intensity level of the muscle group subsequent to the user executing the predefined body action;

determining, by providing the set of images as input to the trained artificial neural network, a set of parameters comprising:

an onset curve corresponding to the user, the muscle group, and the predefined body action; and

a falloff curve corresponding to the user, the muscle group, and the predefined body action; and

directing a computer-generated avatar that represents the body of the user to produce the predefined body action in accordance with the set of parameters.

2. The computer-implemented method of claim 1 , wherein:

the set of parameters further comprises:

a target intensity level associated with the predefined body action; and

a source intensity level associated with a previous predefined body action;

the onset curve describes a rate of change over time of the intensity level of the muscle group from the source intensity level to the target intensity level; and

the falloff curve describes a rate of change over time of the intensity level of the muscle group from the target intensity level to an additional intensity level.

3. The computer-implemented method of claim 1 , further comprising:

identifying an additional set of muscle groups, each muscle group in the additional set of muscle groups engaged by the user to produce an additional predefined body action;

for each muscle group in the additional set of muscle groups, determining:

an additional onset curve that describes a change of the muscle group over time to produce the additional predefined body action; and

an additional falloff curve that describes a change of the muscle group over time subsequent to producing the additional predefined body action;

detecting the user executing the additional predefined body action subsequent to the user executing the predefined body action; and

transitioning the computer-generated avatar from executing the predefined body action to executing the additional predefined body action in accordance with the falloff curve and the additional onset curve in response to detecting the user executing the additional predefined body action subsequent to the user executing the predefined body action.

4. The computer-implemented method of claim 3 , wherein:

the set of parameters further comprises a peak transition velocity of the muscle group; and

transitioning the computer-generated avatar from executing the predefined body action to producing the additional predefined body action comprises transitioning the computer-generated avatar from executing the predefined body action to producing the additional predefined body action such that a transition velocity of a portion of the computer-generated avatar that corresponds to the muscle group remains below the peak transition velocity of the muscle group during the transition.

5. The computer-implemented method of claim 1 , wherein:

the method further comprises detecting that the user has executed the predefined body action; and

directing the computer-generated avatar to execute the predefined body action in accordance with the set of parameters comprises directing the computer-generated avatar to execute the predefined body action in response to detecting that the user has executed the predefined body action.

6. The computer-implemented method of claim 1 , further comprising generating a digital signature of at least a portion of the body of the user based on the set of parameters associated with the muscle group and the predefined body action.

7. The computer-implemented method of claim 6 , wherein directing the computer-generated avatar that represents the user to execute the predefined body action in accordance with the set of parameters associated with the muscle group comprises directing the computer-generated avatar that represents the user to execute the predefined body action in accordance with the digital signature of the portion of the body of the user.

8. The computer-implemented method of claim 1 , wherein the computer-generated avatar that represents the user comprises at least one of:

an augmented reality avatar presented within an augmented reality environment;

a virtual reality avatar presented within a virtual reality environment; or

a video conferencing avatar presented within a video conferencing application.

9. The computer-implemented method of claim 1 , wherein associating the feature of the body of the user with the muscle group based on the predefined body action and the set of images comprises analyzing the set of images in accordance with a convolutional neural network pre-trained to associate muscle groups with features of users' bodies based on images of users executing the predefined body action.

10. The computer-implemented method of claim 1 , wherein:

the method further comprises training the artificial neural network to associate muscle groups with features of users' bodies based on images of users executing the predefined body action; and

identifying the muscle group further comprises providing the artificial neural network with the set of images of the user.

11. A system comprising:

an identifying module, stored in memory, that identifies a muscle group engaged by a user to execute a predefined body action by:

capturing a set of images of the user while the user executes the predefined body action; and

associating a feature of a body of the user with the muscle group based on the predefined body action and the set of images;

a determining module, stored in memory, that:

trains an artificial neural network to determine parameters associated with the user, the muscle group, and the predefined body action, the parameters comprising:

an onset curve that describes a rate of change over time of an intensity level of the muscle group as the user executes the predefined body action; and

a falloff curve that describes a rate of change over time of an intensity level of the muscle group subsequent to the user executing the predefined body action; and

determines, by providing the set of images as input to the trained artificial neural network, a set of parameters comprising:

an onset curve corresponding to the user, the muscle group, and the predefined body action; and

a falloff curve corresponding to the user, the muscle group, and the predefined body action;

a directing module, stored in memory, that directs a computer-generated avatar that represents the body of the user to produce the predefined body action in accordance with the set of parameters; and

at least one physical processor that executes the identifying module, the determining module, and the directing module.

12. The system of claim 11 , wherein:

the set of parameters further comprises:

a target intensity level associated with the predefined body action; and

a source intensity level associated with a previous predefined body action;

the onset curve describes a rate of change over time of the intensity level of the muscle group from the source intensity level to the target intensity level; and

the falloff curve describes a rate of change over time of the intensity level of the muscle group from the target intensity level to an additional intensity level.

13. The system of claim 11 , wherein:

the identifying module further identifies an additional set of muscle groups, each muscle group in the additional set of muscle groups engaged by the user to produce an additional predefined body action;

for each muscle group in the additional set of muscle groups, the determining module further determines:

an additional onset curve that describes a change of the muscle group over time to produce the additional predefined body action; and

an additional falloff curve that describes a change of the muscle group over time subsequent to producing the additional predefined body action;

the directing module further:

detects the user executing the additional predefined body action subsequent to the user executing the predefined body action; and

transitions the computer-generated avatar from executing the predefined body action to executing the additional predefined body action in accordance with the falloff curve and the additional onset curve in response to detecting the user executing the additional predefined body action subsequent to the user executing the predefined body action.

14. The system of claim 13 , wherein:

the set of parameters further comprises a peak transition velocity of the muscle group; and

the directing module transitions the computer-generated avatar from executing the predefined body action to producing the additional predefined body action by transitioning the computer-generated avatar from executing the predefined body action to producing the additional predefined body action such that a transition velocity of a portion of the computer-generated avatar that corresponds to the muscle group remains below the peak transition velocity of the muscle group during the transition.

15. The system of claim 11 , wherein:

the system further comprises a detecting module, stored in memory, that detects that the user has executed the predefined body action;

the physical processor further executes the detecting module; and

the directing module directs the computer-generated avatar to execute the predefined body action in accordance with the set of parameters by directing the computer-generated avatar to execute the predefined body action in response to the detecting module detecting that the user has executed the predefined body action.

16. The system of claim 11 , wherein the determining module further determines the set of parameters associated with the user, the muscle group, and the predefined body action by generating a digital signature of at least a portion of the body of the user based on the set of parameters associated with the muscle group and the predefined body action.

17. The system of claim 16 , wherein the directing module directs the computer-generated avatar that represents the user to execute the predefined body action in accordance with the set of parameters associated with the muscle group by directing the computer-generated avatar that represents the user to execute the predefined body action in accordance with the digital signature of the portion of the body of the user.

18. The system of claim 11 , wherein the determining module associates the feature of the body of the user with the muscle group based on the predefined body action and the set of images by analyzing the set of images in accordance with a convolutional neural network pre-trained to associate muscle groups with features of users' bodies based on images of users executing the predefined body action.

19. The system of claim 11 , wherein:

the determining module further trains the artificial neural network to associate muscle groups with features of users' bodies based on images of users executing the predefined body action; and

the identifying module further identifies the muscle group by providing the artificial neural network with the set of images of the user.

20. A non-transitory computer-readable medium comprising computer-readable instructions that, when executed by at least one processor of a computing system, cause the computing system to:

identify a muscle group engaged by a user to execute a predefined body action by:

capturing a set of images of the user while the user executes the predefined body action; and

associating a feature of a body of the user with the muscle group based on the predefined body action and the set of images;

train an artificial neural network to determine parameters associated with the user, the muscle group, and the predefined body action, the parameters comprising:

an onset curve that describes a rate of change over time of an intensity level of the muscle group as the user executes the predefined body action; and

a falloff curve that describes a rate of change over time of an intensity level of the muscle group subsequent to the user executing the predefined body action;

determine, by providing the set of images as input to the trained artificial neural network, a set of parameters comprising:

an onset curve corresponding to the user, the muscle group, and the predefined body action; and

a falloff curve corresponding to the user, the muscle group, and the predefined body action; and

direct a computer-generated avatar that represents the body of the user to produce the predefined body action in accordance with the set of parameters.

Assignments (2)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 4, 2019
From: STEPTOE, WILLIAM ARTHUR HUGH; HOWARD, MICHAEL ANDREW; OZEL, MELINDA; NAKPIL, GIOVANNI F.; NAYLOR, TIMOTHY
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 050628/0727 →
Continuity (1)
Provisional Application 62732562 · Sep 17, 2018
Cited By (4)
US 12,315,054 US 12,602,849 US 12,651,410 US 12,711,686