IP Library › Granted Patent US 11,107,261

Granted Patent B2

US 11,107,261 · App. 16/663,062 · Granted Aug 31, 2021

Virtual avatar animation based on facial feature movement

Inventors: Nicolas Scapel (Sunnyvale, CA); Guillaume Pierre André Barlier (San Mateo, CA); Aurelio Guzman (San Jose, CA); Jason Rickwald (Santa Cruz, CA)

Assignee: Apple Inc.

G06T13/40 G06K9/00255 G06K9/00275

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,107,261
App. No.: 16/663,062
Granted: Aug 31, 2021
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

The present disclosure generally relates to displaying visual effects such as virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

Claims (197)

1. An electronic device, comprising:

a display apparatus;

one or more cameras;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including:

a first avatar feature;

a second avatar feature different from the first avatar feature; and

a third avatar feature different from the first and second avatar features;

while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and

in response to detecting the movement of the one or more facial features:

in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction:

moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and

moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction, wherein moving the second avatar feature and the third avatar feature in the second direction based on the detected movement of the first facial feature includes:

in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by a first magnitude, moving the second avatar feature and the third avatar feature by a first amount of movement; and

in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by a second magnitude different from the first magnitude, moving the second avatar feature and the third avatar feature by a second amount of movement that is different from the first amount of movement; and

in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature:

moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

2. The electronic device of claim 1 , the one or more programs further including instructions for:

in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature without more than a first threshold amount of movement of the first facial feature:

moving the second avatar feature based on at least one of the direction and the magnitude of movement of the second facial feature; and

forgoing moving the first and third avatar features.

3. The electronic device of claim 1 , the one or more programs further including instructions for:

in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature without more than a second threshold amount of movement of the second facial feature and a third facial feature:

moving the second and third avatar features based on the detected movement of the first facial feature.

4. The electronic device of claim 1 , the one or more programs further including instructions for:

moving the first avatar feature based on the detected movement of the first facial feature; and

moving the second avatar feature based on the detected movement of the first and second facial features.

5. The electronic device of claim 1 , wherein moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction includes:

in accordance with a determination that the detected movement of the first facial feature is less than or equal to a second threshold amount of movement of the first facial feature, moving the first avatar feature in the first direction by a first amount determined based on the detected movement of the first facial feature; and

in accordance with a determination that the detected movement of the first facial feature is greater than the second threshold amount of movement of the first facial feature:

moving the first avatar feature in the first direction by a second amount of movement greater than the first amount of movement, and

displaying a first subfeature of the first avatar feature that was not displayed when the movement of the first facial feature was less than the second threshold amount of movement.

6. The electronic device of claim 1 , wherein:

the first facial feature is a jaw of the face;

the first avatar feature is an avatar jaw feature;

the second avatar feature is one or more avatar eyes; and

the third avatar feature is an avatar nose.

7. The electronic device of claim 1 , the one or more programs further including instructions for:

moving the second avatar feature and the third avatar feature in a fourth direction different from the third direction and based on the detected movement of the first facial feature in the third direction.

8. The electronic device of claim 7 , the one or more programs further including instructions for:

further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in the third direction:

moving the first avatar feature in the third direction based on the detected movement of the first facial feature in the third direction.

9. The electronic device of claim 1 , the one or more programs further including instructions for:

further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by the first magnitude, moving the first avatar feature by a third amount of movement; and

further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by the second magnitude, moving the first avatar feature by a fourth amount of movement different from the third amount of movement.

10. The electronic device of claim 1 , the one or more programs further including instructions for:

detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the virtual avatar, including:

in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face, changing the appearance of the virtual avatar includes moving a first portion of the virtual avatar relative to a second portion of the virtual avatar.

11. The electronic device of claim 1 , wherein the virtual avatar further includes a fourth avatar feature that moves independently of the detected changes in pose of the face in the field of view of the one or more cameras, the one or more programs further including instructions for:

displaying the fourth avatar feature having a relative position with respect to the first, second, and third avatar features;

displaying movement of the fourth avatar feature relative to the first, second, and third avatar features; and

while continuing to display the movement of the fourth avatar feature, changing an appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features.

12. The electronic device of claim 11 , the one or more programs further including instructions for:

determining that the face is no longer detected within the field of view of the one or more cameras; and

in response to determining that the face is no longer detected within the field of view of the one or more cameras:

continuing to display the movement of the fourth avatar feature, and

changing the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features.

13. The electronic device of claim 11 , the one or more programs further including instructions for:

displaying, via the display apparatus, a representation of image data captured via the one or more cameras, wherein the representation of image data includes a representation of a subject; and

displaying, via the display apparatus, one or more avatar features of the virtual avatar in place of at least a portion of the representation of the subject, wherein the one or more avatar features of the virtual avatar does not include the fourth avatar feature.

14. The electronic device of claim 1 , the one or more programs further including instructions for:

in accordance with a determination that the detected movement of the one or more facial features includes movement of a fourth facial feature from a first position to a second position, moving a fifth avatar feature from an initial position to a modified position based on a magnitude of displacement of the fourth facial feature when moving from the first position to the second position.

15. The electronic device of claim 1 , the one or more programs further including instructions for:

in accordance with a determination that the detected movement of the one or more facial features includes movement of a fifth facial feature, moving a sixth avatar feature based on at least one of a direction and magnitude of movement of the fifth facial feature.

16. A method, comprising:

at an electronic device having one or more cameras and a display apparatus:

a first avatar feature;

a second avatar feature different from the first avatar feature; and

a third avatar feature different from the first and second avatar features;

in response to detecting the movement of the one or more facial features:

in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction:

moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and

in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature:

moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

17. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

moving the second avatar feature based on at least one of the direction and the magnitude of movement of the second facial feature; and

forgoing moving the first and third avatar features.

18. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature without more than a second threshold amount of movement of the second facial feature and a third facial feature:

moving the second and third avatar features based on the detected movement of the first facial feature.

19. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

moving the first avatar feature based on the detected movement of the first facial feature; and

moving the second avatar feature based on the detected movement of the first and second facial features.

20. The computer-readable storage medium of claim 16 , wherein moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction includes:

in accordance with a determination that the detected movement of the first facial feature is greater than the second threshold amount of movement of the first facial feature:

moving the first avatar feature in the first direction by a second amount of movement greater than the first amount of movement, and

displaying a first subfeature of the first avatar feature that was not displayed when the movement of the first facial feature was less than the second threshold amount of movement.

21. The computer-readable storage medium of claim 16 , wherein:

the first facial feature is a jaw of the face;

the first avatar feature is an avatar jaw feature;

the second avatar feature is one or more avatar eyes; and

the third avatar feature is an avatar nose.

22. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

23. The computer-readable storage medium of claim 22 , the one or more programs further including instructions for:

further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in the third direction:

moving the first avatar feature in the third direction based on the detected movement of the first facial feature in the third direction.

24. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

25. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the virtual avatar, including:

26. The computer-readable storage medium of claim 16 , wherein the virtual avatar further includes a fourth avatar feature that moves independently of the detected changes in pose of the face in the field of view of the one or more cameras, the one or more programs further including instructions for:

displaying the fourth avatar feature having a relative position with respect to the first, second, and third avatar features;

displaying movement of the fourth avatar feature relative to the first, second, and third avatar features; and

27. The computer-readable storage medium of claim 26 , the one or more programs further including instructions for:

determining that the face is no longer detected within the field of view of the one or more cameras; and

in response to determining that the face is no longer detected within the field of view of the one or more cameras:

continuing to display the movement of the fourth avatar feature, and

changing the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features.

28. The computer-readable storage medium of claim 26 , the one or more programs further including instructions for:

displaying, via the display apparatus, a representation of image data captured via the one or more cameras, wherein the representation of image data includes a representation of a subject; and

29. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

30. The computer-readable storage medium of claim 16 , the one or more programs further including instructions for:

31. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display apparatus and one or more cameras, the one or more programs including instructions for:

a first avatar feature;

a second avatar feature different from the first avatar feature; and

a third avatar feature different from the first and second avatar features;

in response to detecting the movement of the one or more facial features:

in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction:

moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and

in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature:

moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

32. The method of claim 31 , further comprising:

moving the second avatar feature based on at least one of the direction and the magnitude of movement of the second facial feature; and

forgoing moving the first and third avatar features.

33. The method of claim 31 , further comprising:

in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature without more than a second threshold amount of movement of the second facial feature and a third facial feature:

moving the second and third avatar features based on the detected movement of the first facial feature.

34. The method of claim 31 , further comprising:

moving the first avatar feature based on the detected movement of the first facial feature; and

moving the second avatar feature based on the detected movement of the first and second facial features.

35. The method of claim 31 , wherein moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction includes:

in accordance with a determination that the detected movement of the first facial feature is greater than the second threshold amount of movement of the first facial feature:

moving the first avatar feature in the first direction by a second amount of movement greater than the first amount of movement, and

displaying a first subfeature of the first avatar feature that was not displayed when the movement of the first facial feature was less than the second threshold amount of movement.

36. The method of claim 31 , wherein:

the first facial feature is a jaw of the face;

the first avatar feature is an avatar jaw feature;

the second avatar feature is one or more avatar eyes; and

the third avatar feature is an avatar nose.

37. The method of claim 31 , further comprising:

38. The method of claim 37 , further comprising:

further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in the third direction:

moving the first avatar feature in the third direction based on the detected movement of the first facial feature in the third direction.

39. The method of claim 31 , further comprising:

40. The method of claim 31 , further comprising:

detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the virtual avatar, including:

41. The method of claim 31 , wherein the virtual avatar further includes a fourth avatar feature that moves independently of the detected changes in pose of the face in the field of view of the one or more cameras, the method further comprising:

displaying the fourth avatar feature having a relative position with respect to the first, second, and third avatar features;

displaying movement of the fourth avatar feature relative to the first, second, and third avatar features; and

42. The method of claim 41 , further comprising:

determining that the face is no longer detected within the field of view of the one or more cameras; and

in response to determining that the face is no longer detected within the field of view of the one or more cameras:

continuing to display the movement of the fourth avatar feature, and

changing the appearance of the fourth avatar feature based on the relative position of the fourth avatar feature with respect to the first, second, and third avatar features.

43. The method of claim 41 , further comprising:

displaying, via the display apparatus, a representation of image data captured via the one or more cameras, wherein the representation of image data includes a representation of a subject; and

44. The method of claim 31 , further comprising:

45. The method of claim 31 , further comprising:

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 13, 2020

From: SCAPEL, NICOLAS; BARLIER, GUILLAUME PIERRE ANDRE; GUZMAN, AURELIO; RICKWALD, JASON

To: APPLE INC.

Reel/Frame 051494/0414 →

Continuity (2)

Provisional Application 62794496 · Jan 18, 2019

Related Publication 20200234481A1 · Jul 23, 2020

Cited By (12)

US 12,218,894 US 12,223,612 US 12,287,913 US 12,340,481 US 12,379,834 US 12,422,977 US 12,450,811 US 12,482,161 US 12,619,345 US 12,638,913 US 12,647,673 US 12,713,125