Emoji recording and sending
The present disclosure generally relates to generating and modifying virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.
1. An electronic device, comprising:
a display apparatus;
one or more cameras;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display apparatus, a virtual avatar, having a plurality of avatar features, that changes appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the face including a plurality of detected facial features, including a first facial feature;
after the face was detected in the field of view of the one or more cameras with a first change in pose, the first change in pose including a change to the first facial feature, determining that tracking of the face in the field of view of the one or more cameras has failed; and
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a first avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the first avatar feature is updated based on change characteristics of the first facial feature that were detected prior to determining that tracking of the face has failed, wherein updating the appearance of the first avatar feature includes:
in accordance with a determination that the first facial feature had first change characteristics prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a first appearance that is different from an appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the first facial feature had second change characteristics, different from the first change characteristics, prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a second appearance that is different from the appearance of the avatar when tracking of the face failed and different from the first appearance.
2. The electronic device of claim 1 , the one or more programs further including instructions for:
in accordance with a determination that the first change characteristics include a first changed direction of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first changed direction of the first facial feature; and
in accordance with a determination that the first change characteristics include a second changed direction of the first facial feature different from the first changed direction, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second changed direction of the first facial feature.
3. The electronic device of claim 1 , the one or more programs further including instructions for:
in accordance with a determination that the first change characteristics include a first magnitude of change of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first magnitude of change of the first facial feature; and
in accordance with a determination that the first change characteristics include a second magnitude of change of the first facial feature different from the first magnitude of change, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second magnitude of change of the first facial feature.
4. The electronic device of claim 1 , wherein the plurality of detected facial features includes a second facial feature and the first change in pose includes a change to the second facial feature, the one or more programs further including instructions for:
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a second avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the second avatar feature is updated based on change characteristics of the second facial feature that were detected prior to determining that tracking of the face has failed:
in accordance with a determination that the second facial feature had third change characteristics prior to detecting that tracking of the face failed, updating the appearance of a second avatar feature of the virtual avatar to a third appearance that is different from the appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the second facial feature had fourth change characteristics prior to detecting that tracking of the face failed, updating the appearance of the second avatar feature of the virtual avatar to a fourth appearance that is different from the appearance of the avatar when tracking of the face failed and different from the third appearance.
5. The electronic device of claim 4 , wherein:
the appearance of the first avatar feature is updated at a first rate of change; and
the appearance of the second avatar feature is updated at a second rate of change, different than the first rate of change.
6. The electronic device of claim 5 , wherein the first avatar feature is an avatar head, and the second avatar feature is a facial feature of the virtual avatar.
7. The electronic device of claim 4 , wherein:
updating the appearance of the first avatar feature includes updating the appearance in accordance with a first physics model; and
updating the appearance of the second avatar feature includes updating the appearance in accordance with a second physics model, different than the first physics model.
8. The electronic device of claim 4 , wherein:
the appearance of the first avatar feature is updated at a first decreasing rate of change; and
the appearance of the second avatar feature is updated at a second decreasing rate of change, different than the first decreasing rate of change.
9. The electronic device of claim 8 , wherein:
the first avatar feature is an avatar head;
the second avatar feature is a facial feature of the virtual avatar; and
the second decreasing rate of change decreases at a faster rate than a rate of decrease of the first decreasing rate of change.
10. The electronic device of claim 8 , wherein:
the appearance of the first avatar feature is updated until a value associated with a range of motion of the first avatar feature reaches a predetermined value; and
the appearance of the second avatar feature is updated until a value associated with a range of motion of the second avatar feature reaches a predetermined value.
11. The electronic device of claim 8 , wherein:
the appearance of the first avatar feature is updated based on a scaled value of the first decreasing rate of change; and
the appearance of the second avatar feature is updated based on a scaled value of the second decreasing rate of change.
12. The electronic device of claim 4 , wherein the appearance of the second avatar feature is updated until the second avatar feature reaches a predetermined position.
13. The electronic device of claim 4 , the one or more programs further including instructions for:
in response to determining that the face has not been detected within the field of view of the one or more cameras for a threshold amount of time:
updating the appearance of the first avatar feature to a first default pose; and
updating the appearance of the second avatar feature to a second default pose.
14. The electronic device of claim 13 , wherein the threshold amount of time is achieved after the appearance of the first avatar feature is updated and after the appearance of the second avatar feature is updated.
15. The electronic device of claim 1 , the one or more programs further including instructions for:
after updating the appearance of the first avatar feature after tracking of the face failed, resuming tracking of the face while the face is in a first pose; and
in response to resuming tracking of the face, updating the appearance of the first avatar feature to a pose based on the first pose of the face.
16. The electronic device of claim 15 , wherein updating the appearance of the first avatar feature to a pose based on the first pose of the face includes gradually updating the appearance of the first avatar feature to the pose based on the first pose of the face.
17. The electronic device of claim 15 , the one or more programs further including instructions for:
in response to resuming tracking of the face:
prior to updating the appearance of the first avatar feature to the pose based on the first pose of the face, disabling a physics model associated with at least one of the first avatar feature, the second avatar feature, or a third avatar feature;
after updating the appearance of the first avatar feature to the pose based on the first pose of the face, gradually resuming the disabled physics model;
detecting a change in the first pose of the face; and
updating the appearance of the first avatar feature based on the detected change in the first pose of the face and the resumed physics model.
18. The electronic device of claim 1 , wherein:
displaying the virtual avatar includes displaying, via the display apparatus, a representation of a subject and a background positioned in the field of view of the one or more cameras, wherein the virtual avatar is displayed on the representation of the subject.
19. The electronic device of claim 1 , the one or more programs further including instructions for:
prior to determining that tracking of the face in the field of view of the one or more cameras has failed, detecting the face within the field of view of the one or more cameras with a second change in pose, the second change in pose including a second change to the plurality of detected facial features; and
in response to detecting the face in the field of view of the one or more cameras with the second change in pose:
in accordance with a determination that the plurality of detected facial features had fifth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a fifth appearance that is based on the fifth change characteristics; and
in accordance with a determination that the plurality of detected facial features had sixth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a sixth appearance that is based on the sixth change characteristics and different from the fifth appearance.
20. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display apparatus and one or more cameras, the one or more programs including instructions for:
displaying, via the display apparatus, a virtual avatar, having a plurality of avatar features, that changes appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the face including a plurality of detected facial features, including a first facial feature;
after the face was detected in the field of view of the one or more cameras with a first change in pose, the first change in pose including a change to the first facial feature, determining that tracking of the face in the field of view of the one or more cameras has failed; and
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a first avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the first avatar feature is updated based on change characteristics of the first facial feature that were detected prior to determining that tracking of the face has failed, wherein updating the appearance of the first avatar feature includes:
in accordance with a determination that the first facial feature had first change characteristics prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a first appearance that is different from an appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the first facial feature had second change characteristics, different from the first change characteristics, prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a second appearance that is different from the appearance of the avatar when tracking of the face failed and different from the first appearance.
21. The non-transitory computer-readable storage medium of claim 20 , the one or more programs further including instructions for:
in accordance with a determination that the first change characteristics include a first changed direction of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first changed direction of the first facial feature; and
in accordance with a determination that the first change characteristics include a second changed direction of the first facial feature different from the first changed direction, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second changed direction of the first facial feature.
22. The non-transitory computer-readable storage medium of claim 20 , the one or more programs further including instructions for:
in accordance with a determination that the first change characteristics include a first magnitude of change of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first magnitude of change of the first facial feature; and
in accordance with a determination that the first change characteristics include a second magnitude of change of the first facial feature different from the first magnitude of change, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second magnitude of change of the first facial feature.
23. The non-transitory computer-readable storage medium of claim 20 , wherein the plurality of detected facial features includes a second facial feature and the first change in pose includes a change to the second facial feature, the one or more programs further including instructions for:
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a second avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the second avatar feature is updated based on change characteristics of the second facial feature that were detected prior to determining that tracking of the face has failed:
in accordance with a determination that the second facial feature had third change characteristics prior to detecting that tracking of the face failed, updating the appearance of a second avatar feature of the virtual avatar to a third appearance that is different from the appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the second facial feature had fourth change characteristics prior to detecting that tracking of the face failed, updating the appearance of the second avatar feature of the virtual avatar to a fourth appearance that is different from the appearance of the avatar when tracking of the face failed and different from the third appearance.
24. The non-transitory computer-readable storage medium of claim 23 , wherein:
the appearance of the first avatar feature is updated at a first rate of change; and
the appearance of the second avatar feature is updated at a second rate of change, different than the first rate of change.
25. The non-transitory computer-readable storage medium of claim 24 , wherein the first avatar feature is an avatar head, and the second avatar feature is a facial feature of the virtual avatar.
26. The non-transitory computer-readable storage medium of claim 23 , wherein:
updating the appearance of the first avatar feature includes updating the appearance in accordance with a first physics model; and
updating the appearance of the second avatar feature includes updating the appearance in accordance with a second physics model, different than the first physics model.
27. The non-transitory computer-readable storage medium of claim 23 , wherein:
the appearance of the first avatar feature is updated at a first decreasing rate of change; and
the appearance of the second avatar feature is updated at a second decreasing rate of change, different than the first decreasing rate of change.
28. The non-transitory computer-readable storage medium of claim 27 , wherein:
the first avatar feature is an avatar head;
the second avatar feature is a facial feature of the virtual avatar; and
the second decreasing rate of change decreases at a faster rate than a rate of decrease of the first decreasing rate of change.
29. The non-transitory computer-readable storage medium of claim 27 , wherein:
the appearance of the first avatar feature is updated until a value associated with a range of motion of the first avatar feature reaches a predetermined value; and
the appearance of the second avatar feature is updated until a value associated with a range of motion of the second avatar feature reaches a predetermined value.
30. The non-transitory computer-readable storage medium of claim 27 , wherein:
the appearance of the first avatar feature is updated based on a scaled value of the first decreasing rate of change; and
the appearance of the second avatar feature is updated based on a scaled value of the second decreasing rate of change.
31. The non-transitory computer-readable storage medium of claim 23 , wherein the appearance of the second avatar feature is updated until the second avatar feature reaches a predetermined position.
32. The non-transitory computer-readable storage medium of claim 23 , the one or more programs further including instructions for:
in response to determining that the face has not been detected within the field of view of the one or more cameras for a threshold amount of time:
updating the appearance of the first avatar feature to a first default pose; and
updating the appearance of the second avatar feature to a second default pose.
33. The non-transitory computer-readable storage medium of claim 32 , wherein the threshold amount of time is achieved after the appearance of the first avatar feature is updated and after the appearance of the second avatar feature is updated.
34. The non-transitory computer-readable storage medium of claim 20 , the one or more programs further including instructions for:
after updating the appearance of the first avatar feature after tracking of the face failed, resuming tracking of the face while the face is in a first pose; and
in response to resuming tracking of the face, updating the appearance of the first avatar feature to a pose based on the first pose of the face.
35. The non-transitory computer-readable storage medium of claim 34 , wherein updating the appearance of the first avatar feature to a pose based on the first pose of the face includes gradually updating the appearance of the first avatar feature to the pose based on the first pose of the face.
36. The non-transitory computer-readable storage medium of claim 34 , the one or more programs further including instructions for:
in response to resuming tracking of the face:
prior to updating the appearance of the first avatar feature to the pose based on the first pose of the face, disabling a physics model associated with at least one of the first avatar feature, the second avatar feature, or a third avatar feature;
after updating the appearance of the first avatar feature to the pose based on the first pose of the face, gradually resuming the disabled physics model;
detecting a change in the first pose of the face; and
updating the appearance of the first avatar feature based on the detected change in the first pose of the face and the resumed physics model.
37. The non-transitory computer-readable storage medium of claim 20 , wherein:
displaying the virtual avatar includes displaying, via the display apparatus, a representation of a subject and a background positioned in the field of view of the one or more cameras, wherein the virtual avatar is displayed on the representation of the subject.
38. The non-transitory computer-readable storage medium of claim 20 , the one or more programs further including instructions for:
prior to determining that tracking of the face in the field of view of the one or more cameras has failed, detecting the face within the field of view of the one or more cameras with a second change in pose, the second change in pose including a second change to the plurality of detected facial features; and
in response to detecting the face in the field of view of the one or more cameras with the second change in pose:
in accordance with a determination that the plurality of detected facial features had fifth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a fifth appearance that is based on the fifth change characteristics; and
in accordance with a determination that the plurality of detected facial features had sixth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a sixth appearance that is based on the sixth change characteristics and different from the fifth appearance.
39. A method, comprising:
at an electronic device having one or more cameras and a display apparatus:
displaying, via the display apparatus, a virtual avatar, having a plurality of avatar features, that changes appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the face including a plurality of detected facial features, including a first facial feature;
after the face was detected in the field of view of the one or more cameras with a first change in pose, the first change in pose including a change to the first facial feature, determining that tracking of the face in the field of view of the one or more cameras has failed; and
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a first avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the first avatar feature is updated based on change characteristics of the first facial feature that were detected prior to determining that tracking of the face has failed, wherein updating the appearance of the first avatar feature includes:
in accordance with a determination that the first facial feature had first change characteristics prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a first appearance that is different from an appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the first facial feature had second change characteristics, different from the first change characteristics, prior to detecting that tracking of the face failed, updating the appearance of the first avatar feature of the virtual avatar to a second appearance that is different from the appearance of the avatar when tracking of the face failed and different from the first appearance.
40. The method of claim 39 , further comprising:
in accordance with a determination that the first change characteristics include a first changed direction of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first changed direction of the first facial feature; and
in accordance with a determination that the first change characteristics include a second changed direction of the first facial feature different from the first changed direction, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second changed direction of the first facial feature.
41. The method of claim 39 , further comprising:
in accordance with a determination that the first change characteristics include a first magnitude of change of the first facial feature, updating the appearance of the first avatar feature to the first appearance includes updating the appearance of the first avatar feature based on the first magnitude of change of the first facial feature; and
in accordance with a determination that the first change characteristics include a second magnitude of change of the first facial feature different from the first magnitude of change, updating the appearance of the first avatar feature to the second appearance includes updating the appearance of the first avatar feature based on the second magnitude of change of the first facial feature.
42. The method of claim 39 , wherein the plurality of detected facial features includes a second facial feature and the first change in pose includes a change to the second facial feature, the method further comprising:
in response to determining that tracking of the face in the field of view of the one or more cameras has failed, updating an appearance of a second avatar feature of the plurality of avatar features after tracking of the face failed, wherein the appearance of the second avatar feature is updated based on change characteristics of the second facial feature that were detected prior to determining that tracking of the face has failed:
in accordance with a determination that the second facial feature had third change characteristics prior to detecting that tracking of the face failed, updating the appearance of a second avatar feature of the virtual avatar to a third appearance that is different from the appearance of the avatar when tracking of the face failed; and
in accordance with a determination that the second facial feature had fourth change characteristics prior to detecting that tracking of the face failed, updating the appearance of the second avatar feature of the virtual avatar to a fourth appearance that is different from the appearance of the avatar when tracking of the face failed and different from the third appearance.
43. The method of claim 42 , wherein:
the appearance of the first avatar feature is updated at a first rate of change; and
the appearance of the second avatar feature is updated at a second rate of change, different than the first rate of change.
44. The method of claim 43 , wherein the first avatar feature is an avatar head, and the second avatar feature is a facial feature of the virtual avatar.
45. The method of claim 42 , wherein:
updating the appearance of the first avatar feature includes updating the appearance in accordance with a first physics model; and
updating the appearance of the second avatar feature includes updating the appearance in accordance with a second physics model, different than the first physics model.
46. The method of claim 42 , wherein:
the appearance of the first avatar feature is updated at a first decreasing rate of change; and
the appearance of the second avatar feature is updated at a second decreasing rate of change, different than the first decreasing rate of change.
47. The method of claim 46 , wherein:
the first avatar feature is an avatar head;
the second avatar feature is a facial feature of the virtual avatar; and
the second decreasing rate of change decreases at a faster rate than a rate of decrease of the first decreasing rate of change.
48. The method of claim 46 , wherein:
the appearance of the first avatar feature is updated until a value associated with a range of motion of the first avatar feature reaches a predetermined value; and
the appearance of the second avatar feature is updated until a value associated with a range of motion of the second avatar feature reaches a predetermined value.
49. The method of claim 46 , wherein:
the appearance of the first avatar feature is updated based on a scaled value of the first decreasing rate of change; and
the appearance of the second avatar feature is updated based on a scaled value of the second decreasing rate of change.
50. The method of claim 42 , wherein the appearance of the second avatar feature is updated until the second avatar feature reaches a predetermined position.
51. The method of claim 42 , further comprising:
in response to determining that the face has not been detected within the field of view of the one or more cameras for a threshold amount of time:
updating the appearance of the first avatar feature to a first default pose; and
updating the appearance of the second avatar feature to a second default pose.
52. The method of claim 51 , wherein the threshold amount of time is achieved after the appearance of the first avatar feature is updated and after the appearance of the second avatar feature is updated.
53. The method of claim 39 , further comprising:
after updating the appearance of the first avatar feature after tracking of the face failed, resuming tracking of the face while the face is in a first pose; and
in response to resuming tracking of the face, updating the appearance of the first avatar feature to a pose based on the first pose of the face.
54. The method of claim 53 , wherein updating the appearance of the first avatar feature to a pose based on the first pose of the face includes gradually updating the appearance of the first avatar feature to the pose based on the first pose of the face.
55. The method of claim 53 , further comprising:
in response to resuming tracking of the face:
prior to updating the appearance of the first avatar feature to the pose based on the first pose of the face, disabling a physics model associated with at least one of the first avatar feature, the second avatar feature, or a third avatar feature;
after updating the appearance of the first avatar feature to the pose based on the first pose of the face, gradually resuming the disabled physics model;
detecting a change in the first pose of the face; and
updating the appearance of the first avatar feature based on the detected change in the first pose of the face and the resumed physics model.
56. The method of claim 39 , wherein:
displaying the virtual avatar includes displaying, via the display apparatus, a representation of a subject and a background positioned in the field of view of the one or more cameras, wherein the virtual avatar is displayed on the representation of the subject.
57. The method of claim 39 , further comprising:
prior to determining that tracking of the face in the field of view of the one or more cameras has failed, detecting the face within the field of view of the one or more cameras with a second change in pose, the second change in pose including a second change to the plurality of detected facial features; and
in response to detecting the face in the field of view of the one or more cameras with the second change in pose:
in accordance with a determination that the plurality of detected facial features had fifth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a fifth appearance that is based on the fifth change characteristics; and
in accordance with a determination that the plurality of detected facial features had sixth change characteristics while tracking the face, updating the appearance of the first avatar feature of the virtual avatar to a sixth appearance that is based on the sixth change characteristics and different from the fifth appearance.