IP Library › Granted Patent US 10,719,968
Granted Patent B2
US 10,719,968 · App. 16/387,092 · Granted Jul 21, 2020

Augmented expression system

Inventors: Chen Cao (Los Angeles, CA); Yang Gao (Toronto, CA); Zehao Xue (Los Angeles, CA)
Assignee: Snap Inc.
G06T13/20G06K9/00302G06T7/73G06T19/006G06T19/20G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,719,968
App. No.
16/387,092
Granted
Jul 21, 2020
Kind
B2
Abstract

Embodiments described herein relate to an augmented expression system to generate and cause display of a specially configured interface to present an augmented reality perspective. The augmented expression system receives image and video data of a user and tracks facial landmarks of the user based on the image and video data, in real-time to generate and present a 3-dimensional (3D) bitmoji of the user.

Claims (81)

1. A method comprising:

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation of the image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from among the set of facial landmarks;

applying the graphical element to a position upon the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

2. The method of claim 1 , wherein the method further comprises:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

3. The method of claim 1 , wherein the method further comprises:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

4. The method of claim 3 , wherein the facial tracking input is a first facial tracking input, and the method further comprises:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.

5. The method of claim 1 , wherein the method further comprises:

detecting a rotation of a head of the user based on at least the facial tracking input; and

rotating the 3D model based on the rotation of the head of the user.

6. The method of claim 1 , wherein the client device is a first client device, and the method further comprises:

distributing the data packet that includes the flattened presentation of the 3D model to a second client device; and

causing display of the flattened presentation at the second client device.

7. The method f claim 6 , wherein the message includes an ephemeral message.

8. The method of claim 1 , wherein the 3D model comprises a set of regions, each region of the 3D model corresponding to a facial landmark from among the set of facial landmarks of the user, and the method further comprises:

identifying a region of the 3D model based on the facial tracking input; and

dynamically animating the region in response to the receiving the facial tracking input.

9. A system comprising:

a memory; and

at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising:

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation of the image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from among the set of facial landmarks;

applying the graphical element to a position upon animating the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

10. The system of claim 9 , wherein the instructions cause the system to perform operations further comprising:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

11. The system of claim 9 , wherein the instructions cause the system to perform operations further comprising:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

12. The system of claim 11 , wherein the facial tracking input is a first facial tracking input, and the instructions cause the system to perform operations further comprising:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.

13. The system of claim 9 , wherein the instructions cause the system to perform operations further comprising:

detecting a rotation of a head of the user based on at least the facial tracking input; and

rotating the 3D model based on the rotation of the head of the user.

14. The system of claim 9 , wherein the client device is a first client device, and the instructions cause the system to perform operations further comprising:

distributing the data packet that includes the flattened presentation of the 3D model to a second client device; and

causing display of the flattened presentation at the second client device.

15. The system of claim 14 , wherein the message includes an ephemeral message.

16. The system of claim 9 , wherein the 3D model comprises a set of regions, each region of the 3D model corresponding to a facial landmark from among the set of facial landmarks of the user, and the instructions cause the system to perform operations further comprising:

identifying a region of the 3D model based on the facial tracking input; and dynamically animating the region in response to the receiving the facial tracking input.

17. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation e image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from anion the set of facial landmarks;

applying the graphical element to a position upon the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

18. The non-transitory machine-readable storage medium of claim 17 , further comprising instructions that cause the machine to perform operations further comprising:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

19. The non-transitory machine-readable storage medium of claim 17 , further comprising instructions that cause the machine to perform operations further comprising:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

20. The non-transitory machine-readable storage medium of claim 19 , further comprising instructions that cause the machine to perform operations further comprising:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 12, 2020
From: CAO, CHEN; GAO, YANG; XUE, ZEHAO
To: SNAP INC.
Reel/Frame 052924/0900 →
Continuity (2)
Provisional Application 62659337 · Apr 18, 2018
Related Publication 20190325631A1 · Oct 24, 2019
Cited By (2)
US 12,198,470 US 12,469,196