IP Library Granted Patent US 10,025,972
Granted Patent B2
US 10,025,972 · App. 14/942,784 · Granted Jul 17, 2018

Systems and methods for dynamically generating emojis based on image analysis of facial features

Inventors: Michael James Matas (Healdsburg, CA); Michael Waldman Reckhow (San Francisco, CA); Yaniv Taigman (Los Altos, CA)
Assignee: Facebook, Inc.
G06K9/00315G06F17/212G06F17/24G06K9/00281G06K9/00335
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,025,972
App. No.
14/942,784
Granted
Jul 17, 2018
Kind
B2
Abstract

Systems, methods, and non-transitory computer-readable media can acquire real-time image data depicting at least a portion of a face of a user of a computing system (or device). The real-time image data can be analyzed to determine a state associated with at least the portion of the face. An emoji can be provided based on the state associated with at least the portion of the face. The emoji can be inputted in a communication to be made by the user.

Claims (63)

1. A computer-implemented method comprising:

acquiring, by a computing system, real-time image data depicting at least a portion of a face of a user of the computing system;

analyzing, by the computing system, the real-time image data to determine a state associated with at least the portion of the face;

providing, by the computing system, an emoji based on the state associated with at least the portion of the face; and

inputting, by the computing system, the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

2. The computer-implemented method of claim 1 , further comprising:

analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;

identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and

modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

3. The computer-implemented method of claim 2 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

4. The computer-implemented method of claim 1 , further comprising:

analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;

selecting a graphical representation for the at least one of the object or the gesture; and

providing the graphical representation in conjunction with the emoji.

5. The computer-implemented method of claim 1 , wherein analyzing the real-time image data to determine the state associated with at least the portion of the face further comprises:

analyzing one or more virtual points on at least the portion of the face to identify a virtual point arrangement; and

matching, within an allowable deviation, the virtual point arrangement with a facial expression model out of a plurality of facial expression models.

6. The computer-implemented method of claim 5 , further comprising:

identifying the emoji out of a plurality of emojis based on the facial expression model, wherein each of the plurality of emojis is respectively associated with each of the plurality of facial expression models.

7. The computer-implemented method of claim 5 , wherein matching the virtual point arrangement with the facial expression model is based on at least one of a machine learning training process or a crowdsource training process.

8. The computer-implemented method of claim 1 , further comprising:

analyzing the real-time image data to determine a second state associated with at least the portion of the face; and

updating, based on the second state associated with at least the portion of the face, the emoji provided.

9. The computer-implemented method of claim 1 , further comprising:

publishing the communication as at least one of a message, a journal entry, a diary entry, a blog entry, a comment, a response, or a post.

10. The computer-implemented method of claim 1 , further comprising:

dynamically presenting a preview of the emoji prior to inputting the emoji in the communication to be made by the user.

11. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to perform:

acquiring real-time image data depicting at least a portion of a face of a user of the system;

analyzing the real-time image data to determine a state associated with at least the portion of the face;

providing an emoji based on the state associated with at least the portion of the face; and

inputting the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

12. The system of claim 11 , wherein the instructions cause the system to further perform:

analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;

identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and

modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

13. The system of claim 12 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

14. The system of claim 11 , wherein the instructions cause the system to further perform:

analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;

selecting a graphical representation for the at least one of the object or the gesture; and

providing the graphical representation in conjunction with the emoji.

15. The system of claim 11 , wherein analyzing the real-time image data to determine the state associated with at least the portion of the face further comprises:

analyzing one or more virtual points on at least the portion of the face to identify a virtual point arrangement; and

matching, within an allowable deviation, the virtual point arrangement with a facial expression model out of a plurality of facial expression models.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:

acquiring real-time image data depicting at least a portion of a face of a user of the computing system;

analyzing the real-time image data to determine a state associated with at least the portion of the face;

providing an emoji based on the state associated with at least the portion of the face; and

inputting the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions cause the computing system to further perform:

analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;

identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and

modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions cause the computing system to further perform:

analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;

selecting a graphical representation for the at least one of the object or the gesture; and

providing the graphical representation in conjunction with the emoji.

20. The non-transitory computer-readable storage medium of claim 16 , wherein analyzing the real-time image data to determine the state associated with at least the portion of the face further comprises:

analyzing one or more virtual points on at least the portion of the face to identify a virtual point arrangement; and

matching, within an allowable deviation, the virtual point arrangement with a facial expression model out of a plurality of facial expression models.

Assignments (3)
CHANGE OF NAME Recorded Feb 18, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 059232/0232 →
CONFIDENTIAL INFORMATION AND INVENTION ASSIGNMENT AGREEMENT Recorded May 25, 2018
From: MATAS, MICHAEL JAMES
To: FACEBOOK, INC.
Reel/Frame 046977/0734 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 25, 2018
From: RECKHOW, MICHAEL WALDMAN; TAIGMAN, YANIV
To: FACEBOOK, INC.
Reel/Frame 045905/0143 →
Continuity (1)
Related Publication 20170140214A1 · May 18, 2017
Cited By (6)
US 12,274,551 US 12,401,911 US 12,401,912 US 12,418,727 US 12,445,736 US 12,666,159