IP Library › Granted Patent US 11,048,916
Granted Patent B2
US 11,048,916 · App. 16/409,390 · Granted Jun 29, 2021

Automated avatar generation

Inventors: Maksim Gusarov (Marina del Rey, CA); Igor Kudriashov (Saratov, RU); Valerii Filev (Riga, LV); Sergei Kotcur (Sochi, RU)
Assignee: Snap Inc.
G06K9/00281G06K9/00248G06T13/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,048,916
App. No.
16/409,390
Granted
Jun 29, 2021
Kind
B2
Abstract

Systems, devices, media, and methods are presented for generating facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial landmarks within the portion of the face. The systems and methods determine one or more characteristics representing the portion of the face, in response to detecting the portion of the face. Based on the one or more characteristics and the set of facial landmarks, the systems and methods generate a representation of a face.

Claims (84)

1. A method, comprising:

receiving, by one or more processors, a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining, by the one or more processors, a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating, by the one or more processors, a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements representing a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.

2. The method of claim 1 further comprising:

presenting a confirmation message that includes the representation of the face and an acceptance user interface element and a rejection user interface element;

in response to receiving input selecting the rejection user interface element, causing presentation of the plurality of user interface elements for modifying the physical attribute of the generated representation of the face; and

in response to receiving input selecting one of the plurality of user interface elements, modifying the representation of the face based on the selected one of the plurality of user interface elements.

3. The method of claim 1 further comprising:

presenting a message associated with the physical attribute that includes the representation of the face and an acceptance user interface element and a rejection user interface element;

in response to receiving input selecting the rejection user interface element, causing presentation of the plurality of user interface elements for modifying a first type of physical attribute of the generated representation of the face; and

in response to receiving input selecting one of the plurality of user interface elements, modifying the representation of the face based on the selected one of the plurality of user interface elements.

4. The method of claim 1 , wherein determining the hair texture comprises:

identifying a set of colors within the hair region of the face; and

determining variations in the set of colors that indicate edges, objects or shapes within hair associated with the hair texture.

5. The method of claim 4 further comprising:

comparing a dimension of the hair region to the one or more of a set of facial landmarks; and

determining a hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks.

6. The method of claim 1 further comprising selecting a set of hat templates from clothing templates in response to determining that an obstruction obscures the hair region of a portion of facial landmarks representing a forehead of the face.

7. The method of claim 1 , further comprising:

identifying the set of facial landmarks within the portion of the face; and

in response to identifying the set of facial landmarks, determining one or more characteristics representing the portion of the face depicted in the 2D image by:

identifying a set of colors within the hair region of the face, wherein the determination of the hair texture is based on variations in the set of colors that indicate edges, objects or shapes within hair associated with the hair texture.

8. The method of claim 1 further comprising activating a given user interface element of the plurality of user interface elements to perform the modification associated with the given user interface element in response to detecting touch at an area of a display occupied by the given user interface element.

9. The method of claim 1 further comprising:

determining a first physical attribute type of the portion of the face based on one or more distances between two or more facial landmarks; and

determining a second physical attribute type by identifying an area of interest on the portion of the face and extracting an average color depicted within the area of interest.

10. The method of claim 7 , further comprising:

determining one or more distances between two or more facial landmarks of the set of facial landmarks to determine the one or more characteristics, wherein determining the distances comprises:

determining a first distance between facial landmarks of the set of facial landmarks representing eyes depicted on the portion of the face;

determining a second distance between facial landmarks of the set of facial landmarks representing the eyes and a nose depicted on the portion of the face;

determining a third distance between facial landmarks of the set of facial landmarks representing the eyes and a mouth depicted on the portion of the face; and

determining a fourth distance between facial landmarks of the set of facial landmarks representing the eyes and a chin depicted on the portion of the face.

11. The method of claim 1 , further comprising:

identifying one or more iris within the face; and

determining a prevailing color for the one or more iris, wherein identifying the one or more iris further comprises:

determining a shape of one or more eyes surrounding the one or more iris;

determining a height of the shape; and

determining an iris dimension based on the height of the shape of the one or more eyes.

12. The method of claim 1 further comprising:

determining one or more eyebrow regions of the face; and

identifying one or more shapes of the one or more eyebrow region.

13. The method of claim 1 further comprising:

identifying an obstruction on the face depicted in the 2D image; and

selecting a type of physical element to add to the generated representation of the face based on a location or position of the obstruction on the face.

14. The method of claim 1 , wherein a first hair attribute of a plurality of hair attributes of the hair region comprises hair volume, further comprising determining the hair volume is determined by:

identifying a first portion of the set of facial landmarks that represents an outline of the face; and

computing a distance a hair region extends from the identified portion of the set of facial landmarks that represents the outline of the face to an outer opposing edge of the hair region.

15. The method of claim 14 , wherein a second hair attribute of the plurality of hair attributes comprises hair length, further comprising determining the hair length by:

identifying a second portion of the set of facial landmarks that represents a chin; and

computing a distance from the hair region to the second portion of the set of facial landmarks that represents the chin.

16. The method of claim 1 further comprising:

selecting a region of the face where obstructions commonly appear;

determining whether an obstruction is present in the selected region using one or more of a set of facial landmarks; and

in response to determining that the obstruction is present, matching the obstruction to a template selected from a set of templates based on a location of the obstruction.

17. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors and storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements represent a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.

18. The method of claim 16 , further comprising selecting a set of hat templates from clothing templates in response to determining that an obstruction obscures the hair region of a portion of facial landmarks representing a forehead of the face.

19. The method of claim 16 , further comprising:

selecting a set of glasses templates in response to determining that an obstruction encompasses or is positioned proximate to facial landmarks representing eyes and nose of the face; and

selecting a set of facial hair templates in response to determining that the obstruction is positioned proximate to or obscures facial landmarks representing a mouth or jaw line of the face.

20. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements represent a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 16, 2021
From: GUSAROV, MAKSIM; KUDRIASHOV, IGOR; FILEV, VALERII; KOTCUR, SERGEI
To: SNAPCHAT, INC.
Reel/Frame 055608/0114 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 16, 2021
From: SNAPCHAT, INC.
To: SNAP INC.
Reel/Frame 055608/0306 →
Continuity (2)
Continuation 15086749 · Mar 31, 2016
Related Publication 20190266390A1 · Aug 29, 2019
Cited By (4)
US 12,206,635 US 12,316,589 US 12,363,056 US 12,616,903