Granted Patent B2

US 10,657,695 · App. 15/797,875 · Granted May 19, 2020

Inventors: Jesse Chand (Los Angeles, CA); Jeremy Voss (Los Angeles, CA)

Assignee: Snap Inc.

G06T13/40G06K9/00288G06T17/20G06T19/20G10L15/265H04L51/10H04L65/1069H04L65/403G06F3/0482G06T2219/2016G10L15/26H04N7/157

Abstract

Claims (76)

1. A method comprising:

detecting an initiation of a communication session at a first client device, the communication session between the first client device and a second client device;

causing the first client device to capture image data that depicts a face of a user of the first client device, and audio data in response to the detecting the initiation of the communication session;

generating a mesh representation of the face of the user based on the image data, the mesh representation including a point-of-gaze indicator;

selecting a set of avatar features from among a collection of avatar features based on a user account associated with the first client device, the user account including an identification of the set of avatar features;

generating an avatar to be associated with the user of the first client device based on the set of avatar features and the mesh representation;

causing display of a communication interface at the second client device in response to the initiation of the communication session, the communication interface comprising a presentation of the avatar associated with the user of the first client device and a display of a chat transcript that comprises a plurality of messages sent during the communication session between the first client device and the second client device;

orienting the presentation of the avatar associated with the user within the communication interface based on point-of-gaze indicator of the the mesh representation;

transcribing the audio data to a text string;

translating the text string from a first language to a second language, the second language based on a language preference associated with the second client device; and

presenting the text string within the communication interface at a position based on the presentation of the avatar associated with the user.

2. The method of claim 1 , wherein the image data comprises RGB color values, and the generating the mesh representation of the face of the user based on the image data includes:

parsing RGB color values from the image data; and

generating the mesh representation of the face based on the RGB color values.

3. The method of claim 1 , wherein the image data comprises corneal reflection data that indicates the point of gaze of the user.

4. The method of claim 1 , wherein the selecting the avatar features further comprises:

identifying a user account associated with the first client device; and

associating the set of avatar features with the user account.

5. The method of claim 1 , wherein the avatar comprises a set of avatar features, and wherein the method further comprises:

receiving a selection of the set of avatar features from a collection of avatar features; and

associating the selection of the set of avatar features with a user account associated with the user.

6. The method of claim 1 , wherein the translating the text string from the first language to the second language includes:

identifying the first language based on the user account associated with the first client device.

7. The method of claim 1 , wherein the presentation of the avatar is a first presentation of a first avatar, and wherein the method further comprises:

receiving a chat request to join the communication session with the first client device and the second client device from a third client device;

causing display of a second presentation of a second avatar that represents the third user within the communication interface at the second client device in response to the receiving the chat request; and

adjusting a size of the first presentation of the first avatar and the second presentation of the second avatar in response to the causing display of the second presentation of the second avatar within the communication interface at the second client device.

8. The method of claim 1 , wherein the image data is captured by a camera of the first client device, and the method further comprises:

detecting a deactivation of the camera at the first client device; and

removing the avatar associated with the user in response to the detecting the deactivation of the camera.

9. The method of claim 1 , wherein the method further comprises:

comparing the mesh representation of the face of the user to a reference mesh, the reference mesh depicting a facial expression; and

wherein the animating the presentation of the avatar is based on the mesh representation and the facial expression.

10. The method of claim 1 , wherein the presentation of the avatar associated with the user is at a location within the communication interface, and wherein the method further comprises:

detecting movement of the face of the user based on the image data;

dynamically changing the location of the presentation of the avatar based on the movement of the face of the user.

11. The method of claim 1 , wherein the mesh representation of the face of the user depicts a facial expression of the user.

12. The method of claim 1 , wherein the animating the presentation of the avatar associated with the user based on the mesh representation of the face of the user includes:

authenticating the user based on the image data; and

animating the presentation of the avatar in response to the authenticating the user based on the image data.

13. A system comprising:

a memory; and

at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising:

detecting an initiation of a communication session at a first client device, the communication session between the first client device and a second client device;

causing the first client device to capture image data that depicts a face of a user of the first client device, and audio data in response to the detecting the initiation of the communication session;

generating a mesh representation of the face of the user based on the image data, the mesh representation including a point-of-gaze indicator;

selecting a set of avatar features from among a collection of avatar features based on a user account associated with the first client device, the user account including an identification of the set of avatar features;

generating an avatar to be associated with the user of the first client device based on the set of avatar features and the mesh representation;

causing display of a communication interface at the second client device in response to the initiation of the communication session, the communication interface comprising a presentation of the avatar associated with the user of the first client device and a display of a chat transcript that comprises a plurality of messages sent during the communication session between the first client device and the second client device;

orienting the presentation of the avatar associated with the user within the communication interface based on point-of-gaze indicator of the the mesh representation;

transcribing the audio data to a text string;

translating the text string from a first language to a second language, the second language based on a language preference associated with the second client device; and

presenting the text string within the communication interface at a position based on the presentation of the avatar associated with the user.

14. The system of claim 13 , wherein the image data comprises RGB color values, and the generating the mesh representation of the face of the user based on the image data includes:

parsing RGB color values from the image data; and

generating the mesh representation of the face based on the RGB color values.

15. The system of claim 13 , wherein the image data comprises corneal reflection data that indicates the point of gaze of the user.

16. The system of claim 13 , wherein the selecting the set of avatar features further comprises:

identifying a user account associated with the first client device; and

associating the set of avatar features with the user account.

17. The system of claim 13 , wherein the avatar comprises a set of avatar features, and wherein the operations cause the system to perform operations further comprising:

receiving a selection of the set of avatar features from a collection of avatar features; and

associating the selection of the set of avatar features with a user account associated with the user.

18. The system of claim 13 , wherein the translating the text string from the first language to the second language includes:

identifying the first language based on the user account associated with the first client device.

19. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

detecting an initiation of a communication session at a first client device, the communication session between the first client device and a second client device;

causing the first client device to capture image data that depicts a face of a user of the first client device, and audio data in response to the detecting the initiation of the communication session;

generating a mesh representation of the face of the user based on the image data, the mesh representation including a point-of-gaze indicator;

selecting a set of avatar features from among a collection of avatar features based on a user account associated with the first client device, the user account including an identification of the set of avatar features;

generating an avatar to be associated with the user of the first client device based on the set of avatar features and the mesh representation;

causing display of a communication interface at the second client device in response to the initiation of the communication session, the communication interface comprising a presentation of the avatar associated with the user of the first client device and a display of a chat transcript that comprises a plurality of messages sent during the communication session between the first client device and the second client device;

orienting the presentation of the avatar associated with the user within the communication interface based on point-of-gaze indicator of the the mesh representation;

transcribing the audio data to a text string;

translating the text string from a first language to a second language, the second language based on a language preference associated with the second client device; and

presenting the text string within the communication interface at a position based on the presentation of the avatar associated with the user.

Assignments (1)

Continuity (1)

Cited By (88)

US 1,089,291 US 12,192,617 US 12,198,398 US 12,212,614 US 12,223,612 US 12,223,672 US 12,229,901 US 12,235,991 US 12,236,512 US 12,254,577 US 12,271,536 US 12,277,632 US 12,284,146 US 12,284,698 US 12,287,913 US 12,288,273 US 12,293,433 US 12,299,775 US 12,307,564 US 12,314,553 US 12,315,495 US 12,321,577 US 12,340,453 US 12,340,481 US 12,361,934 US 12,379,834 US 12,387,444 US 12,394,077 US 12,394,154 US 12,395,456 US 12,412,347 US 12,417,562 US 12,418,504 US 12,422,977 US 12,429,953 US 12,436,598 US 12,469,273 US 12,472,435 US 12,475,621 US 12,475,658 US 12,482,161 US 12,499,483 US 12,499,638 US 12,504,866 US 12,513,098 US 12,517,626 US 12,518,437 US 12,518,738 US 12,530,847 US 12,530,852 US 12,536,751 US 12,541,930 US 12,548,267 US 12,555,274 US 12,555,310 US 12,567,102 US 12,579,204 US 12,580,784 US 12,580,980 US 12,586,562 US 12,602,842 US 12,614,354 US 12,614,359 US 12,619,345 US 12,620,188 US 12,620,216 US 12,632,890 US 12,633,073 US 12,638,913 US 12,646,266 US 12,646,268 US 12,647,673 US 12,651,292 US 12,651,409 US 12,670,674 US 12,682,508 US 12,695,929 US 12,699,802 US 12,700,187 US 12,701,334 US 12,705,875 US 12,711,538 US 12,713,125 US 12,725,332 US 12,731,342 US 12,731,343 US 12,737,973 US 12,744,990