Granted Patent B2

US 12,711,451 · App. 18/541,970 · Granted Aug 18, 2026

Inventors: Victor Shaburov (Castro Valley, CA); Yurii Monastyrshyn (Santa Monica, CA)

Assignee: SNAP INC.

G06Q10/06395G06Q10/06393G06V40/174H04N21/440218G10L2015/227G10L17/26G10L25/63H04N7/15H04N21/44218H04N21/4788

Abstract

Claims (63)

1 . A computer-implemented method for workforce analytics, the method comprising:

receiving an audio stream comprising a conversation between at least two individuals;

detecting a voice feature comprising a speaking rate of a first individual of the at least two individuals;

determining that the voice feature comprising the speaking rate of the first individual corresponds to a speech emotion of a plurality of speech emotions;

detecting the first individual in one or more frames of a video stream;

dynamically determining, over the one or more frames, a deformation of a virtual face mesh of the first individual;

locating feature reference points of the first individual;

aligning the virtual face mesh to the first individual in one or more of the frames based at least in part on the feature reference points, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual;

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics;

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes;

determining that the deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

evaluating an emotional status for the first individual based on the at least one facial emotion and the speech emotion; and

generating a work quality parameter for the first individual based on the emotional status.

2 . The method of claim 1 , further comprising establishing a video conference between the individual and a customer.

3 . The method of claim 1 , further comprising:

generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

4 . The method of claim 3 , further comprising recording the quality metrics of the individual in an employee record, wherein each of the quality metrics is time-stamped.

5 . The method of claim 3 , further comprising aggregating the quality metrics associated with the individual over a predetermined period to produce a work performance characteristic of the individual.

6 . The method of claim 3 , wherein the at least one work quality parameter includes a tiredness characteristic of the individual.

7 . The method of claim 3 , wherein the at least one work quality parameter includes a negative emotion characteristic of the individual.

8 . The method of claim 3 , wherein the at least one work quality parameter includes a positive emotion characteristic of the individual.

9 . The method of claim 3 , wherein the at least one work quality parameter includes a smile characteristic of the individual.

10 . The method of claim 1 , wherein the determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions includes:

comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and

selecting the facial emotion based on a comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions.

11 . The method of claim 10 , wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a convolution neural network.

12 . The method of claim 10 , wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a state vector machine.

13 . The method of claim 1 , wherein the feature reference points include facial landmarks.

14 . The method of claim 1 , wherein the detecting of the individual includes applying a Viola-Jones algorithm to images associated with the individual.

15 . The method of claim 1 , wherein the locating of the feature reference points includes applying an Active Shape Model algorithm to images associated with the individual.

16 . The method of claim 1 , wherein the plurality of facial emotions include at least one of: a neutral facial emotion, a positive facial emotion, or a negative facial emotion;

wherein the positive facial emotion includes at least one of happiness, gratitude, kindness, or enthusiasm; and

wherein the negative facial emotion includes at least one of anger, stress, depression, frustration, embarrassment, irritation, sadness, indifference, confusion, or annoyance.

17 . A system, comprising:

a computing device including at least one processor and a memory storing processor-executable codes, which, when implemented by the at least one processor, cause to perform operations comprising at least:

receiving an audio stream comprising a conversation between at least two individuals;

detecting a voice feature comprising a speaking rate of a first individual of the at least two individuals;

determining that the voice feature comprising the speaking rate of the first individual corresponds to a speech emotion of a plurality of speech emotions;

detecting the first individual in one or more frames of a video stream;

dynamically determining, over the one or more frames, a deformation of a virtual face mesh of the first individual;

locating feature reference points of the first individual;

aligning the virtual face mesh to the first individual in one or more of the frames based at least in part on the feature reference points, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual;

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics;

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes;

determining that the deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

evaluating an emotional status for the first individual based on the at least one facial emotion and the speech emotion; and

generating a work quality parameter for the first individual based on the emotional status.

18 . A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method, comprising:

receiving an audio stream comprising a conversation between at least two individuals;

detecting a voice feature comprising a speaking rate of a first individual of the at least two individuals;

determining that the voice feature comprising the speaking rate of the first individual corresponds to a speech emotion of a plurality of speech emotions;

detecting the first individual in one or more frames of a video stream;

dynamically determining, over the one or more frames, a deformation of a virtual face mesh of the first individual;

locating feature reference points of the first individual;

aligning the virtual face mesh to the first individual in one or more of the frames based at least in part on the feature reference points, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual;

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics;

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes;

determining that the deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

evaluating an emotional status for the first individual based on the at least one facial emotion and the speech emotion; and

generating a work quality parameter for the first individual based on the emotional status.

Assignments (4)

Continuity (4)

References Cited (171)