Automatic analysis of rapport
In selected embodiments, one or more wearable mobile devices provide videos and other sensor data of one or more participants in an interaction, such as a customer service or a sales interaction between a company employee and a customer. A computerized system uses machine learning expression classifiers, temporal filters, and a machine learning function approximator to estimate the quality of the interaction. The computerized system may include a recommendation selector configured to select suggestions for improving the current interaction and/or future interactions, based on the quality estimates and the weights of the machine learning approximator.
1. A computer system for estimating quality of an interaction between a first participant and a second participant, the system comprising:
a first plurality of machine learning classifiers of extended facial expressions, each classifier of the first plurality of classifiers being configured to generate a stream of first estimates of the degree to which a predetermined emotion or affective state corresponding to said each classifier of the first plurality of classifiers is present in a data stream of the first participant;
a second plurality of machine learning classifiers of extended facial expressions, each classifier of the second plurality of classifiers being configured to generate a stream of second estimates of the degree to which a predetermined emotion or affective state corresponding to said each classifier of the second plurality of classifiers is present in a data stream of the second participant, wherein the first and second data streams are synchronized;
a first plurality of temporal filters, each filter of the first plurality of temporal filters comprising an input connected to receive output of an associated classifier of the first plurality of classifiers, and an output;
a second plurality of temporal filters, each filter of the second plurality of temporal filters comprising an input connected to receive output of an associated classifier of the second plurality of classifiers, and an output;
a plurality of correlators configured to receive output signals from the first and second pluralities of temporal filters and to identify correlation patterns in the output signals of the first and second pluralities of temporal filters; and
a function approximator configured to receive at least some of output signals of the plurality of correlators, the output signals of the first plurality of temporal filters, and the output signals of the second plurality of temporal filters, the function approximator being machine trained to generate one or more estimates of quality of the interaction between the first participant and the second participant based on at least some of the output signals of the plurality of correlators, the output signals of the first plurality of temporal filters, and the output signals of the second plurality of temporal filters; and
a recommendation selector coupled to the function approximator to receive from the function approximator the one or more estimates and machine learning weights of the function approximator, the recommendation selector being configured to generate one or more suggestions regarding the interaction.
2. A computer system as in claim 1 , wherein the data stream of the first participant comprises a first video of extended facial expressions.
3. A computer system as in claim 2 , wherein the first plurality of machine learning classifiers, the second plurality of machine learning classifiers, the first plurality of temporal filters, the second plurality of temporal filters, the plurality of correlators, the machine learning function approximator, and the recommendation selector are configured to provide the one or more suggestions in real time.
4. A computer system as in claim 2 , wherein the computer implemented system is implemented using at least one wearable device.
5. A computer system as in claim 2 , wherein the computer implemented system is implemented using at least one set of glasses.
6. A computer system as in claim 2 , wherein the computer system is configured to receive at least one data stream of the data stream of the first participant and the data stream of the second participant from a wearable device.
7. A computer system as in claim 2 , wherein the computer system is configured to receive the data stream of the first participant from a second wearable device of the second participant, and to receive the data stream of the second participant from a first wearable device of the first participant.
8. A computer system as in claim 2 , wherein:
the data stream of the first participant comprises first video and first non-visual sensor data; and
the data stream of the second participant comprises second video and second non-visual sensor data.
9. A computer-implemented method for estimating quality of an interaction between a first participant and a second participant, comprising steps of:
processing a data stream of the first participant with a first plurality of machine learning classifiers of extended facial expressions, each classifier of the first plurality of classifiers being configured to generate a stream of first estimates of the degree to which a predetermined emotion or affective state corresponding to said each classifier of the first plurality of classifiers is present in the data stream of the first participant;
processing a data stream of the second participant with a second plurality of machine learning classifiers of extended facial expressions, each classifier of the second plurality of classifiers being configured to generate a stream of second estimates of the degree to which a predetermined emotion or affective state corresponding to said each classifier of the second plurality of classifiers is present in the data stream of the second participant, wherein the first and second data streams are synchronized;
processing signals at outputs of the first plurality of classifiers with a first plurality of temporal filters, each filter of the first plurality of temporal filters comprising an input connected to receive output of an associated classifier of the first plurality of classifiers, and a temporal filter output;
processing signals at outputs of the second plurality of classifiers with a second plurality of temporal filters, each filter of the second plurality of temporal filters comprising an input connected to receive output of an associated classifier of the second plurality of classifiers, and a temporal filter output;
correlating output signals from the first and second pluralities of temporal filters to identify correlation patterns in the output signals of the first and second pluralities of temporal filters, thereby obtaining a plurality of correlator output signals; and
generating with a machine learning function approximator one or more estimates of quality of the interaction between the first participant and the second participant based on at least some of the plurality of correlator output signals, the temporal filter outputs of the first plurality of temporal filters, and the temporal filter outputs of the second plurality of temporal filters, wherein the method is performed using at least one wearable device and further comprising:
selecting one or more suggestions for improving the interaction based on the one or more estimates of quality of the interaction and machine learning weights of the function approximator; and
providing the one or more suggestions to at least one of the first participant and the second participant.