Determining engagement using sensor data
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting engagement. One of the methods is a method of determining an engagement score for each of a plurality of participants in a video conference call, and includes, for each of the plurality of participants: receiving an encoded video stream of the participant, the encoded video stream representing an encoding of a sequence of one or more video frames; decoding the encoded video stream to determine the sequence of video frames; for each of a plurality of engagement categories representing respective different modes by which the participant may be engaged in the conference call, processing one or more respective video frames from the sequence of video frames to generate a sub-score for the engagement category; and combining the sub-scores for the plurality of engagement categories to generate the engagement score for the participant.
1 . A method of determining an engagement score for each of a plurality of participants in a video conference call, the engagement score for each participant representing a degree to which the participant is engaged in the video conference call, the method comprising, for each of the plurality of participants:
receiving, at a server system and from a user device of the participant, an encoded video stream of the participant, the encoded video stream representing an encoding of a sequence of one or more video frames;
decoding the encoded video stream to determine the sequence of video frames;
for each of a plurality of engagement categories representing respective different modes by which the participant may be engaged in the conference call, processing one or more respective video frames from the sequence of video frames to generate a sub-score for the engagement category; and
combining the sub-scores for the plurality of engagement categories to generate the engagement score for the participant, wherein the sub-scores comprise a valence value and an arousal value, wherein the valence value and the arousal value are integer values, and the valence value and the arousal value are combined according to a predetermined mapping between the valence value, the arousal value, and the sub-score for the engagement category.
2 . The method of claim 1 , further comprising, for each of the plurality of participants, generating a respective engagement score at each of a plurality of time points during the video conference call.
3 . The method of claim 2 , further comprising, for each of the plurality of participants and after the video conference call has concluded, combining the respective engagement scores corresponding to the plurality of times points to generate an overall engagement score representing an aggregate degree to which the participant was engaged throughout the video conference call.
4 . The method of claim 1 , further comprising:
combining the respective engagement scores for each of the plurality of participants to generate a collective engagement score representing a degree to which the plurality of participants are collectively engaged in the video conference call.
5 . The method of claim 4 , further comprising generating a respective collective engagement score at each of a plurality of time points during the video conference call.
6 . The method of claim 5 , further comprising, after the video conference call has concluded, combining the respective collective engagement scores corresponding to the plurality of time points to generate an overall collective engagement score representing an aggregate degree to which the plurality of participants were engaged throughout the video conference call.
7 . The method of claim 1 , further comprising, for each of the plurality of participants:
encoding (i) the sequence of video frames of the participant and (ii) the engagement score for the participant to generate an augmented encoded video stream; and
providing the augmented encoded video stream to respective user devices of the other participants of the plurality of participants.
8 . The method of claim 1 , wherein, for one or more of the plurality of engagement categories:
generating the sub-score for the engagement category further comprises processing one or more of (i) audio data corresponding to the sequence of video frames or (ii) interaction data representing a user input provided by the participant to the user device of the participant.
9 . The method of claim 1 , wherein the server system is configured to process the sequence of video frames using a single neural network to generate respective sub-scores for at least a subset of the plurality of engagement categories.
10 . The method of claim 9 , wherein the single neural network comprises:
a first subnetwork that has been configured through training to process the sequence of video frames and to generate an embedding of the sequence of video frames; and
a plurality of second subnetworks each corresponding to one or more respective different engagement categories, wherein each second subnetwork has been configured through training to process the embedding of the sequence of video frames and to generate the respective sub-scores for the one or more engagement categories corresponding to the second subnetwork.
11 . The method of claim 1 , further comprising, for each of the plurality of participants:
displaying, by the user device of the participant, one or more of (i) the engagement score of the participant or (ii) a collective engagement score representing a degree to which the plurality of participants are collectively engaged in the video conference call.
12 . The method of claim 1 , further comprising:
storing, by the server system, video data determined from the respective encoded video streams of the plurality of participants; and
associating the video data with engagement data determined from the respective engagement scores of the plurality of participants.
13 . The method of claim 1 , wherein the plurality of engagement categories comprises at least two of:
a facial expression category whose sub-score is generated according to an identified facial expression of the participant,
a head movement category whose sub-score is generated according to an identified head movement of the participant,
a verbal cue category whose sub-score is generated according to a verbalization made by the participant,
a gesture category whose sub-score is generated according to an identified gesture made by the participant,
an interaction category whose sub-score is generated according to a user input provided by the participant to the user device of the participant,
a presence category whose sub-score is generated according to a presence of the participant in the sequence of video frames, or
a gaze category whose sub-score is generated according to an identified gaze of the participant.
14 . The method of claim 1 , wherein the server system generates a respective engagement score for each of the plurality of participants in parallel.
15 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations comprising determining an engagement score for each of a plurality of participants in a video conference call, the engagement score for each participant representing a degree to which the participant is engaged in the video conference call, the determining comprising, for each of the plurality of participants:
receiving, at a server system and from a user device of the participant, an encoded video stream of the participant, the encoded video stream representing an encoding of a sequence of one or more video frames;
decoding the encoded video stream to determine the sequence of video frames;
for each of a plurality of engagement categories representing respective different modes by which the participant may be engaged in the conference call, processing one or more respective video frames from the sequence of video frames to generate a sub-score for the engagement category; and
combining the sub-scores for the plurality of engagement categories to generate the engagement score for the participant, wherein the sub-scores comprise a valence value and an arousal value, wherein the valence value and the arousal value are integer values, and the valence value and the arousal value are combined according to a predetermined mapping between the valence value, the arousal value, and the sub-score for the engagement category.
16 . The system of claim 15 , wherein the operations further comprise, for each of the plurality of participants:
encoding (i) the sequence of video frames of the participant and (ii) the engagement score for the participant to generate an augmented encoded video stream; and
providing the augmented encoded video stream to respective user devices of the other participants of the plurality of participants.
17 . The system of claim 15 , wherein the plurality of engagement categories comprises at least two of:
a facial expression category whose sub-score is generated according to an identified facial expression of the participant,
a head movement category whose sub-score is generated according to an identified head movement of the participant,
a verbal cue category whose sub-score is generated according to a verbalization made by the participant,
a gesture category whose sub-score is generated according to an identified gesture made by the participant,
an interaction category whose sub-score is generated according to a user input provided by the participant to the user device of the participant,
a presence category whose sub-score is generated according to a presence of the participant in the sequence of video frames, or
a gaze category whose sub-score is generated according to an identified gaze of the participant.
18 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations comprising determining an engagement score for each of a plurality of participants in a video conference call, the engagement score for each participant representing a degree to which the participant is engaged in the video conference call, the determining comprising, for each of the plurality of participants:
receiving, at a server system and from a user device of the participant, an encoded video stream of the participant, the encoded video stream representing an encoding of a sequence of one or more video frames;
decoding the encoded video stream to determine the sequence of video frames;
for each of a plurality of engagement categories representing respective different modes by which the participant may be engaged in the conference call, processing one or more respective video frames from the sequence of video frames to generate a sub-score for the engagement category; and
combining the sub-scores for the plurality of engagement categories to generate the engagement score for the participant, wherein the sub-scores comprise a valence value and an arousal value, wherein each of the sub-scores is weighted, further wherein at least two of the sub-scores have different weights, wherein each user has an individual weighting scheme for the sub-scores such that at least two of the users have different weighting schemes.
19 . The non-transitory computer storage media of claim 18 , wherein the operations further comprise, for each of the plurality of participants:
encoding (i) the sequence of video frames of the participant and (ii) the engagement score for the participant to generate an augmented encoded video stream; and
providing the augmented encoded video stream to respective user devices of the other participants of the plurality of participants.
20 . The non-transitory computer storage media of claim 18 , wherein the plurality of engagement categories comprises at least two of:
a facial expression category whose sub-score is generated according to an identified facial expression of the participant,
a head movement category whose sub-score is generated according to an identified head movement of the participant,
a verbal cue category whose sub-score is generated according to a verbalization made by the participant,
a gesture category whose sub-score is generated according to an identified gesture made by the participant,
an interaction category whose sub-score is generated according to a user input provided by the participant to the user device of the participant,
a presence category whose sub-score is generated according to a presence of the participant in the sequence of video frames, or
a gaze category whose sub-score is generated according to an identified gaze of the participant.