Highlighting expressive participants in an online meeting
The present disclosure relate to highlighting audience members with reactions to a presenter of an online meeting. Unlike physical, fact-to-face meeting that enables spontaneous interactions among the presenter and the audiences that are collocated with the presenter, presenting materials during an online meeting raises an issue of the present not being able to see real-time reactions or feedback by the audience members. The present disclosure addresses the issue by dynamically determining one or more audience members who indicate reactions during the online meeting or presentation and displaying faces of the one or more audience members under spotlight to the presenter. The presenter sees faces of the audience members with reactions during the online presentation and responds to the audience members and keep the audience engaged. The spotlight audience server analyzes video frames and determines types of reactions of the audience members.
1. A computer-implemented method comprising:
receiving data of a video stream, the data associated with a plurality of audience members participating in an online presentation;
deriving, from the received data, a plurality of features-associated with a reaction made by one or more of the plurality of audience members, wherein the deriving a plurality of features comprises:
determining, via an image processing neural network, a head yaw rotation value associated with a head gesture in the video stream; and
inputting the head yaw rotation value into a yaw-detection model to determine the reaction comprises a head shake gesture;
generating at least one expressiveness score based on the reaction made by the one or more of the plurality of audience members, the expressiveness score being generated using a combination of the plurality of features, with one or more features of the plurality of features that are pre-defined as being less-preferred for their conveying of negative engagement by the one or more of the plurality of audience members being weighted less than one or more other features of the plurality of features that are pre-defined as being more-preferred for their conveying of positive engagement by the one or more of the plurality of audience members;
selecting, based on the at least one expressiveness score, a member of the plurality of audience members to highlight; and
transmitting information associated with the selected member of the plurality of audience members to cause an online meeting application to highlight a display of the selected member of the plurality of audience members to a presenter of the online meeting.
2. The computer-implemented method of claim 1 , wherein the plurality of features associated with the reaction made by the one or more of the plurality of audience members comprises one or more of smiles, downturned mouth, mouth open, and closed eyes.
3. The computer-implemented method of claim 1 , the method further comprising:
causing the online meeting application to periodically update, based on a predetermined time interval, the display of the selected member of the plurality of audience members.
4. The computer-implemented method of claim 1 , the method further comprising:
determining a first probabilistic score associated with a facial expression associated with a reaction made by an audience member of the plurality of audience members through classification;
determining a second probabilistic score associated with a furrowed brow by the audience member;
determining a third probabilistic score associated with the head shake gesture; and
generating, based at least on a combination of the first probabilistic score, the second probabilistic score, and the third probabilistic score, an expressiveness score associated with the audience member.
5. The computer-implemented method of claim 4 , wherein the received data associated with the audience member includes a video frame, and wherein determining the first probabilistic score associated with the facial expression uses the image processing neural network with one or regions of interests in the video frame as input.
6. The computer-implemented method of claim 5 , wherein determining the third probabilistic score associated with the head shake gesture uses the yaw-detection model, and wherein the yaw-detection model is a Hidden Markov Model (HMM).
7. The computer-implemented method of claim 6 , further comprising:
determining, by the image processing neural network, a Y-position of at least one facial landmark; and
determining, by the yaw detection model, using the Y-position and the head yaw rotation value, a head nod over time.
8. The computer-implemented method of claim 6 , wherein the second probabilistic score associated with a furrowed brow corresponds to a degree of confusion shown by the audience member.
9. The computer-implemented method of claim 1 , wherein highlighting the display of the selected member of the plurality of audience members comprises displaying live video data including placing a spotlight on the selected member.
10. A system for displaying reactive audience under spotlight in an online meeting, the system comprising:
a processor; and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
receive data associated with a plurality of audience members participating in an online presentation;
derive, from the received data, a plurality of features associated with a reaction made by one or more of the plurality of audience members, wherein the deriving a plurality of features comprises:
determining, via an image processing neural network, a head yaw rotation value associated with a head gesture in the video stream; and
inputting the head yaw rotation value into a yaw-detecting model to determine the reaction comprises a head shake gesture;
generate at least one expressiveness score based on the reaction made by the one or more of the plurality of audience members, the expressiveness score being generated using a combination of the plurality of features, with one or more features of the plurality of features that are pre-defined as being less-preferred for their conveying of negative engagement by the one or more of the plurality of audience members being weighted less than one or more other features of the plurality of features that are pre-defined as being more-preferred for their conveying of positive engagement by the one or more of the plurality of audience members;
select, based on the at least one expressiveness score, a member of the plurality of audience members to highlight; and
transmit information associated with the selected member of the plurality of audience members to cause an online meeting application to highlight a display of the selected member of the plurality of audience members to a presenter of the online meeting.
11. The system of claim 10 , wherein the plurality of features associated with the reaction made by the one or more of the plurality of audience members comprises one or more of smiles, downturned mouth, and brow furrow.
12. The system of claim 10 , the computer-executable instructions when executed further cause the system to:
cause the online meeting application to periodically update, based on a predetermined time interval, the display of the selected member of the plurality of audience members.
13. The system of claim 10 , the computer-executable instructions when executed further cause the system to:
determine a first probabilistic score associated with a facial expression associated with a reaction made by an audience member of the plurality of audience members through classification;
determine a second probabilistic score associated with a furrowed brow by the audience member;
determine a third probabilistic score associated with the head shake gesture; and
generate, based at least on a combination of the first probabilistic score, the second probabilistic score; and the third probabilistic score, an expressiveness score associated with the audience member.
14. The system of claim 13 , wherein the received data associated with the audience member includes a video frame, and wherein determining the first probabilistic score associated with the facial expression uses the image processing neural network with one or regions of interests in the video frame as input.
15. The system of claim 14 , wherein determining the third probabilistic score associated with the head shake gesture uses the yaw-detection model, and wherein the yaw-detection model is a Hidden Markov Model (HMM).
16. A non-transitory computer readable storage media storing computer-executable instructions that when executed by a processor cause a computer system to:
receive data associated with a plurality of audience members participating in an online presentation;
derive, from the received data, a plurality of features associated with a reaction made by one or more of the plurality of audience members, wherein the deriving a plurality of features comprises:
determining, via an image processing neural network, a head yaw rotation value associated with a head gesture in the video stream; and
inputting the head yaw rotation value into a yaw-detecting model to determine the reaction comprises a head shake gesture;
generate at least one expressiveness score based on the reaction made by the one or more of the plurality of audience members, the expressiveness score being generated using a combination of the plurality of features, with one or more features of the plurality of features that are pre-defined as being less-preferred for their conveying of negative engagement by the one or more of the plurality of audience members being weighted less than one or more other features of the plurality of features that are pre-defined as being more-preferred for their conveying of positive engagement by the one or more of the plurality of audience members;
select, based on the at least one expressiveness score, a member of the plurality of audience members to highlight; and
transmit information associated with the selected member of the plurality of audience members to cause an online meeting application to highlight a display of the selected member of the plurality of audience members to a presenter of the online meeting.
17. The computer storage media of claim 16 , wherein the plurality of features associated with the reaction made by the one or more of the plurality of audience members comprises one or more of smiles, downturned mouth, and brow furrow.
18. The computer storage media of claim 16 , the computer-executable instructions when executed further cause the system to:
cause the online meeting application to periodically update, based on a predetermined time interval, the display of the selected member of the plurality of audience members.
19. The computer storage media of claim 16 , the computer-executable instructions when executed further cause the system to:
determine a first probabilistic score associated with a facial expression associated with a reaction made by an audience member of the plurality of audience members through classification;
determine a second probabilistic score associated with a furrowed brow by the audience member;
determine a third probabilistic score associated with the head shake gesture; and
generate, based at least on a combination of the first probabilistic score, the second probabilistic score, and the third probabilistic score, an expressiveness score associated with the audience member.
20. The computer storage media of claim 19 ,
wherein determining the first probabilistic score associated with the facial expression uses the image processing neural network with one or more regions of interests in the video frame as input, and
wherein determining the third probabilistic score associated with the head shake gesture uses the yaw-detection model, and wherein the yaw-detection model is a Hidden Markov Model (HMM).