Machine-learning based method for assessing hybrid in-person and remote audience member engagement
Disclosed herein are systems and methods for ML-based assessment of a live presentation by a presenter and engagement of in-person and remote audience members. The method includes: obtaining a live presentation video stream, a live presenter audio stream, and presentation material; evaluating the live presentation using a prepared presentation evaluation MLM; obtaining, for each audience member, whether in-person or remote, participating in the live presentation as an in-person audience member or a remote audience member, a plurality of video streams capturing each respective audience member, a plurality of audio streams for each respective audience member, and a plurality of capture streams capturing an interaction of each audience member and a respective computing device; evaluating an engagement of at least one audience member using a prepared engagement evaluation MLM; and generating a comprehension score based on the presentation score and the individual engagement score for at least one audience member.
1 . A method for machine learning (ML)-based assessment of a live presentation by a presenter and engagement of in-person and remote audience members, comprising:
obtaining a live presentation video stream of a live presentation, a live presenter audio stream of the live presentation, and presentation material, wherein the live presentation is associated with the presentation material;
evaluating the live presentation using a prepared presentation evaluation machine learning model (MLM) configured to generate a presentation score based on the live presentation video stream, the live presenter audio stream, and the presentation material;
obtaining, for each audience member participating in the live presentation as an in-person audience member or a remote audience member, a plurality of video streams capturing each respective audience member, a plurality of audio streams for each respective audience member, and a plurality of capture streams capturing an interaction of each audience member and a respective computing device;
evaluating an engagement of at least one audience member using a prepared engagement evaluation MLM configured to generate an individual engagement score for the at least one audience member based on the plurality of video streams, the plurality of audio streams, and the plurality of capture streams; and
generating, by a scoring engine, a comprehension score based at least in part on the presentation score and the individual engagement score for the at least one audience member.
2 . The method of claim 1 , wherein the prepared presentation evaluation MLM is further configured to generate a live presentation score based on individual engagement scores for in-person audience members and remote presentation score based on individual engagement scores for remote audience members.
3 . The method of claim 1 , wherein the prepared engagement evaluation MLM is configured to evaluate the individual engagement score for an audience member participating as an in-person audience member differently than an audience member participating as a remote audience member by assigning a first set of weights to the plurality of video streams, the plurality of audio streams, and the plurality of capture streams capturing each respective audience member participating as an in-person audience member and a second set of weights to the plurality of video streams, the plurality of audio streams, and the plurality of capture streams capturing each respective audience member participating as a remote audience member.
4 . The method of claim 1 , further comprising:
based on the comprehension score, an output of the prepared presentation evaluation MLM, and an output of the prepared engagement evaluation MLM, generating, by a recommendation engine, a presenter recommendation to engage in-person audience members, a presenter recommendation to engage remote audience members, an in-person audience member recommendation, or a remote audience member recommendation; and
displaying, on a display, at least one of the presenter recommendation to engage in-person audience members, the presenter recommendation to engage remote audience members, the in-person audience member recommendation, the remote audience member recommendation, the comprehension score, the presentation score, or the individual engagement score for the at least one audience member.
5 . The method of claim 4 , wherein the presenter recommendation to engage in-person audience members comprises at least one of: ask a question, introduce a poll or quiz, call on an audience member, identifying an audience member via audio stream, acknowledge contribution of an audience member, slow down, speed up, repeat material, summarize material, present a visual aid, or introduce a break,
wherein the presenter recommendation to engage remote audience members comprises at least one of: ask a question, introduce a poll or quiz, call on an audience member, acknowledge contribution of remote audience member, slow down, speed up, repeat material, summarize material, present a visual aid, check framing, monitor audio, avoid filter words, rephrase for clarity, emphasize key points, stay on time, or introduce an audio stream capturing audio from the in-person audience members,
wherein the in-person audience member recommendation comprises at least one of: take notes, ask questions, respond to prompts, use non-verbal feedback, participate in chat, adjust computing environment, or change physical posture, and
wherein the remote audience member recommendation comprises at least one of: take notes, ask questions, respond to prompts, use non-verbal feedback, participate in chat, adjust computing environment, change physical posture, pause lecture, or re-watch portion of lecture.
6 . The method of claim 1 , further comprising:
identifying a subject matter for the presentation and recognizing topics for the presentation using a topic MLM configured to determine main topics of the presentation based on the presentation material;
evaluating the presentation of each topic of the presentation using the prepared presentation evaluation MLM configured to generate a respective presentation score for each topic based on the subject matter, recognized topic, presentation video stream, the presenter audio stream, and the presentation material; and
evaluating the engagement of at least one audience member for each topic of the presentation using the prepared engagement evaluation MLM configured to generate a respective engagement score of the at least one audience member for each topic based on the subject matter, the recognized topic, the plurality of video streams, the plurality of audio streams, and the plurality of capture streams.
7 . The method of claim 1 , further comprising:
evaluating a group engagement of audience members for the presentation by generating, using the score engine, a group engagement score based on a plurality of engagement scores for each audience member participating as an in-person audience member or a remote audience member.
8 . The method of claim 6 , further comprising:
evaluating a group engagement of audience members for each topic of the presentation using the prepared engagement evaluation MLM configured to generate a respective group engagement score of audience members for each topic based on the subject matter, the recognized topic, the plurality of video streams, the plurality of audio streams, and the plurality of capture streams.
9 . The method of claim 1 , further comprising:
obtaining an audio stream of the in-person audience members, a pre-recorded presentation video stream of a pre-recorded presentation, a pre-recorded presenter audio stream of the pre-recorded presentation, wherein the live presentation is associated with the pre-recorded presentation; and
evaluating the pre-recorded presentation using the prepared evaluation MLM to further generate the presentation score based on the audio stream of the in-person audience members, the live presentation video stream, the live presenter audio stream, the pre-recorded presentation video stream, the pre-recorded presenter audio stream, and the presentation material.
10 . The method of claim 1 , further comprising preparing the presentation evaluation MLM by:
(1) providing, to the presentation evaluation MLM, a presentation training dataset comprising at least one of:
(a) presenter video streams of individuals giving a presentation labeled and annotated with quality scores for clarity, engagement, accuracy, or content delivery to evaluate at least one of body language, gestures, facial expressions, or visual aids of the presenter,
(b) corresponding audio streams labeled and annotated for features of at least one of voice quality, pauses, tone, or fluency to analyze speech delivery, clarity, tone, or impact of the presenter,
(c) presentation material aligned with the labeled presenter video streams labeled and annotated to assess at least one of content quality, relevance, or alignment between the presentation and the presentation materials, and
(d) ground truth labels for the presenter video streams, the corresponding audio streams, or the presentation material to serve as a target output for the presentation evaluation MLM model; and
(2) preparing the presentation evaluation MLM using the presentation training dataset.
11 . The method of claim 1 , further comprising preparing the engagement evaluation MLM by:
(1) providing, to the engagement evaluation MLM, an engagement training dataset comprising at least one of:
(a) individual video streams of audience members of a presentation labeled and annotated with facial expressions, gaze direction, body posture, or head movement to capture visual engagement cues,
(b) corresponding audio streams labeled and annotated with speech activity, tone analysis, interruptions and pauses, or emotion detection to analyze participation through voice contribution and tone of speech,
(c) capture streams of the audience members comprising at least one of keyboard input, mouse movements, or screen interactions, wherein the capture streams are labeled and annotated with active engagement, passive behavior, or multitasking to measure interaction of the audience members with a respective computing device,
(d) an event list comprises at least one of interaction events, participation events, attention-related events, engagement-related events, time-based events, or behavioral and biometric events annotated with engagement levels, activity frequency, or contribution quality to capture session-specific events that indicate audience member engagement, and
(e) ground truth labels for the individual video streams, the corresponding audio streams, the capture streams, or the event list to serve as a target output for the engagement evaluation MLM; and
(2) preparing the engagement evaluation MLM using the provided engagement training dataset.
12 . The method of claim 6 , further comprising preparing the topic MLM by:
(1) providing, to the topic MLM, a topic training dataset comprising at least one of:
(a) presentation materials labeled and annotated with subject matter, key topics, or keywords and phrases to extract and analyze a textual and visual content of the presentation to infer the subject matter and individual topics,
(b) topic labeled datasets comprising at least domain-specific data labeled and annotated with subject matter tags and topics to train the topic MLM to recognize subject matter and topics,
(c) audio and video presentation data comprising at least presentation recordings and corresponding audio streams labeled and annotated with subject matter and topics to identify spoken words, tone, and context for recognizing topics, and
(d) ground truth labels for the presentation materials, topic labeled datasets, or audio and video presentation data to serve as a target output for the topic MLM; and
(2) preparing the topic MLM using the provided topic training dataset.
13 . The method of claim 1 , further comprising:
generating, by a recommendation engine, and displaying, on a display, a post-presentation presenter recommendation or a post-presentation audience member recommendation based on outputs of the prepared presentation MLM and the prepared engagement evaluation MLM.
14 . A system for machine learning (ML)-based assessment of a live presentation by a presenter and engagement of in-person and remote audience members, comprising:
at least one memory; and
at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
obtain a live presentation video stream of a live presentation, a live presenter audio stream of the live presentation, and presentation material, wherein the live presentation is associated with the presentation material;
evaluate the live presentation using a prepared presentation evaluation machine learning model (MLM) configured to generate a presentation score based on the live presentation video stream, the live presenter audio stream, and the presentation material;
obtain, for each audience member participating in the live presentation as an in-person audience member or a remote audience member, a plurality of video streams capturing each respective audience member, a plurality of audio streams for each respective audience member, and a plurality of capture streams capturing an interaction of each audience member and a respective computing device;
evaluate an engagement of at least one audience member using a prepared engagement evaluation MLM configured to generate an individual engagement score for the at least one audience member based on the plurality of video streams, the plurality of audio streams, and the plurality of capture streams; and
generate, by a scoring engine, a comprehension score based at least in part on the presentation score and the individual engagement score for the at least one audience member.
15 . The system of claim 14 , wherein the prepared presentation evaluation MLM is further configured to generate a live presentation score based on individual engagement scores for in-person audience members and remote presentation score based on individual engagement scores for remote audience members.
16 . The system of claim 14 , wherein the prepared engagement evaluation MLM is configured to evaluate the individual engagement score for an audience member participating as an in-person audience member differently than an audience member participating as a remote audience member by assigning a first set of weights to the plurality of video streams, the plurality of audio streams, and the plurality of capture streams capturing each respective audience member participating as an in-person audience member and a second set of weights to the plurality of video streams, the plurality of audio streams, and the plurality of capture streams capturing each respective audience member participating as a remote audience member.
17 . The system of claim 14 , wherein the at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
based on the comprehension score, an output of the prepared presentation evaluation MLM, and an output of the prepared engagement evaluation MLM, generate, by a recommendation engine, a presenter recommendation to engage in-person audience members, a presenter recommendation to engage remote audience members, an in-person audience member recommendation, or a remote audience member recommendation; and
cause for display, on a display, at least one of the presenter recommendations to engage in-person audience members, the presenter recommendation to engage remote audience members, the in-person audience member recommendation, the remote audience member recommendation, the comprehension score, the presentation score, or the individual engagement score for the at least one audience member.
18 . The system of claim 17 , wherein the presenter recommendation to engage in-person audience members comprises at least one of: ask a question, introduce a poll or quiz, call on an audience member, identifying an audience member via audio stream, acknowledge contribution of an audience member, slow down, speed up, repeat material, summarize material, present a visual aid, or introduce a break,
wherein the presenter recommendation to engage remote audience members comprises at least one of: ask a question, introduce a poll or quiz, call on an audience member, acknowledge contribution of remote audience member, slow down, speed up, repeat material, summarize material, present a visual aid, check framing, monitor audio, avoid filter words, rephrase for clarity, emphasize key points, stay on time, or introduce an audio stream capturing audio from the in-person audience members,
wherein the in-person audience member recommendation comprises at least one of: take notes, ask questions, respond to prompts, use non-verbal feedback, participate in chat, adjust computing environment, or change physical posture, and
wherein the remote audience member recommendation comprises at least one of: take notes, ask questions, respond to prompts, use non-verbal feedback, participate in chat, adjust computing environment, change physical posture, pause lecture, or re-watch portion of lecture.
19 . The system of claim 14 , wherein the at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
(1) provide, to the presentation evaluation MLM, a presentation training dataset comprising at least one of:
(a) presenter video streams of presenters presenting a presentation labeled and annotated with quality scores for clarity, engagement, accuracy, or content delivery to evaluate at least one of body language, gestures, facial expressions, or visual aids of the presenter,
(b) corresponding audio streams labeled and annotated for features of at least one of voice quality, pauses, tone, or fluency to analyze speech delivery, clarity, tone, or impact of the presenter,
(c) presentation material aligned with the labeled presenter video streams labeled and annotated to assess at least one of content quality, relevance, or alignment between the presentation and the presentation materials, and
(d) ground truth labels for the presenter video streams, the corresponding audio streams, or the presentation material to serve as a target output for the presentation evaluation MLM model; and
(2) prepare the presentation evaluation MLM using the presentation training dataset.
20 . A non-transitory computer readable medium storing thereon computer executable instructions for machine learning (ML)-based assessment of a live presentation by a presenter and engagement of in-person and remote audience members, including instructions for:
obtaining a live presentation video stream of a live presentation, a live presenter audio stream of the live presentation, and presentation material, wherein the live presentation is associated with the presentation material;
evaluating the live presentation using a prepared presentation evaluation machine learning model (MLM) configured to generate a presentation score based on the live presentation video stream, the live presenter audio stream, and the presentation material;
obtaining, for each audience member participating in the live presentation as an in-person audience member or a remote audience member, a plurality of video streams capturing each respective audience member, a plurality of audio streams for each respective audience member, and a plurality of capture streams capturing an interaction of each audience member and a respective computing device;
evaluating an engagement of at least one audience member using a prepared engagement evaluation MLM configured to generate an individual engagement score for the at least one audience member based on the plurality of video streams, the plurality of audio streams, and the plurality of capture streams; and
generating, by a scoring engine, a comprehension score based at least in part on the presentation score and the individual engagement score for the at least one audience member.