Real-time summarization of virtual conference transcripts

1 . A method comprising:

receiving, during a virtual conference from a client device, a request to generate a real-time summary of the virtual conference, a plurality of client devices participating in the virtual conference and exchanging audio and video streams;

receiving, during the virtual conference, audio streams from the plurality of client devices participating the virtual conference;

executing, during the virtual conference, an automatic speech recognition (“ASR”) model on the audio streams to generate a plurality of utterances from one or more participants;

generating a group of consecutive utterances from the plurality of utterances;

executing a trained machine learning (“ML”) model to generate a content segment based on the group of consecutive utterances and a segmentation threshold;

receiving an indication that the content segment was not generated;

determining that the group of consecutive utterances satisfies a chunk size threshold;

reducing the segmentation threshold to obtain a reduced segmentation threshold;

executing the trained ML model to generate the content segment based on the group of consecutive utterances and the reduced segmentation threshold;

executing a large language model (“LLM”) to generate a segment summary based on the content segment; and

providing the segment summary to the client device during the virtual conference.

2 . The method of claim 1 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, further comprising:

receiving a second plurality of utterances generated by the ASR model of the audio streams;

generating a second group of consecutive utterances from the second plurality of utterances and the remainder;

determining a second segment based on the second group of consecutive utterances;

generating, using the LLM, a second segment summary based on the second segment; and

providing the second segment summary to the client device.

3 . The method of claim 1 , further comprising:

determining that the group of consecutive utterances does not satisfy the chunk size threshold;

receiving a second plurality of utterances generated by the ASR model of the audio streams;

generating a second group of consecutive utterances from the second plurality of utterances and the group of consecutive utterances;

determining a second segment based on the second group of consecutive utterances;

generating, using the LLM, a second segment summary based on the second segment; and

providing the second segment summary to the client device.

4 . The method of claim 1 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, further comprising:

determining that the remainder satisfies the chunk size threshold;

generating a second segment based on the remainder;

generating, using the LLM, a second segment summary based on the second segment; and

providing the second segment summary to the client device.

5 . The method of claim 1 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, further comprising:

in response to determining that a virtual conference has concluded: determining that the remainder does not satisfy the chunk size threshold; and

adding the remainder to the content segment.

6 . The method of claim 1 , wherein the chunk size threshold comprises a minimum elapsed time, a maximum elapsed time, a minimum number of utterances, a maximum number of utterances, a minimum size for the group of consecutive utterances, or a maximum size for the group of consecutive utterances.

7 . A system comprising:

a communications interface;

a non-transitory computer-readable medium; and

one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:

receive, during a virtual conference from a client device, a request to generate a real-time summary of the virtual conference, a plurality of client devices participating in the virtual conference and exchanging audio and video streams;

receive, during the virtual conference, audio streams from the plurality of client devices participating the virtual conference;

execute, during the virtual conference, an automatic speech recognition (“ASR”) model on the audio streams to generate a plurality of utterances from one or more participants;

generate a group of consecutive utterances from the plurality of utterances;

executing a trained machine learning (“ML”) model to determine a content segment based on the group of consecutive utterances and a segmentation threshold;

receive an indication that the content segment was not generated;

determine that the group of consecutive utterances satisfies a chunk size threshold;

reduce the segmentation threshold to obtain a reduced segmentation threshold;

execute the trained ML model to generate the content segment based on the group of consecutive utterances and the reduced segmentation threshold;

generate, using a large language model (“LLM”), a segment summary based on the content segment; and

provide the segment summary to the client device during the virtual conference.

8 . The system of claim 7 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

receive a second plurality of utterances generated by the ASR model of the audio streams;

generate a second group of consecutive utterances from the second plurality of utterances and the remainder;

determine a second segment based on the second group of consecutive utterances;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

9 . The system of claim 7 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

determining that the group of consecutive utterances does not satisfy the chunk size threshold;

receiving a second plurality of utterances generated by the ASR model of the audio streams;

generating a second group of consecutive utterances from the second plurality of utterances and the group of consecutive utterances;

determining a second segment based on the second group of consecutive utterances;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

10 . The system of claim 7 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

determine that the remainder satisfies the chunk size threshold;

generate a second segment based on the remainder;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

11 . The system of claim 7 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

in response to determining that a virtual conference has concluded:

determine that the remainder does not satisfy the chunk size threshold; and

add the remainder to the content segment.

12 . The system of claim 7 , wherein the chunk size threshold comprises a minimum elapsed time, a maximum elapsed time, a minimum number of utterances, a maximum number of utterances, a minimum size for the group of consecutive utterances, or a maximum size for the group of consecutive utterances.

13 . A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to:

receive, during a virtual conference from a client device, a request to generate a real-time summary of the virtual conference, a plurality of client devices participating in the virtual conference and exchanging audio and video streams;

receive, during the virtual conference, audio streams from the plurality of client devices participating the virtual conference;

execute, during the virtual conference, an automatic speech recognition (“ASR”) model on the audio streams to generate a plurality of utterances from one or more participants;

generate a group of consecutive utterances from the plurality of utterances;

executing a trained machine learning (“ML”) model to determine a content segment based on the group of consecutive utterances and a segmentation threshold;

receive an indication that the content segment was not generated;

determine that the group of consecutive utterances satisfies a chunk size threshold;

reduce the segmentation threshold to obtain a reduced segmentation threshold;

execute the trained ML model to generate the content segment based on the group of consecutive utterances and the reduced segmentation threshold;

generate, using a large language model (“LLM”), a segment summary based on the content segment; and

provide the segment summary to the client device during the virtual conference.

14 . The non-transitory computer-readable medium of claim 13 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and further comprising processor-executable instructions configured to cause the one or more processors to:

receive a second plurality of utterances generated by the ASR model of the audio streams;

generate a second group of consecutive utterances from the second plurality of utterances and the remainder;

determine a second segment based on the second group of consecutive utterances;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

15 . The non-transitory computer-readable medium of claim 13 , further comprising

processor-executable instructions configured to cause the one or more processors to:

determining that the group of consecutive utterances does not satisfy the chunk size threshold;

receiving a second plurality of utterances generated by the ASR model of the audio streams;

generating a second group of consecutive utterances from the second plurality of utterances and the group of consecutive utterances;

determining a second segment based on the second group of consecutive utterances;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

16 . The non-transitory computer-readable medium of claim 13 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and further comprising processor-executable instructions configured to cause the one or more processors to:

determine that the remainder satisfies the chunk size threshold;

generate a second segment based on the remainder;

generate, using the LLM, a second segment summary based on the second segment; and

provide the second segment summary to the client device.

17 . The non-transitory computer-readable medium of claim 13 , wherein the content segment comprises a subset of the group of consecutive utterances, wherein the subset comprises less than all of the utterances in the group of consecutive utterances and a remainder comprises the utterances from the group of consecutive utterances not included in the subset, and further comprising processor-executable instructions configured to cause the one or more processors to:

in response to determining that a virtual conference has concluded:

determine that the remainder does not satisfy the chunk size threshold; and

add the remainder to the content segment.