Techniques for intelligent video highlight summarization

1 . A method, comprising:

accessing, by a computing system, a video source comprising one or more video clips, each clip of the one or more video clips comprising a series of images, audio, and text elements associated with the series of images;

generating, by the computing system, a first set of embeddings based at least in part on the video source, the first set of embeddings comprising an aligned image portion, an audio portion, and a text portion;

receiving, by the computing system, a query providing guidance for extracting clips from the video source, the guidance comprising a first-part guidance and a second-part guidance, the first-part guidance comprising preferences for events, the second-part guidance comprising criteria, and the criteria comprising a duration budget;

generating, by the computing system, a second set of embeddings based at least in part on the query; and

performing, by a machine learning (ML) model of the computing system, a video analysis comprising:

dividing the first set of embeddings into a plurality of sections;

determining a similarity score between each section of the plurality of sections and the second set of embeddings; and

generating a summary of the one or more video clips using the first set of embeddings and the second set of embeddings, the summary including a subset of the one or more video clips being selected based at least in part on the first-part guidance and the second-part guidance.

2 . The method of claim 1 , wherein the first set of embeddings are generated by:

encoding the series of images, text elements associated with the images into video embeddings;

encoding the audio of the video source into audio embeddings; and

aligning the video embeddings and the audio embedding.

3 . The method of claim 2 , wherein the text elements associated with the images are captions.

4 . The method of claim 1 , wherein the preferences comprise events of interest in the video source to a user, and the events comprise temporal events and spatiotemporal events.

5 . The method of claim 1 , wherein generating the second set of embeddings comprises combining encoded embeddings of the query and encoded embeddings of the text elements associated with the images.

6 . The method of claim 1 , wherein the ML model is an autoregressive decoder transformer.

7 . The method of claim 1 , wherein performing the video analysis to select the subset of the one or more video clips comprises:

performing a similarity search using the second set of embeddings against the first set of embeddings;

generating similarity scores for the one or more video clips based at least in part on the first-part guidance; and

selecting the subset of the one or more video clips based at least in part on the generated similarity scores and the second-part guidance.

8 . The method of claim 7 , wherein the similarity search is cosine similarity search.

9 . The method of claim 7 , wherein selecting the subset of the one or more video clips is performed by maximizing summation of the similarity scores of the selected subset.

10 . The method of claim 7 , further comprising distributing the series of images in a video clip to have gradual increase and decrease of individual similarity score of each image.

11 . A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:

accessing, by the computing system, a video source comprising one or more video clips, each clip of the one or more video clips comprising a series of images, audio, and text elements associated with the series of images;

generating, by the computing system, a first set of embeddings based at least in part on the video source, the first set of embeddings comprising an aligned image portion, an audio portion and a text portion;

receiving, by the computing system, a query providing guidance for extracting clips from the video source, the guidance comprising a first-part guidance and a second-part guidance, the first-part guidance comprising preferences for events, and the second-part guidance comprising criteria, and the criteria comprising a duration budget;

generating, by the computing system, a second set of embeddings based at least in part on the query; and

performing, by a machine learning (ML) model of the computing system, a video analysis comprising:

dividing the first set of embeddings into a plurality of sections;

determining a similarity score between each section of the plurality of sections and the second set of embeddings; and

generating a summary of the one or more video clips using the first set of embeddings and the second set of embeddings, the summary including a subset of the one or more video clips being selected based at least in part on the first-part guidance and the second-part guidance.

12 . The non-transitory computer-readable medium of claim 11 , wherein the first set of embeddings are generated by:

encoding the series of images, text elements associated with the images into video embeddings, wherein the text elements associated with the images are captions;

encoding the audio of the video source into audio embeddings; and

aligning the video embeddings and the audio embedding.

13 . The non-transitory computer-readable medium of claim 11 , wherein generating the second set of embeddings comprises combining encoded embeddings of the query and encoded embeddings of the text elements associated with the images.

14 . The non-transitory computer-readable medium of claim 11 , wherein performing the video analysis to select the subset of the one or more video clips comprises:

performing a similarity search using the second set of embeddings against the first set of embeddings;

generating similarity scores for the one or more video clips based at least in part on the first-part guidance; and

selecting the subset of the one or more video clips by maximizing summation of the similarity scores of the selected subset and the second-part guidance, wherein the second-part guidance comprises a duration of time and likelihood of repeat events.

15 . A computing system, comprising:

one or more processors; and

one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors of the computing system, cause the computing system to:

access, by the computing system, a video source comprising one or more video clips, each clip of the one or more video clips comprising a series of images, audio, and text elements associated with the series of images;

generate, by the computing system, a first set of embeddings based at least in part on the video source, the first set of embeddings comprising an aligned image portion, an audio portion and a text portion;

receive, by the computing system, a query providing guidance for extracting clips from the video source, the guidance comprising a first-part guidance and a second-part guidance, the first-part guidance comprising preferences for events, and the second-part guidance comprising criteria, and the criteria comprising a duration budget;

generate, by the computing system, a second set of embeddings based at least in part on the query; and

perform, by a machine learning (ML) model of the computing system, a video analysis comprising:

dividing the first set of embeddings into a plurality of sections;

determining a similarity score between each section of the plurality of sections and the second set of embeddings; and

generating a summary of the one or more video clips using the first set of embeddings and the second set of embeddings, the summary including a subset of the one or more video clips being selected based at least in part on the first-part guidance and the second-part guidance.

16 . The computing system of claim 15 , wherein the first set of embeddings are generated by:

encoding the series of images, text elements associated with the images into video embeddings, wherein the text elements associated with the images are captions;

encoding the audio of the video source into audio embeddings; and

aligning the video embeddings and the audio embedding.

17 . The computing system of claim 15 , wherein generating the second set of embeddings comprises combining encoded embeddings of the query and encoded embeddings of the text elements associated with the images.

18 . The computing system of claim 15 , wherein performing the video analysis to select the subset of the one or more video clips comprises:

performing a similarity search using the second set of embeddings against the first set of embeddings;

generating similarity scores for the one or more video clips based at least in part on the first-part guidance; and

selecting the subset of the one or more video clips by maximizing summation of the similarity scores of the selected subset and the second-part guidance, wherein the second-part guidance comprises a duration of time and likelihood of repeat events.

19 . The method of claim 1 wherein the summary of the one or more video clips is selected based at least in part on the first-part guidance and the second-part guidance configured at least in part so that

the summary maximizes a sum over the similarity scores associated with each section in the summary;

the summary ensures a total duration of the summary is less than the duration budget; or

the summary removes redundant sections of the plurality of sections.