Video assembly using generative artificial intelligence
Embodiments of the present invention provide systems, methods, and computer storage media for identifying the relevant segments that effectively summarize the larger input video and/or form a rough cut, and assembling them into one or more smaller trimmed videos. For example, visual scenes and corresponding scene captions are extracted from the input video and associated with an extracted diarized and timestamped transcript to generate an augmented transcript. The augmented transcript is applied to a large language model to extract sentences that characterize a trimmed version of the input video (e.g., a natural language summary, a representation of identified sentences from the transcript). As such, corresponding video segments are identified (e.g., using similarity to match each sentence in a generated summary with a corresponding transcript sentence) and assembled into one or more trimmed videos. In some embodiments, the trimmed video is generated based on a user's query and/or desired length.
1 . One or more computer storage media storing computer-useable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
accessing an input video and a transcript corresponding to the input video, the input video segmented into a plurality of scenes;
generating, based on aligning scene captions of the plurality of scenes with the transcript to output an augmented transcript and applying the augmented transcript to a language model, a representation of sentences characterizing a trimmed version of the input video;
identifying a subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video; and
generating the trimmed version of the input video by assembling the subset of video segments into a trimmed video.
2 . The one or more computer storage media of claim 1 , the operations further comprising:
segmenting the input video into the plurality of scenes based on a computation of similarity of frame embeddings of each corresponding frame of each scene of the plurality of scenes by a language-image pretrained model.
3 . The one or more computer storage media of claim 1 , the operations further comprising:
generating the scene captions by an image caption generator model.
4 . The one or more computer storage media of claim 1 , the operations further comprising:
generating the transcript by an automated speech recognition model, the transcript comprising speaker diarization and word-level timing; and
segmenting the transcript into a plurality of sentences by a sentence segmentation model.
5 . A method comprising:
accessing an input video and a transcript corresponding to the input video, the input video segmented into a plurality of scenes;
generating, based on aligning scene captions of the plurality of scenes with the transcript to output an augmented transcript and processing the augmented transcript using a language model, a representation of sentences characterizing a trimmed version of the input video;
identifying a subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video; and
generating the trimmed version of the input video by assembling the subset of video segments into a trimmed video.
6 . The one or more computer storage media of claim 1 , the operations further comprising:
applying a prompt to the language model to generate a summarized version of the input video as the representation of sentences characterizing the trimmed version of the input video; and
identifying the subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video by:
computing similarity of each sentence embedding of each of the sentences characterizing the trimmed version of the input video to each sentence embedding of each sentence of the augmented transcript; and
for each of the sentences characterizing the trimmed version of the input video, selecting a video segment of the subset of video segments based on the similarity computed for a corresponding sentence of the video segment from the augmented transcript.
7 . The one or more computer storage media of claim 1 , the operations further comprising:
for each of the sentences characterizing the trimmed version of the input video:
determining a threshold number of similar sentences from the augmented transcript; and
selecting each video segment of the subset of video segments from a corresponding one of the similar sentences based on a desired length of the trimmed video.
8 . The one or more computer storage media of claim 1 , the operations further comprising:
applying a prompt to the language model to generate a rough cut version of the input video and wherein each sentence of the representation of sentences characterizing the trimmed version of the input video corresponds to a corresponding sentence from the augmented transcript.
9 . The method of claim 5 , further comprising:
for each of the sentences characterizing the trimmed version of the input video:
determining a threshold number of similar sentences from the augmented transcript; and
selecting each video segment of the subset of video segments from a corresponding one of the similar sentences based on a desired length of the trimmed video.
10 . The method of claim 9 , further comprising:
segmenting the input video into the plurality of scenes based on a computation of similarity of frame embeddings of each corresponding frame of each scene of the plurality of scenes by a language-image pretrained model.
11 . The method of claim 9 , further comprising:
generating the scene captions by an image caption generator model.
12 . The method of claim 9 , further comprising:
generating the transcript by an automated speech recognition model, the transcript comprising speaker diarization and word-level timing; and
segmenting the transcript into a plurality of sentences by a sentence segmentation model.
13 . The method of claim 9 , further comprising:
applying a prompt to the language model to generate a summarized version of the input video as the representation of sentences characterizing the trimmed version of the input video; and
identifying the subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video by:
computing similarity of each sentence embedding of each of the sentences characterizing the trimmed version of the input video to each sentence embedding of each sentence of the augmented transcript; and
for each of the sentences characterizing the trimmed version of the input video, selecting a video segment of the subset of video segments based on the similarity computed for a corresponding sentence of the video segment from the augmented transcript.
14 . The method of claim 9 , further comprising:
applying a prompt to the language model to generate a rough cut version of the input video and wherein each sentence of the representation of sentences characterizing the trimmed version of the input video corresponds to a corresponding sentence from the augmented transcript.
15 . A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a video ingestion component configured to access an input video and a transcript corresponding to the input video, the input video segmented into a plurality of scenes;
a prompt component configured to trigger-generating, based on aligning scene captions of the plurality of scenes with the transcript to output an augmented transcript and applying the augmented transcript to a language model, a representation of sentences characterizing a trimmed version of the input video;
a selection component configured to identify a subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video; and
an assembly component configured to generate the trimmed version of the input video by assembling the subset of video segments into a trimmed video.
16 . The computer system of claim 15 , the computer program instructions further comprising:
segmenting the input video into the plurality of scenes based on a computation of similarity of frame embeddings of each corresponding frame of each scene of the plurality of scenes by a language-image pretrained model.
17 . The computer system of claim 15 , the computer program instructions further comprising:
generating the scene captions by an image caption generator model.
18 . The computer system of claim 15 , the computer program instructions further comprising:
generating the transcript by an automated speech recognition model, the transcript comprising speaker diarization and word-level timing; and
segmenting the transcript into a plurality of sentences by a sentence segmentation model.
19 . The computer system of claim 15 , the computer program instructions further comprising:
the prompt component further configured to apply a prompt to the language model to generate a summarized version of the input video as the representation of sentences characterizing the trimmed version of the input video; and
the selection component further configured to identify the subset of video segments of the input video corresponding to each of the sentences characterizing the trimmed version of the input video by:
computing similarity of each sentence embedding of each of the sentences characterizing the trimmed version of the input video to each sentence embedding of each sentence of the augmented transcript; and
for each of the sentences characterizing the trimmed version of the input video, selecting a video segment of the subset of video segments based on the similarity computed for a corresponding sentence of the video segment from the augmented transcript.
20 . The computer system of claim 15 , the computer program instructions further comprising:
for each of the sentences characterizing the trimmed version of the input video:
determining a threshold number of similar sentences from the augmented transcript; and
selecting each video segment of the subset of video segments from a corresponding one of the similar sentences based on a desired length of the trimmed video.