Granted Patent B1

US 12,647,658 · App. 19/349,220 · Granted Jun 2, 2026

Inventor: Michael Mogill (Atlanta, GA)

Assignee: Crisp, Inc.

H04N21/854G06V10/764G06V10/7715G10L15/26H04N21/44016H04N21/816

Abstract

Claims (88)

1 . An apparatus for automated video production, the apparatus comprising:

at least a processor; and

a memory communicatively connected to the at least a processor, wherein the memory contains instructions configuring the at least a processor to:

receive raw content comprising a plurality of image frames and an associated speech waveform;

extract one or more content features from the plurality of image frames and the associated speech waveform;

assign at least a content identifier to at least a portion of the raw content as a function of the one or more content features;

modify the speech waveform to generate a modified waveform using a speech model, wherein generating the modified waveform comprises:

detecting one or more filler words from the speech waveform; and

removing the one or more filler words from the speech waveform;

generate at least a supplemental segment that is contextually matched to the at least a content identifier using a generative model, wherein generating the at least a supplemental segment comprises:

inserting the at least a supplemental segment into the plurality of image frames at a position aligned with the at least a content identifier;

create a unified edit timeline comprising the at least a content identifier, the modified waveform, and the at least a supplemental segment; and

generate a user interface comprising the unified edit timeline.

2 . The apparatus of claim 1 , wherein extracting the one or more content features comprises:

generating a transcript of the speech waveform using an automatic speech recognition module of the speech model, wherein the transcript is time-aligned with the plurality of image frames; and

extracting the one or more content features from the transcript.

3 . The apparatus of claim 2 , wherein generating the transcript of the speech waveform comprises:

receiving the speech waveform comprising speech data from a plurality of different speakers;

generating masked clusters of the speech waveform;

converting the masked clusters into a time domain to obtain a plurality of separated speech signals corresponding to the plurality of different speakers;

extracting one or more acoustic features of the one or more content features from a target speech signal; and

generating a sequence of words from the one or more acoustic features to produce the transcript corresponding to the target speech signal.

4 . The apparatus of claim 3 , wherein detecting the one or more filler words comprises distinguishing the one or more filler words from the speech waveform based on a frequency difference in the one or more acoustic features.

5 . The apparatus of claim 3 , wherein assigning the at least a content identifier comprises:

segmenting the speech waveform into a plurality of speaker-specific segments based on the plurality of speech signals; and

assigning a speaker identifier of the at least a content identifier to each of the plurality of speaker-specific segments.

6 . The apparatus of claim 1 , wherein assigning the at least a content identifier comprises:

extracting one or more visual features of the one or more content features from the plurality of image frames using a computer vision module;

generating an embedding representation of the one or more visual features;

classifying the embedding representation into one or more scene categories; and

assigning a scene identifier of the at least a content identifier corresponding to the one or more scene categories to at least a portion of the raw content.

7 . The apparatus of claim 1 , wherein assigning the at least a content identifier comprises:

generating an embedding representation of the one or more content features;

classifying the embedding representation into one or more emotion categories; and

assigning an emotion identifier of the at least a content identifier corresponding to the one or more emotion categories to at least a portion of the raw content.

8 . The apparatus of claim 1 , wherein modifying the speech waveform comprises:

identifying temporal intervals within the speech waveform corresponding to the one or more filler words;

removing the temporal intervals from the speech waveform to generate the modified waveform; and

modifying the plurality of image frames to temporally align with the modified waveform.

9 . The apparatus of claim 1 , wherein generating the at least a supplemental segment comprises:

selecting the at least a supplemental segment using a generative adversarial network of the generative model by comparing the one or more content features with a plurality of stored video segments; and

inserting the at least a supplemental segment into the raw content at a position aligned with the one or more content features.

10 . The apparatus of claim 1 , wherein creating the unified edit timeline comprises:

segmenting the raw content into a plurality of discrete units as a function of the at least a content identifier, the modified waveform, and the at least a supplemental segment; and

assigning the plurality of discrete units to one or more tracks within a timeline structure of the unified edit timeline, wherein the one or more tracks comprises a primary track and at least one secondary track.

11 . A method for automated video production, the method comprising:

receiving, using at least a processor, raw content comprising a plurality of image frames and an associated speech waveform;

extracting, using the at least a processor, one or more content features from the plurality of image frames and the associated speech waveform;

assigning, using the at least a processor, at least a content identifier to at least a portion of the raw content as a function of the one or more content features;

modifying, using the at least a processor, the speech waveform to generate a modified waveform using a speech model, wherein generating the modified waveform comprises:

detecting one or more filler words from the speech waveform; and

removing the one or more filler words from the speech waveform;

generating, using the at least a processor, at least a supplemental segment that is contextually matched to the at least a content identifier using a generative model, wherein generating the at least a supplemental segment comprises:

inserting the at least a supplemental segment into the plurality of image frames at a position aligned with the at least a content identifier;

creating, using the at least a processor, a unified edit timeline comprising the at least a content identifier, the modified waveform, and the at least a supplemental segment; and

generating, using the at least a processor, a user interface comprising the unified edit timeline.

12 . The method of claim 11 , wherein extracting the one or more content features comprises:

generating a transcript of the speech waveform using an automatic speech recognition module of the speech model, wherein the transcript is time-aligned with the plurality of image frames; and

extracting the one or more content features from the transcript.

13 . The method of claim 12 , wherein generating the transcript of the speech waveform comprises:

receiving the speech waveform comprising speech data a plurality of different speakers;

generating masked clusters of the speech waveform;

converting the masked clusters into a time domain to obtain a plurality of separated speech signals corresponding to the plurality of different speakers;

extracting one or more acoustic features of the one or more content features from a target speech signal; and

generating a sequence of words from the one or more acoustic features to produce the transcript corresponding to the target speech signal.

14 . The method of claim 13 , wherein detecting the one or more filler words comprises distinguishing the one or more filler words from the speech waveform based on a frequency difference in the one or more acoustic features.

15 . The method of claim 13 , wherein assigning the at least a content identifier comprises:

segmenting the speech waveform into a plurality of speaker-specific segments based on the plurality of speech signals; and

assigning a speaker identifier of the at least a content identifier to each of the plurality of speaker-specific segments.

16 . The method of claim 11 , wherein assigning the at least a content identifier comprises:

extracting one or more visual features of the one or more content features from the plurality of image frames using a computer vision module;

generating an embedding representation of the one or more visual features;

classifying the embedding representation into one or more scene categories; and

assigning a scene identifier of the at least a content identifier corresponding to the one or more scene categories to at least a portion of the raw content.

17 . The method of claim 11 , wherein assigning the at least a content identifier comprises:

generating an embedding representation of the one or more content features;

classifying the embedding representation into one or more emotion categories; and

assigning an emotion identifier of the at least a content identifier corresponding to the one or more emotion categories to at least a portion of the raw content.

18 . The method of claim 11 , wherein modifying the speech waveform comprises:

identifying temporal intervals within the speech waveform corresponding to the one or more filler words;

removing the temporal intervals from the speech waveform to generate the modified waveform; and

modifying the plurality of image frames to temporally align with the modified waveform.

19 . The method of claim 11 , wherein generating the at least a supplemental segment comprises:

selecting the at least a supplemental segment using a generative adversarial network of the generative model by comparing the one or more content features with a plurality of stored video segments; and

inserting the at least a supplemental segment into the raw content at a position aligned with the one or more content features.

20 . The method of claim 11 , wherein creating the unified edit timeline comprises:

segmenting the raw content into a plurality of discrete units as a function of the at least a content identifier, the modified waveform, and the at least a supplemental segment; and

assigning the plurality of discrete units to one or more tracks within a timeline structure of the unified edit timeline, wherein the one or more tracks comprises a primary track and at least one secondary track.

Assignments (1)

References Cited (14)