Audio-based media edit point selection
A media edit point selection process can include a media editing software application programmatically converting speech to text and storing a timestamp-to-text map. The map correlates text corresponding to speech extracted from an audio track for the media clip to timestamps for the media clip. The timestamps correspond to words and some gaps in the speech from the audio track. The probability of identified gaps corresponding to a grammatical pause by the speaker is determined using the timestamp-to-text map and a semantic model. Potential edit points corresponding to grammatical pauses in the speech are stored for display or for additional use by the media editing software application. Text can optionally be displayed to a user during media editing.
1 . A method comprising:
producing, by a processor, a timestamp-to-text map of gaps in speech represented in an audio track for each video clip of a plurality of existing video clips stored in a gallery, each gap identified based on a time-based probability of the gap corresponding to a sentence ending;
determining, by the processor, a comparative probability of the gap corresponding to a grammatical pause in the speech represented in the audio track, using the timestamp-to-text map and a deep-learning neural network trained to output, for each gap, using only the audio track, a word-based probability of a word at a beginning of the gap being followed by another word in a sentence;
accessing, by the processor, an input-adjustable grammatical threshold, above which, based on the comparative probability, the gap is treated as the grammatical pause, whereby varying a value of the input-adjustable grammatical threshold causes the processor to select more or fewer edit points for the video clip;
automatically identifying, by the processor, while not providing user control of editing, based on the comparative probability relative to the value of the input-adjustable grammatical threshold, the edit points for the video clip, each edit point corresponding to the grammatical pause in the speech represented in the audio track of the video clip;
storing, by the processor, the edit points for the video clip;
automatically performing editing actions based on the edit points, while not providing user control of editing;
automatically creating an edited video presentation from the plurality of existing video clips based on the editing actions; and
storing the edited video presentation.
2 . The method of claim 1 , wherein the edited video presentation comprises at least one of a highlight reel or a preview of a video presentation.
3 . The method of claim 1 , wherein the edited video presentation is configured for viewing and refinement to produce a final presentation.
4 . The method of claim 1 , further comprising kernel-additive modeling the audio track of the video clip to identify audio sources to isolate the speech from among the audio sources.
5 . A computing system comprising:
a memory device; and
a processing device coupled to the memory device, the processing device to perform operations comprising:
producing a timestamp-to-text map of gaps in speech represented in an audio track for each video clip of a plurality of existing video clips stored in a gallery, each gap identified based on a time-based probability of the gap corresponding to a sentence ending;
determining, a comparative probability of the gap corresponding to a grammatical pause in the speech represented in the audio track, using the timestamp-to-text map and a deep-learning neural network trained to output, for each gap, using only the audio track, a word-based probability of a word at a beginning of the gap being followed by another word in a sentence;
accessing an input-adjustable grammatical threshold, above which, based on the comparative probability, the gap is treated as the grammatical pause, whereby varying a value of the input-adjustable grammatical threshold causes the processing device to select more or fewer edit points for the video clip;
automatically identifying, while not providing user control of editing, using an edit point determination module, based on the comparative probability relative to the value of the input-adjustable grammatical threshold, the edit points for the video clip, each edit point corresponding to the grammatical pause in the speech represented in the audio track of the video clip;
storing, in a memory device, the edit points for the video clip;
automatically performing editing actions based on the edit points, while not providing user control of editing;
automatically creating an edited video presentation from the plurality of existing video clips based on the editing actions; and
storing the edited video presentation.
6 . The computing system of claim 5 , wherein the edited video presentation comprises at least one of a highlight reel or a preview of a video presentation.
7 . The computing system of claim 5 , wherein the edited video presentation is configured for viewing and refinement to produce a final presentation.
8 . The computing system of claim 5 , wherein the operations further comprise kernel-additive modeling the audio track of the video clip to identify audio sources to isolate the speech from among the audio sources.
9 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
producing a timestamp-to-text map of gaps in speech represented in an audio track for each video clip of a plurality of existing video clips stored in a gallery, each gap identified based on a time-based probability of the gap corresponding to a sentence ending;
determining, a comparative probability of the gap corresponding to a grammatical pause in the speech represented in the audio track, using the timestamp-to-text map and a deep-learning neural network trained to output, for each gap, using only the audio track, a word-based probability of a word at a beginning of the gap being followed by another word in a sentence;
a step for automatically identifying, while not providing user control of editing, based on the comparative probability relative to a value of an input-adjustable grammatical threshold, edit points for the video clip, an edit point corresponding to the grammatical pause in the speech represented in the audio track of the video clip;
storing the edit points for the video clip;
automatically performing editing actions based on the edit points, while not providing user control of editing;
automatically creating an edited video presentation from the plurality of existing video clips based on the editing actions; and
storing the edited video presentation.
10 . The non-transitory computer-readable medium of claim 9 , wherein the edited video presentation comprises at least one of a highlight reel or a preview of a video presentation.
11 . The non-transitory computer-readable medium of claim 9 , wherein the edited video presentation is configured for viewing and refinement to produce a final presentation.
12 . The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise kernel-additive modeling the audio track of the video clip to identify audio sources to isolate the speech from among the audio sources.