Multi-level audio segmentation using deep embeddings
Embodiments are disclosed for generating an audio segmentation of an audio sequence using deep embeddings. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input including an audio sequence and extracting features for each frame of the audio sequence, where each frame is associated with a beat of the audio sequence. The method may further comprise clustering frames of the audio sequence into one or more clusters based on the extracted features and generating segments of the audio sequence based on the clustered frames, where each segment includes frames of the audio sequence from a same cluster. The method may further comprise constructing a multi-level audio segmentation of the audio sequence and performing a segment fusioning process that merges shorter segments with neighboring segments based on cluster assignments.
1 . A computer-implemented method comprising:
receiving an input including an audio sequence and a first segment value indicating a number of clusters to group frames of the audio sequence into;
extracting features for each frame of the audio sequence, each frame associated with a beat of the audio sequence;
clustering the frames of the audio sequence into one or more clusters based on the extracted features and the first segment value;
generating segments of the audio sequence based on the clustered frames, each segment of the audio sequence formed by grouping consecutive frames of the audio sequence from a same cluster of the one or more clusters; and
generating a first representation of the audio sequence using the generated segments of the audio sequence, wherein a subset of the generated segments with durations less than a duration threshold are fused with neighboring segments based on a second representation of the audio sequence generated using a second segment value.
2 . The computer-implemented method of claim 1 , wherein extracting the features for each frame of the audio sequence comprises:
processing the audio sequence through an audio model trained to extract features for each frame of the audio sequence using deep audio embeddings.
3 . The computer-implemented method of claim 1 , wherein generating segments of the audio sequence based on the clustered frames comprises:
assigning each frame of the audio sequence a cluster identifier based on the extracted features, wherein frames associated with a same cluster identifier have similar extracted features.
4 . The computer-implemented method of claim 1 , further comprising:
constructing a multi-level audio segmentation of the audio sequence, wherein each level of the multi-level audio segmentation includes a different number of unique clusters, and wherein the multi-level audio segmentation includes the first representation with a first number of unique clusters based on the first segment value and the second representation with a second number of unique clusters based on the second segment value.
5 . The computer-implemented method of claim 4 , wherein generating the first representation of the audio sequence using the generated segments of the audio sequence further comprises:
identifying the subset of the generated segments of the audio sequence that have a duration less than the duration threshold; and
for each segment of the identified subset of the generated segments, performing a segment fusioning process by merging the segment with a neighboring segment in the first representation based on cluster assignments related to the segment and neighboring frames at lower levels of the multi-level audio segmentation of the audio sequence, including the second representation.
6 . The computer-implemented method of claim 4 , further comprising:
generating an audio segmentation representation of the audio sequence based on the generated segments; and
selecting a level of the multi-level audio segmentation as an output based on a segmentation level selection.
7 . The computer-implemented method of claim 1 , further comprising:
applying a beat detection algorithm to the audio sequence to identify the beats of the audio sequence.
8 . The computer-implemented method of claim 1 , further comprising:
associating a first segment of the audio sequence with a second segment of the audio sequence when the first segment and the second segment include frames from a same first cluster of the one or more clusters.
9 . A non-transitory computer-readable storage medium including instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:
receive an input including an audio sequence and a first segment value indicating a number of clusters to group frames of the audio sequence into;
extract features for each frame of the audio sequence, each frame associated with a beat of the audio sequence;
cluster the frames of the audio sequence into one or more clusters based on the extracted features and the first segment value;
generate segments of the audio sequence based on the clustered frames, each segment of the audio sequence formed by grouping consecutive frames of the audio sequence from a same cluster of the one or more clusters; and
generate a first representation of the audio sequence using the generated segments of the audio sequence, wherein a subset of the generated segments with durations less than a duration threshold are fused with neighboring segments based on a second representation of the audio sequence generated using a second segment value.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein to extract the features for each frame of the audio sequence, the instructions, when executed, further cause the at least one processor to:
process the audio sequence through an audio model trained to extract features for each frame of the audio sequence using deep audio embeddings.
11 . The non-transitory computer-readable storage medium of claim 9 , wherein to generate segments of the audio sequence based on the clustered frames, the instructions, when executed, further cause the at least one processor to:
assign each frame of the audio sequence a cluster identifier based on the extracted features, wherein frames associated with a same cluster identifier have similar extracted features.
12 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the at least one processor to:
construct a multi-level audio segmentation of the audio sequence, wherein each level of the multi-level audio segmentation includes a different number of unique clusters, and wherein the multi-level audio segmentation includes the first representation with a first number of unique clusters based on the first segment value and the second representation with a second number of unique clusters based on the second segment value.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein to generate the first representation of the audio sequence using the generated segments of the audio sequence, the instructions, when executed, further cause the at least one processor to:
identify the subset of the generated segments of the audio sequence that have a duration less than the duration threshold; and
for each segment of the identified subset of the generated segments, perform a segment fusioning process by merging the segment with a neighboring segment in the first representation based on cluster assignments related to the segment and neighboring frames at lower levels of the multi-level audio segmentation of the audio sequence, including the second representation.
14 . The non-transitory computer-readable storage medium of claim 12 , wherein the instructions, when executed, further cause the at least one processor to:
generate an audio segmentation representation of the audio sequence based on the generated segments; and
select a level of the multi-level audio segmentation as an output based on a segmentation level selection.
15 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the at least one processor to:
apply a beat detection algorithm to the audio sequence to identify the beats of the audio sequence.
16 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the at least one processor to:
associate a first segment of the audio sequence with a second segment of the audio sequence when the first segment and the second segment include frames from a same first cluster of the one or more clusters.
17 . A system, comprising:
a computing device including a memory and at least one processor, the computing device implementing an audio processing system,
wherein the memory includes instructions stored thereon which, when executed, cause the audio processing system to:
receive an input including an audio sequence and a first segment value indicating a number of clusters to group frames of the audio sequence into;
extract features for each frame of the audio sequence, each frame associated with a beat of the audio sequence;
cluster the frames of the audio sequence into one or more clusters based on the extracted features and the first segment value;
generate segments of the audio sequence based on the clustered frames, each segment of the audio sequence formed by grouping consecutive frames of the audio sequence from a same cluster of the one or more clusters; and
generate a first representation of the audio sequence using the generated segments of the audio sequence, wherein a subset of the generated segments with durations less than a duration threshold are fused with neighboring segments based on a second representation of the audio sequence generated using a second segment value.
18 . The system of claim 17 , wherein the instructions to extract the features for each frame of the audio sequence, further cause the audio processing system to:
process the audio sequence through an audio model trained to extract features for each frame of the audio sequence using deep audio embeddings.
19 . The system of claim 17 , wherein the instructions to generate segments of the audio sequence based on the clustered frames, further cause the audio processing system to:
assign each frame of the audio sequence a cluster identifier based on the extracted features, wherein frames associated with a same cluster identifier have similar extracted features.
20 . The system of claim 17 , wherein the instructions further cause the audio processing system to:
construct a multi-level audio segmentation of the audio sequence, wherein each level of the multi-level audio segmentation includes a different number of unique clusters, and wherein the multi-level audio segmentation includes the first representation with a first number of unique clusters based on the first segment value and the second representation with a second number of unique clusters based on the second segment value.