Audio processing for detecting occurrences of loud sound characterized by brief audio bursts
A boundary of a highlight of audiovisual content depicting an event is identified. The audiovisual content may be a broadcast, such as a television broadcast of a sporting event. The highlight may be a segment of the audiovisual content deemed to be of particular interest. Audio data for the audiovisual content is stored, and the audio data is automatically analyzed to detect one or more audio events indicative of one or more occurrences to be included in the highlight. Each audio event may be a brief, high-energy audio burst such as the sound made by a tennis serve. A time index within the audiovisual content, before or after the audio event, may be designated as the boundary, which may be the beginning or end of the highlight.
1 . A method for selecting an audio event for inclusion in an audio data highlight, the method comprising:
receiving, by one or more processors, audio data from a data store;
detecting, by the one or more processors, one or more high energy audio bursts corresponding to one or more audio events within the audio data;
determining, by the one or more processors, one or more event time positions within the audio data of the audio events;
analyzing, by the one or more processors, one or more portions of the audio data within a time spread range containing the one or more event time positions to generate one or more spectral distributions of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the one or more portions of the audio data within the time spread range;
determining, by the one or more processors, one or more number of peaks in the one or more spectral distributions in the time spread range;
in response to determining that a first number of peaks of the one or more number of peaks is above a threshold, suppressing, by the one or more processors, a first audio event of the one or more audio events associated with the first number of peaks; and
in response to determining that a second number of peaks of the one or more number of peaks is below a threshold, adding, by the one or more processors, a second audio event of the one or more audio events associated with the second number of peaks in a highlight of the audio data, such that the highlight includes the second audio event and not the first audio event.
2 . The method of claim 1 , wherein the receiving includes:
receiving, by the one or more processors, the audio data from a live broadcast audiovisual stream.
3 . The method of claim 1 , wherein analyzing, by the one or more processors, the one or more portions of the audio data within the time spread range comprises:
analyzing, by the one or more processors, the one or more portions of the audio data within the time spread range in a joint time-frequency domain.
4 . The method of claim 1 , wherein analyzing the one or more portions of the audio data further comprises:
selecting, by the one or more processors, an analysis time window;
sliding, by the one or more processors, the analysis time window along the audio data;
computing, by the one or more processors, normalized magnitudes of audio samples at each position of the analysis time window; and
detecting, by the one or more processors, the one or more high energy audio bursts based on the computed normalized magnitudes of the audio samples.
5 . The method of claim 1 , further comprising:
preprocessing, by the one or more processors, the audio data prior to analyzing the audio data, by resampling the audio data to a predetermined sampling rate.
6 . The method of claim 1 , further comprising:
preprocessing, by the one or more processors, the audio data prior to analyzing the audio data, by filtering the audio data to be within a predetermined spectral band.
7 . The method of claim 1 , wherein the audio data comprises audio from a sports broadcast.
8 . The method of claim 1 , wherein the one or more audio events include an audio announcement, music, a tennis ball hit, or a cheering crowd noise.
9 . A system for selecting an audio event for inclusion in an audio data highlight, comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, performs one or more operations, comprising:
receiving audio data from a data store;
detecting one or more high energy audio bursts corresponding to one or more audio events within the audio data;
determining one or more event time positions within the audio data of the audio events;
analyzing one or more portions of the audio data within a time spread range containing the one or more event time positions to generate one or more spectral distributions of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the one or more portions of the audio data within the time spread range;
determining one or more number of peaks in the one or more spectral distributions in the time spread range;
in response to determining that a first number of peaks of the one or more number of peaks is above a threshold, suppressing a first audio event of the one or more audio events associated with the first number of peaks; and
in response to determining that a second number of peaks of the one or more number of peaks is below a threshold, adding a second audio event of the one or more audio events associated with the second number of peaks in a highlight of the audio data, such that the highlight includes the second audio event and not the first audio event.
10 . The system of claim 9 , wherein the receiving includes:
receiving the audio data from a live broadcast audiovisual stream.
11 . The system of claim 9 , wherein analyzing the one or more portions of the audio data within the time spread range comprises:
analyzing the one or more portions of the audio data within the time spread range in a joint time-frequency domain.
12 . The system of claim 9 , wherein analyzing the one or more portions of the audio data further comprises:
selecting an analysis time window;
sliding the analysis time window along the audio data;
computing normalized magnitudes of audio samples at each position of the analysis time window; and
detecting the one or more high energy audio bursts based on the computed normalized magnitudes of the audio samples.
13 . The system of claim 9 , further comprising:
preprocessing the audio data prior to analyzing the audio data, by resampling the audio data to a predetermined sampling rate.
14 . The system of claim 9 , further comprising:
preprocessing the audio data prior to analyzing the audio data, by filtering the audio data to be within a predetermined spectral band.
15 . The system of claim 9 , wherein the audio data comprises audio from a sports broadcast.
16 . The system of claim 9 , wherein the one or more audio events include an audio announcement, music, a tennis ball hit, or a cheering crowd noise.
17 . A non-transitory computer readable medium including one or more sequences of instructions that, when executed by one or more processors, causes:
receiving, by the one or more processors, audio data from a data store;
detecting, by the one or more processors, one or more high energy audio bursts corresponding to one or more audio events within the audio data;
determining, by the one or more processors, one or more event time positions within the audio data of the audio events;
analyzing, by the one or more processors, one or more portions of the audio data within a time spread range containing the one or more event time positions to generate one or more spectral distributions of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the one or more portions of the audio data within the time spread range;
determining, by the one or more processors, one or more number of peaks in the one or more spectral distributions in the time spread range;
in response to determining that a first number of peaks of the one or more number of peaks is above a threshold, suppressing, by the one or more processors, a first audio event of the one or more audio events associated with the first number of peaks; and
in response to determining that a second number of peaks of the one or more number of peaks is below a threshold, adding, by the one or more processors, a second audio event of the one or more audio events associated with the second number of peaks in a highlight of the audio data, such that the highlight includes the second audio event and not the first audio event.
18 . The non-transitory computer readable medium of claim 17 , wherein the receiving includes:
receiving, by the one or more processors, the audio data from a live broadcast audiovisual stream.
19 . The non-transitory computer readable medium of claim 17 , wherein analyzing, by the one or more processors, the one or more portions of the audio data within the time spread range comprises:
analyzing, by the one or more processors, the one or more portions of the audio data within the time spread range in a joint time-frequency domain.
20 . The non-transitory computer readable medium of claim 17 , wherein analyzing the one or more portions of the audio data further comprises:
selecting, by the one or more processors, an analysis time window;
sliding, by the one or more processors, the analysis time window along the audio data;
computing, by the one or more processors, normalized magnitudes of audio samples at each position of the analysis time window; and
detecting, by the one or more processors, the one or more high energy audio bursts based on the computed normalized magnitudes of the audio samples.