Audio processing for detecting occurrences of loud sound characterized by brief audio bursts
A boundary of a highlight of audiovisual content depicting an event is identified. The audiovisual content may be a broadcast, such as a television broadcast of a sporting event. The highlight may be a segment of the audiovisual content deemed to be of particular interest. Audio data for the audiovisual content is stored, and the audio data is automatically analyzed to detect one or more audio events indicative of one or more occurrences to be included in the highlight. Each audio event may be a brief, high-energy audio burst such as the sound made by a tennis serve. A time index within the audiovisual content, before or after the audio event, may be designated as the boundary, which may be the beginning or end of the highlight.
1. A method of selecting an audio event within an audio data to be included in a highlight of the audio data, the method comprising:
analyzing, by a computer in a time domain, an audio data to detect a high energy audio burst corresponding to an audio event within the audio data;
determining, by the computer, an event time position within the audio data of the audio event;
analyzing, by the computer in a frequency domain, a portion of the audio data within a time spread range containing the event time position to generate a spectral distribution of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the portion of the audio data within the time spread range by extracting, using a two-dimensional diamond-shaped spectrogram area filter, a subset of spectral peaks that form the spectral distribution of the audio in the time spread range;
determining, by the computer, the number of peaks in the spectral distribution of audio in the time spread range; and
in response to the computer determining that a number of peaks is below a threshold, adding, by the computer, the audio event in a highlight of the audio data.
2. The method of claim 1 , wherein the filtering comprises:
sliding, by the computer, the two-dimensional diamond-shaped spectrogram area filter along time and frequency axes of the spectrogram to a first position;
determining, by the computer at the first position of the two-dimensional diamond-shaped spectrogram area filter, whether a central spectral peak magnitude is greater than remaining spectral peak magnitudes; and
in response to the computer determining that the central spectral peak magnitude is greater than the remaining spectral peak magnitudes for the first position of the two-dimensional diamond-shaped spectrogram area filter:
adding, by the computer, the central spectral peak in the subset of the spectral peaks that form the spectral distribution of the audio in the time spread range.
3. The method of claim 1 , wherein analyzing, in the frequency domain, the portion of the audio data within the time spread range comprises:
analyzing, by the computer, the portion of the audio data within the time spread range in a joint time-frequency domain.
4. The method of claim 1 , wherein analyzing the audio data in the time domain further comprises:
selecting, by the computer, an analysis time window;
sliding, by the computer, the analysis time window along the audio data;
computing, by the computer, normalized magnitudes of audio samples at each position of the analysis time window; and
detecting, by the computer, the high energy audio burst based on the computed normalized magnitudes of the audio samples.
5. The method of claim 1 , further comprising:
preprocessing, by the computer, the audio data prior to analyzing the audio data in the time domain, by resampling the audio data to a predetermined sampling rate.
6. The method of claim 1 , further comprising:
preprocessing, by the computer, the audio data prior to analyzing the audio data in the time domain, by filtering the audio data to be within a predetermined spectral band.
7. The method of claim 1 , wherein the audio data comprises audio from a sports broadcast.
8. The method of claim 7 , wherein the sports comprises tennis, and wherein the event comprises a tennis serve.
9. A system for selecting an audio event within an audio data to be included in a highlight of the audio data, the system comprising:
at least one processor; and
a computer-readable non-transitory storage medium storing computer program instructions that when executed by the at least one processor cause the system to perform operations comprising:
analyzing, in a time domain, an audio data to detect a high energy audio burst corresponding to an audio event within the audio data;
determining an event time position within the audio data of the audio event;
analyzing, in a frequency domain, a portion of the audio data within a time spread range containing the event time position to generate a spectral distribution of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the portion of the audio data within the time spread range by extracting, using a two-dimensional diamond-shaped spectrogram area filter, a subset of spectral peaks that form the spectral distribution of the audio in the time spread range;
determining the number of peaks in the spectral distribution of audio in the time spread range; and
in response to the determining that a number of peaks is below a threshold, adding the audio event in a highlight of the audio data.
10. The system of claim 9 , wherein the filtering comprises:
sliding the two-dimensional diamond-shaped spectrogram area filter along time and frequency axes of the spectrogram to a first position;
determining, at the first position of the two-dimensional diamond-shaped spectrogram area filter, whether a central spectral peak magnitude is greater than remaining spectral peak magnitudes; and
in response to determining that the central spectral peak magnitude is greater than the remaining spectral peak magnitudes for the first position of the two-dimensional diamond-shaped spectrogram area filter:
adding the central spectral peak in the subset of the spectral peaks that form the spectral distribution of the audio in the time spread range.
11. The system of claim 9 , wherein analyzing, in the frequency domain, the portion of the audio data within the time spread range comprises:
analyzing the portion of the audio data within the time spread range in a joint time-frequency domain.
12. The system of claim 9 , wherein analyzing the audio data in the time domain further comprises:
selecting an analysis time window;
sliding the analysis time window along the audio data;
computing normalized magnitudes of audio samples at each position of the analysis time window; and
detecting the high energy audio burst based on the computed normalized magnitudes of the audio samples.
13. The system of claim 9 , the operations further comprising:
preprocessing the audio data prior to analyzing the audio data in the time domain by resampling the audio data to a predetermined sampling rate.
14. The system of claim 9 , the operations further comprising:
preprocessing the audio data prior to analyzing the audio data in the time domain by filtering the audio data to be within a predetermined spectral band.
15. A non-transitory computer-readable medium storing computer program instructions that when executed cause operations comprising:
analyzing, in a time domain, an audio data to detect a high energy audio burst corresponding to an audio event within the audio data;
determining an event time position within the audio data of the audio event;
analyzing, in a frequency domain, a portion of the audio data within a time spread range containing the event time position to generate a spectral distribution of audio in the time spread range, wherein the analyzing includes filtering a spectrogram of the portion of the audio data within the time spread range by extracting, using a two-dimensional diamond-shaped spectrogram area filter, a subset of spectral peaks that form the spectral distribution of the audio in the time spread range;
determining the number of peaks in the spectral distribution of audio in the time spread range; and
in response to the determining that a number of peaks is below a threshold, adding the audio event in a highlight of the audio data.
16. The non-transitory computer-readable medium of claim 15 , wherein the filtering comprises:
sliding the two-dimensional diamond-shaped spectrogram area filter along time and frequency axes of the spectrogram to a first position;
determining, at the first position of the two-dimensional diamond-shaped spectrogram area filter, whether a central spectral peak magnitude is greater than remaining spectral peak magnitudes; and
in response to determining that the central spectral peak magnitude is greater than the remaining spectral peak magnitudes for the first position of the two-dimensional diamond-shaped spectrogram area filter:
adding the central spectral peak in the subset of the spectral peaks that form the spectral distribution of the audio in the time spread range.
17. The non-transitory computer-readable medium of claim 15 , wherein analyzing, in the frequency domain, the portion of the audio data within the time spread range comprises:
analyzing the portion of the audio data within the time spread range in a joint time-frequency domain.