Electronic device and method of recognizing audio scene
An electronic device and method of recognizing an audio scene are provided. The method of recognizing an audio scene includes: separating, according to a predetermined criterion, an input audio signal into channels; recognizing, according to each of the separated channels, at least one audio scene from the input audio signal by using a plurality of neural networks trained to recognize an audio scene; and determining, based on a result of the recognizing of the at least one audio scene, at least one audio scene included in audio content by using a neural network trained to combine audio scene recognition results for respective channels, wherein the plurality of neural networks includes: a first neural network trained to recognize the audio scene based on a time-frequency shape of an audio signal, a second neural network trained to recognize the audio scene based on a shape of a spectral envelope of the audio signal, and a third neural network trained to recognize the audio scene based on a feature vector extracted from the audio signal.
1. A method of recognizing an audio scene, the method comprising:
obtaining, according to a predetermined criterion, a mid channel and a side channel from an input audio signal;
identifying, from the input audio signal, an audio scene probability for the mid channel and an audio scene probability for the side channel, of at least one audio scene based on audio scene recognition results for the mid channel and the side channel that are respectively obtained by using a plurality of neural networks trained to recognize an audio scene, wherein the plurality of neural networks are duplicated and applied separately on the mid channel and the side channel, to output the audio scene recognition results for the mid channel and the side channel, respectively; and
identifying, based on the audio scene probability for the mid channel and the audio scene probability for the side channel, at least one audio scene included in audio content by using a fourth neural network trained to combine the audio scene probability for the mid channel and the audio scene probability for the side channel, wherein the fourth neural network is configured as a recurrent neural network,
wherein the plurality of neural networks comprises:
a first neural network trained to recognize the audio scene based on a time-frequency shape of an audio signal,
a second neural network trained to recognize the audio scene based on a shape of a spectral envelope of the audio signal, and
a third neural network trained to recognize the audio scene based on a feature vector extracted from the audio signal, and
wherein the identifying the at least one audio scene further comprises:
determining a predetermined first audio scene to be recognized in the mid channel and of a predetermined second audio scene to be recognized in the side channel, the predetermined second audio scene being different from the predetermined first audio scene, and
performing a recognition of the predetermined first audio scene in the mid channel and a recognition of the predetermined second audio scene in the side channel.
2. The method of claim 1 , wherein the obtaining comprises preprocessing the input audio signal into an input data format of each of the plurality of neural networks trained to recognize the audio scene.
3. The method of claim 2 , wherein the preprocessing comprises processing the input audio signal into the input data format of the first neural network and the input data format of the third neural network by downsampling the input audio signal and converting the downsampled audio signal into a time and frequency-based spectrogram.
4. The method of claim 2 , wherein the preprocessing comprises processing the input audio signal into the input data format of the second neural network by reducing a dimensionality of the shape of the spectral envelope of the input audio signal to a lower dimension.
5. The method of claim 1 , wherein the identifying the audio scene probability for the mid channel and the audio scene probability for the side channel further comprises calculating a probability of being the at least one audio scene based on a spectral envelope of a size adjusted by applying a predetermined weight to a spectral envelope preprocessed into an input data format of the second neural network, for each of the mid channel and the side channel.
6. The method of claim 1 , wherein the feature vector comprises at least one of a dominant vector, a mean spectrum power, monophony, or a spectral zero-crossing rate.
7. The method of claim 1 , wherein the identifying the at least one audio scene further comprises calculating a probability for each of the at least one audio scene included in the audio content based on the audio scene probability for the mid channel and the audio scene probability for the side channel.
8. An electronic device for recognizing an audio scene, the electronic device comprising:
a memory storing at least one instruction; and
at least one processor configured to execute the at least one instruction to implement:
a preprocessing module configured to obtain, according to a predetermined criterion, a mid channel and a side channel from an input audio signal;
a scene recognition module configured to identify, from the input audio signal, an audio scene probability for the mid channel and an audio scene probability for the side channel, of at least one audio scene based on audio scene recognition results for the mid channel and the side channel that are respectively obtained by using a plurality of neural networks trained to recognize an audio scene, wherein the plurality of neural networks are duplicated and applied separately on the mid channel and the side channel, to output the audio scene recognition results for the mid channel and the side channel, respectively; and
a decision module configured to identify, based on the audio scene probability for the mid channel and the audio scene probability for the side channel, at least one audio scene included in audio content by using a fourth neural network trained to combine the audio scene probability for the mid channel and the audio scene probability for the side channel, wherein the fourth neural network is configured as a recurrent neural network,
wherein the plurality of neural networks comprises:
a first neural network trained to recognize the audio scene based on a time-frequency shape of an audio signal,
a second neural network trained to recognize the audio scene based on a shape of a spectral envelope of the audio signal, and
a third neural network trained to recognize the audio scene based on a feature vector extracted from the audio signal, and
wherein the scene recognition module is further configured to:
determine a predetermined first audio scene to be recognized in the mid channel and a predetermined second audio scene to be recognized in the side channel, the predetermined second audio scene being different from the predetermined first audio scene, and
perform a recognition of the predetermined first audio scene in the mid channel and a recognition of the predetermined second audio scene in the side channel.
9. The electronic device of claim 8 , wherein the preprocessing module is further configured to preprocess the input audio signal into an input data format of each of the plurality of neural networks trained to recognize the audio scene.
10. The electronic device of claim 9 , wherein the preprocessing module is further configured to process the input audio signal into the input data format of the first neural network and the input data format of the third neural network by downsampling the input audio signal and converting the downsampled audio signal into a time and frequency-based spectrogram.
11. The electronic device of claim 9 , wherein the preprocessing module is further configured to process the input audio signal into the input data format of the second neural network by reducing a dimensionality of the shape of the spectral envelope of the input audio signal to a lower dimension.
12. The electronic device of claim 8 , wherein the feature vector comprises at least one of a dominant vector, a mean spectrum power, monophony, or a spectral zero-crossing rate.
13. The electronic device of claim 8 , wherein the decision module is further configured to calculate a probability for each of the at least one audio scene included in the audio content based on the audio scene probability for the mid channel and the audio scene probability for the side channel.
14. A non-transitory computer-readable recording medium having recorded thereon a program which, when executed by at least one processor, causes the at least one processor to perform the method of claim 1 .