Model learning device, direction of arrival estimation device, model learning method, direction of arrival estimation method, and program
A method and a device for estimating a direction-of-arrival of a sound source and for learning a model are described. Operations for the estimating comprises estimating a reverberation component of an acoustic intensity vector, extracting an angle mask, estimating a time-frequency mask for noise suppression and sound source separation. A first direction-of-arrival of a sound source is estimated by applying the time-frequency mask to the acoustic intensity vector. A second direction-of-arrival of the sound source is estimated by applying the angle mask to the acoustic intensity vector. A cost function of a model is calculated based on the first and second direction-of-arrivals of the sound source and a label. Embodiments further describe updating a parameter of the model based on the cost function for learning the model.
1 . A model learning device comprising:
processing circuitry configured to:
receive a real number spectrogram extracted from a complex spectrogram of acoustic data having a label indicating a sound source direction-of-arrival (“DOA”) for each time when the sound source direction-of-arrival is known, and an acoustic intensity vector (“IV”) extracted from the complex spectrogram as inputs, and output a reverberation component of the estimated acoustic intensity vector;
receive the acoustic intensity vector as an input and extract, as an angle mask, a time-frequency mask for selecting a time-frequency bin having an azimuth angle larger than an azimuth angle derived in a state where noise suppression and sound source separation are not performed;
receive the real number spectrogram, the acoustic intensity vector from which the reverberation component has been subtracted, and the angle mask as inputs, and output a time-frequency mask for noise suppression and sound source separation;
derive a first sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the time-frequency mask to the acoustic intensity vector from which the reverberation component has been subtracted;
derive a second sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the angle mask to the acoustic intensity vector from which the reverberation component has been subtracted; and
calculate a cost function of a model on a basis of the first derived sound source direction-of-arrival, the second derived sound source direction-of-arrival, and the label, and update, based on the calculated cost function of the model, a parameter of the model as training of the model, wherein the model after being trained estimates a sound source direction-of-arrival as an online operation with accuracy, the model is configured as a hybrid of IV-based DOA estimation and a deep neural network (DNN)-based estimation in the online operation, and the DNN represents a recurrent neural network comprising a unidirectional recurrent layer using forward information up to a current time, thereby estimating the sound source direction-arrival with accuracy while excluding feedback according to acoustic data from a future time.
2 . A direction of arrival estimation device comprising:
processing circuitry configured to:
receive a real number spectrogram extracted from a complex spectrogram of acoustic data, and an acoustic intensity vector extracted from the complex spectrogram as inputs, and output a reverberation component of the estimated acoustic intensity vector;
receive the acoustic intensity vector as an input and extracts, as an angle mask, a time-frequency mask for selecting a time-frequency bin having an azimuth angle larger than an azimuth angle derived in a state where noise suppression and sound source separation are not performed;
receive the real number spectrogram, the acoustic intensity vector from which the reverberation component has been subtracted, and the angle mask as inputs, and output a time-frequency mask for noise suppression and sound source separation; and
derive, by a trained model, a sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the time-frequency mask to the acoustic intensity vector from which the reverberation component has been subtracted, wherein the trained model estimates a sound source direction-of-arrival as an online operation with accuracy, the model is configured as a hybrid of IV-based DOA estimation and a deep neural network (DNN)-based estimation in the online operation, and the DNN represents a recurrent neural network comprising a unidirectional recurrent layer using forward information up to a current time, thereby estimating the sound source direction-arrival with accuracy while excluding feedback according to acoustic data from a future time.
3 . A model learning method comprising:
a step of receiving a real number spectrogram extracted from a complex spectrogram of acoustic data having a label indicating a sound source direction-of-arrival for each time when the sound source direction-of-arrival is known, and an acoustic intensity vector extracted from the complex spectrogram as inputs, and outputting a reverberation component of the estimated acoustic intensity vector;
a step of receiving the acoustic intensity vector as an input and extracting, as an angle mask, a time-frequency mask for selecting a time-frequency bin having an azimuth angle larger than an azimuth angle derived in a state where noise suppression and sound source separation are not performed;
a step of receiving the real number spectrogram, the acoustic intensity vector from which the reverberation component has been subtracted, and the angle mask as inputs, and outputting a time-frequency mask for noise suppression and sound source separation;
a step of deriving a sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the time-frequency mask to the acoustic intensity vector from which the reverberation component has been subtracted;
a step of deriving a sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the angle mask to the acoustic intensity vector from which the reverberation component has been subtracted; and
a step of calculating a cost function of a model on a basis of the derived sound source direction-of-arrival and the label, and updating, based on the calculated cost function of the model, a parameter of the model as training of the model, wherein the model after being trained estimates a sound source direction-of-arrival as an online operation with accuracy, the model is configured as a hybrid of IV-based DOA estimation and a deep neural network (DNN)-based estimation in the online operation, and the DNN represents a recurrent neural network comprising a unidirectional recurrent layer using forward information up to a current time, thereby estimating the sound source direction-arrival with accuracy while excluding feedback according to acoustic data from a future time.
4 . A direction of arrival estimation method comprising:
a step of receiving a real number spectrogram extracted from a complex spectrogram of acoustic data, and an acoustic intensity vector extracted from the complex spectrogram as inputs, and outputting a reverberation component of the estimated acoustic intensity vector;
a step of receiving the acoustic intensity vector as an input and extracting, as an angle mask, a time-frequency mask for selecting a time-frequency bin having an azimuth angle larger than an azimuth angle derived in a state where noise suppression and sound source separation are not performed;
a step of receiving the real number spectrogram, the acoustic intensity vector from which the reverberation component has been subtracted, and the angle mask as inputs, and outputting a time-frequency mask for noise suppression and sound source separation; and
a step of deriving, by a trained model, a sound source direction-of-arrival on a basis of an acoustic intensity vector obtained by applying the time-frequency mask to the acoustic intensity vector from which the reverberation component has been subtracted, wherein the trained model estimates a sound source direction-of-arrival as an online operation with accuracy, the model is configured as a hybrid of IV-based DOA estimation and a deep neural network (DNN)-based estimation in the online operation, and the DNN represents a recurrent neural network comprising a unidirectional recurrent layer using forward information up to a current time, thereby estimating the sound source direction-arrival with accuracy while excluding feedback according to acoustic data from a future time.
5 . A computer-readable non-transitory recording medium storing a computer-executable program instructions that when executed by a processor cause a computer to execute operations of the processing circuitry as configured in the model learning device according to claim 1 .
6 . A computer-readable non-transitory recording medium storing a computer-executable program instructions that when executed by a processor cause a computer to execute operations of the processing circuitry as configured in the direction of arrival estimation device according to claim 2 .