Computing device for attention-based joint training with noise suppression model for sound event detection technology robust against noise environment and method of thereof
View Patent ↗Various embodiments relate to a computing device for attention-based joint training with a noise suppression model for a sound event detection (SED) technology that is robust against a noise environment and a method thereof, and are intended to improve SED performance that is robust against a noise environment by using a joint model in which a noise suppression model and an SED model have been jointed. According to various embodiments, an attention scheme and a weight freezing scheme are added to the joint model.
1. A computing device for a sound event detection (SED) technology robust against a noise environment, the computing device comprising:
a memory;
an input module configured to obtain a sound event input; and
a processor connected to the memory and the input module and configured to execute at least one instruction stored in the memory,
wherein the processor is configured to obtain a joint model for performing noise suppression and SED on the sound event input and identify a type of the sound event input by using the joint model, and
a noise suppression model and an SED model are jointed as the joint model,
wherein the joint model is a model generated by further being trained after the noise suppression model and the SED model are joined,
wherein the SED model in the joint model comprises a plurality of feature extraction layers to which an output of the noise suppression model in the joint model is input, and a classification layer disposed at an end of the plurality of feature extraction layers, and
wherein weights of the classification layer are fixed and not updated during the training after the noise suppression model and the SED model are joined.
2. The computing device of claim 1 , wherein each of the noise suppression model and the SED model has been pre-trained before the noise suppression model and the SED model are joined.
3. The computing device of claim 1 , wherein the joint model comprises a plurality of attention modules connected to the plurality of feature extraction layers, respectively, and configured to incorporate the output of the noise suppression model into the plurality of feature extraction layers, respectively.
4. The computing device of claim 1 , wherein the noise suppression model is implemented based on a deep neural network (DNN),
wherein the SED model is implemented based on a convolutional recurrent neural network (CRNN),
wherein the plurality of feature extraction layers comprise convolution layers, respectively, and
wherein the classification layer comprises a fully-connected layer.
5. A method of a computing device for a sound event detection (SED) technology robust against a noise environment, the method comprising:
obtaining a sound event input;
obtaining a joint model for performing noise suppression and SED on the sound event input; and
identifying a type of the sound event input by using the joint model,
wherein a noise suppression model and an SED model are jointed as the joint model,
wherein the joint model is a model generated by further being trained after the noise suppression model and the SED model are joined,
wherein the SED model in the joint model comprises a plurality of feature extraction layers to which an output of the noise suppression model in the joint model is input, and a classification layer disposed at an end of the plurality of feature extraction layers, and
wherein weights of the classification layer are fixed and not updated during the training after the noise suppression model and the SED model are joined.
6. The method of claim 5 , wherein each of the noise suppression model and the SED model has been pre-trained before the noise suppression model and the SED model are joined.
7. The method of claim 5 ,
wherein the joint model comprises a plurality of attention modules connected to the plurality of feature extraction layers, respectively, and configured to incorporate the output of the noise suppression model into the plurality of feature extraction layers, respectively.