Signal processing apparatus, learning apparatus, signal processing method, learning method and program
A signal processing device according to an embodiment of the present invention includes: a conversion unit configured to convert an input mixed acoustic signal into a plurality of first internal states, a weighting unit configured to generate a second internal state which is a weighted sum of the plurality of first internal states based on auxiliary information regarding an acoustic signal of a target sound source when the auxiliary information is input, and generate the second internal state by selecting one of the plurality of first internal states when the auxiliary information is not input, and a mask estimation unit configured to estimate a mask based on the second internal state.
1. A signal processing device comprising a processor configured to execute operations comprising:
converting an input mixed acoustic signal into a plurality of first internal states of using the input mixed acoustic signal to generate separate acoustic signals according to sound sources;
generating a second internal state independent of the presence of auxiliary information, wherein:
when there is the auxiliary information, the generating further comprises:
receiving the auxiliary information, wherein the auxiliary information includes a feature value of a target sound source, and
determining the second internal state according to a weighted sum of the plurality of first internal states, and,
when the auxiliary information is not input, the generating further comprises determining the second internal state by selecting one of the plurality of first internal states; and
estimating a mask based on the second internal state.
2. The signal processing device according to claim 1 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
3. The signal processing device according to claim 1 , wherein the first internal states and the second internal states represent internal states of a neural network, and
the neural network has been trained so that an error is within a predetermined threshold, and the error is between an acoustic signal obtained by applying the estimated mask to a mixed acoustic signal and a correct acoustic signal of a sound source included in the mixed acoustic signal.
4. The signal processing device according to claim 3 , wherein
converting further comprises converting the input mixed acoustic signal into I first internal states, and
generating further comprises generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and generates the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
5. A learning device for signal processing, the learning device comprising a processor configured to execute operations comprising:
converting an input training mixed acoustic signal into a plurality of first internal states of using the input mixed acoustic signal to generate separate acoustic signals according to sound sources using a neural network;
generating a second internal state independent of the presence of auxiliary information, wherein:
when there is the auxiliary information, the generating further comprises:
receiving the auxiliary information as input, wherein the auxiliary information includes a feature value of a target sound source, and
determining the second internal state according to a weighted sum of the plurality of first internal states using the neural network, and
when the auxiliary information is not input, the generating further comprises determining the second internal state by selecting one of the plurality of first internal states;
estimating a mask based on the second internal state using the neural network; and
updating:
a first parameter of the neural network used for converting the input training mixed acoustic signal,
a second parameter of the neural network used for generating the second internal state, and
a third parameter of the neural network used for estimating the mask based on a comparison result between an acoustic signal obtained by applying the estimated mask to the training mixed acoustic signal and a correct acoustic signal of a sound source included in the training mixed acoustic signal.
6. The learning device according to claim 5 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
7. The learning device according to claim 5 , wherein
the first internal states and the second internal states represent internal states of a neural network, and
the first neural network has been trained so that an error is within a predetermined threshold, and the error is between an acoustic signal obtained by applying the estimated mask to a mixed acoustic signal and a correct acoustic signal of a sound source included in the mixed acoustic signal.
8. The learning device according to claim 7 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and generates the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
9. The learning device according to claim 5 , wherein updating further comprise updating the parameter in consideration of both a loss when the auxiliary information is input and a loss when the auxiliary information is not input.
10. The learning device according to claim 9 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
11. The learning device according to claim 9 , wherein
the first internal states and the second internal states represent internal states of a neural network, and
the 0 neural network has been trained so that an error is within a predetermined threshold, and the error is between an acoustic signal obtained by applying the estimated mask to a mixed acoustic signal and a correct acoustic signal of a sound source included in the mixed acoustic signal.
12. The learning device according to claim 11 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and generates the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
13. The learning device according to claim 5 , the processor further configured to execute operations comprising:
updating:
a first parameter of the neural network used for the converting,
a second parameter of the neural network used for the generating, and
a third parameter of the neural network for the estimating based on a comparison result between an acoustic signal obtained by applying the estimated mask to the training mixed acoustic signal and the correct acoustic signal of the sound source included in the training mixed acoustic signal.
14. The learning device according to claim 13 , wherein
the converting uses a first neural network,
the generating uses a second neural network,
the estimating uses a third neural network, and
the first neural network has been trained so that an error is within a predetermined threshold, and the error is between an acoustic signal obtained by applying the estimated mask to a mixed acoustic signal and a correct acoustic signal of a sound source included in the mixed acoustic signal.
15. The learning device according to claim 13 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
16. The method according to claim 14 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
17. A method for processing signals, comprising:
converting, an input mixed acoustic signal into a plurality of first internal states of using the input mixed acoustic signal to generate separate acoustic signals according to sound sources;
generating a second internal state independent of the presence of auxiliary information, wherein:
when there is auxiliary information, the generating further comprises:
receiving the auxiliary information, wherein the auxiliary information includes a feature value of a target sound source, and
determining the second internal state according to a weighted sum of the plurality of first internal states;
when the auxiliary information is not input, the generating further comprises determining the second internal state by selecting one of the plurality of first internal states; and
estimating a mask based on the second internal state.
18. The method according to claim 17 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.
19. The method according to claim 17 , wherein
the first internal states and the second internal states represent internal states of a neural network, and
the neural network has been trained so that an error is within a predetermined threshold, and the error is between an acoustic signal obtained by applying the estimated mask to a mixed acoustic signal and a correct acoustic signal of a sound source included in the mixed acoustic signal.
20. The method according to claim 19 , wherein
the converting further comprises converting the input mixed acoustic signal into I first internal states, and
the generating further comprises:
generating the second internal state by applying an I-dimensional weight vector estimated based on the I first internal states and the auxiliary information to the I first internal states when the auxiliary information is input, and
generating the second internal state by applying an I-dimensional unit vector in which an i-th (where i=1, . . . , I) element is 1 and other elements are 0 to the I first internal states when the auxiliary information is not input.