Speech signal processing device, speech signal processing method, speech signal processing program, training device, training method, and training program

1. A training apparatus comprising:

a selection unit configured to select a mixed audio signal for training and a plurality of signals relating to processing of an audio signal of a target speaker for training from training data;

a feature conversion unit configured to convert the plurality of signals relating to the processing of the audio signal of the target speaker for training into a plurality of auxiliary features for the plurality of signals using a plurality of auxiliary neural networks corresponding to the plurality of signals;

an audio signal processing unit configured to estimate information regarding processing of an audio signal of the target speaker included in the mixed audio signal for training using a main neural network based on a feature of the mixed audio signal for training and the plurality of auxiliary features; and

an update unit configured to update parameters of neural networks and cause the selection unit, the feature conversion unit, and the audio signal processing unit to repeatedly execute processing until a predetermined criterion is satisfied to set the parameters of the neural networks satisfying the predetermined criterion, wherein the plurality of signals relating to processing of the audio signal of the target speaker are two or more pieces of information of different modalities,

wherein the training apparatus further comprising:

an auxiliary information generation unit configured to generate a weighted sum of the plurality of auxiliary features multiplied by attentions corresponding to the plurality of auxiliary features using a neural network, wherein the audio signal processing unit is configured to receive as an input a second intermediate feature generated by integrating a first intermediate feature obtained by converting the mixed audio signal using a first main neural network included in the main neural network, and the weighted sum and estimate information regarding the audio signal of the target speaker included in the mixed audio signal for training using a second main neural network included in the main neural network, and the auxiliary information generation unit includes:

an attention calculation unit configured to calculate attentions corresponding to the plurality of auxiliary features based on the first intermediate feature and the plurality of auxiliary features; and

an aggregation unit configured to calculate the weighted sum of the plurality of auxiliary features multiplied by the attentions corresponding to the plurality of auxiliary features calculated by the attention calculation unit.

2. The training apparatus according to claim 1 , wherein

the selection unit is configured to select the mixed audio signal for training, the audio signal of the target speaker for training, and video information of speakers at a time of recording the mixed audio signal for training from the training data,

the feature conversion unit includes:

a first auxiliary feature conversion unit configured to convert the audio signal of the target speaker into a first auxiliary feature using a first auxiliary neural network; and

a second auxiliary feature conversion unit configured to convert the video information of the speakers at the time of recording the mixed audio signal for training into a second auxiliary feature using a second auxiliary neural network,

the audio signal processing unit is configured to estimate information regarding the audio signal of the target speaker included in the mixed audio signal for training using the main neural network based on the feature of the mixed audio signal for training, the first auxiliary feature, and the second auxiliary feature, and

the update unit is configured to update parameters of neural networks and cause the selection unit, the first auxiliary feature conversion unit, the second auxiliary feature conversion unit, and the audio signal processing unit to repeatedly execute processing until the predetermined criterion is satisfied to set the parameters of the neural networks satisfying the predetermined criterion.

3. The training apparatus according to claim 2 , wherein the update unit is configured to update parameters of neural networks to allow a weighted sum of a first loss, with respect to a teacher signal, of audio of the target speaker included in the mixed audio signal for training where the audio signal processing unit is estimated using the feature of the mixed audio signal for training, the first auxiliary feature, and the second auxiliary feature, a second loss, with respect to a teacher signal, of audio of the target speaker included in the mixed audio signal for training where the audio signal processing unit is estimated based on the feature of the mixed audio signal for training and the first auxiliary feature, and a third loss, with respect to a teacher signal, of audio of the target speaker included in the mixed audio signal for training that is estimated based on the feature of the mixed audio signal for training and the second auxiliary feature to become smaller.

4. The training apparatus according to claim 1 , wherein

the auxiliary information generation unit further includes:

a normalization unit configured to normalize norms of the plurality of auxiliary features; and

a scaling unit configured to output the weighted sum multiplied by a scale factor calculated based on magnitudes of the norms before normalization to the audio signal processing unit, and

the aggregation unit is configured to calculate a weighted sum of the plurality of normalized auxiliary features multiplied by the attentions corresponding to the plurality of auxiliary features calculated by the attention calculation unit.

5. The training apparatus according to claim 4 , wherein

the audio signal processing unit is configured to estimate the audio signal of the target speaker included in the mixed audio signal for training, and

the update unit is configured to update parameters of neural networks to optimize an objective function based on attentions corresponding to the plurality of auxiliary features calculated by the attention calculation unit, preset desired values of attentions corresponding to the plurality of auxiliary features, the audio signal of the target speaker included in the mixed audio signal for training estimated by the audio signal processing unit, and a teacher signal of audio of the target speaker included in the mixed audio signal for training.

6. The training apparatus according to claim 4 , further comprising a prediction unit configured to predict reliabilities of a plurality of signals relating to processing of the audio signal of the target speaker for training using a neural network based on the plurality of auxiliary features, wherein

the audio signal processing unit is configured to estimate the audio signal of the target speaker included in the mixed audio signal for training, and

the update unit is configured to update parameters of neural networks to optimize an objective function based on the reliabilities of the plurality of signals relating to processing of the audio signal of the target speaker for training predicted by the prediction unit, predetermined reliabilities of the plurality of signals relating to processing of the audio signal of the target speaker for training, the audio signal of the target speaker included in the mixed audio signal for training estimated by the audio signal processing unit, and a teacher signal of audio of the target speaker included in the mixed audio signal for training.

7. The training apparatus according to claim 1 , wherein

the audio signal processing unit is configured to estimate the audio signal of the target speaker included in the mixed audio signal for training, and

the update unit is configured to update parameters of neural networks to optimize an objective function based on attentions corresponding to the plurality of auxiliary features calculated by the attention calculation unit, preset desired values of attentions corresponding to the plurality of auxiliary features, the audio signal of the target speaker included in the mixed audio signal for training estimated by the audio signal processing unit, and a teacher signal of audio of the target speaker included in the mixed audio signal for training.

8. The training apparatus according to claim 1 , further comprising a prediction unit configured to predict reliabilities of a plurality of signals relating to processing of the audio signal of the target speaker for training using a neural network based on the plurality of auxiliary features, wherein

the audio signal processing unit is configured to estimate the audio signal of the target speaker included in the mixed audio signal for training, and

the update unit is configured to update parameters of neural networks to optimize an objective function based on the reliabilities of the plurality of signals relating to processing of the audio signal of the target speaker for training predicted by the prediction unit, predetermined reliabilities of the plurality of signals relating to processing of the audio signal of the target speaker for training, the audio signal of the target speaker included in the mixed audio signal for training estimated by the audio signal processing unit, and a teacher signal of audio of the target speaker included in the mixed audio signal for training.