Multi-model joint denoising training
The present disclosure proposes a method and apparatus for multi-model joint denoising training. Multiple models may be obtained. A set of training samples may be denoised through the multiple models. The multiple models may be trained with the set of denoised training samples.
1 . A method for multi-model joint denoising training, the method comprising:
obtaining multiple machine learning models having identical neural network architectures, wherein each machine learning model is initialized with a different combination of training corpora selected from a source training corpus containing reliable labels, at least one translated training corpus generated through machine translation, and at least one generated training corpus created through neural language generation;
jointly denoising a set of training samples through the multiple models by:
for each model, selecting a subset of training samples from the set of training samples by filtering out a predetermined proportion of training samples having highest prediction losses as determined by other models in the multiple models, and
for each training sample, determining a weight for calculating training loss based on consistency of prediction results obtained from the multiple models, wherein inconsistent prediction results indicate noisy training samples;
training the multiple models simultaneously with their respective selected subsets of denoised training samples using the determined weights; and
iteratively updating labels of training samples from translated and generated training corpora based on consensus predictions from the trained multiple models.
2 . The method of claim 1 , wherein the set of training samples comes from a training data set, the training data set including two or two of a source training corpus, at least one translated training corpus, and at least one generated training corpus.
3 . The method of claim 1 , further comprising:
initializing the multiple models with multiple initialization data groups, and the multiple initialization data groups are formed by at least one training corpus in a training data set including the set of training samples.
4 . The method of claim 1 , wherein the denoising a set of training samples comprises:
for each model in the multiple models, selecting training samples for the model from the set of training samples through other models in the multiple models.
5 . The method of claim 4 , wherein the selecting training samples for the model comprises:
filtering out a predetermined proportion of training samples from the set of training samples through the other models; and
determining remaining training samples in the set of training samples as the training samples for the model.
6 . The method of claim 1 , wherein the denoising a set of training samples comprises:
for each training sample in the set of training samples, determining, through the multiple models, a weight of the training sample for calculating a training loss.
7 . The method of claim 6 , wherein the determining a weight of the training sample comprises:
determining the weight according to consistency of multiple prediction results obtained by the multiple models based on the training sample.
8 . The method of claim 1 , further comprising:
further denoising the set of training samples through the trained multiple models.
9 . The method of claim 8 , wherein the further denoising the set of training samples comprises:
updating labels of one or more training samples in the set of training samples through the trained multiple models.
10 . The method of claim 9 , wherein the updating labels comprises, for each training sample in the one or more training samples:
acquiring multiple prediction results obtained by the multiple trained models based on the training sample; and
updating a label of the training sample based on the multiple prediction results.
11 . The method of claim 9 , wherein the one or more training samples come from a translated training corpus and/or a generated training corpus.
12 . The method of claim 1 , wherein the set of training samples corresponds to a training data set, and the training data set is used for performing the denoising and the training for multiple rounds.
13 . The method of claim 1 , wherein:
the set of training samples comes from a training data set, and the training data set is used for performing the denoising and the training for multiple rounds, and
in each round, the set of training samples corresponds to a training data subset among a plurality of training data subsets in the training data set, and the plurality of training data subsets are used for performing the denoising and the training iteratively.
14 . An apparatus for multi-model joint denoising training, comprising:
at least one processor; and
a memory storing computer-executable instructions that, when executed, cause the at least one processor to:
obtain multiple machine learning models having identical neural network architectures, wherein each machine learning model is initialized with a different combination of training corpora selected from a source training corpus containing reliable labels, at least one translated training corpus generated through machine translation, and at least one generated training corpus created through neural language generation;
jointly denoise a set of training samples through the multiple models by:
for each model, select a subset of training samples from the set of training samples by filtering out a predetermined proportion of training samples having highest prediction losses as determined by other models in the multiple models, and
for each training sample, determine a weight for calculating training loss based on consistency of prediction results obtained from the multiple models, wherein inconsistent prediction results indicate noisy training samples;
train the multiple models simultaneously with their respective selected subsets of denoised training samples using the determined weights; and
iteratively update labels of training samples from translated and generated training corpora based on consensus predictions from the trained multiple models.
15 . A computer program product for multi-model joint denoising training, comprising a computer program that is executed by at least one processor for:
obtaining multiple machine learning models having identical neural network architectures, wherein each machine learning model is initialized with a different combination of training corpora selected from a source training corpus containing reliable labels, at least one translated training corpus generated through machine translation, and at least one generated training corpus created through neural language generation;
jointly denoising a set of training samples through the multiple models by:
for each model, selecting a subset of training samples from the set of training samples by filtering out a predetermined proportion of training samples having highest prediction losses as determined by other models in the multiple models, and
for each training sample, determining a weight for calculating training loss based on consistency of prediction results obtained from the multiple models, wherein inconsistent prediction results indicate noisy training samples;
training the multiple models simultaneously with their respective selected subsets of denoised training samples using the determined weights; and
iteratively updating labels of training samples from translated and generated training corpora based on consensus predictions from the trained multiple models.