Knowledge distillation using deep clustering
View Patent ↗Methods and systems for training a neural network include clustering a full set of training data samples into specialized training clusters. Specialized teacher neural networks are trained using respective specialized training clusters of the specialized training clusters. Soft labels are generated for the full set of training data samples using the specialized teacher neural networks. A student model is trained using the full set of training data samples, the specialized training clusters, and the soft labels.
1. A computer-implemented method for training a neural network, comprising:
clustering, by a clustering neural network, a full set of training data samples into a plurality of specialized training clusters according to acoustic conditions estimated through a generated supervector using an average of phone alignment information, representing frame alignment of phonemes, by employing log likelihoods of the acoustic conditions concatenated with averaged log Mel-frequency features for context-dependent phonemes as features estimated by a general teacher neural network;
training a plurality of specialized teacher neural networks using respective specialized training clusters of the plurality of specialized training clusters;
generating soft labels for the full set of training data samples using the plurality of specialized teacher neural networks; and
training a student model using the full set of training data samples, the specialized training clusters, and the soft labels.
2. The method of claim 1 , wherein the training data samples include acoustic waveforms.
3. The method of claim 1 , wherein clustering includes unsupervised deep clustering.
4. The method of claim 1 , further comprising training a general teacher neural network using the full set of training data samples.
5. The method of claim 4 , wherein generating the soft labels further uses the general teacher neural network.
6. The method of claim 5 , wherein training the student model comprises minimizing a loss function:
ℒ
=
-
(
1
-
λ
)
∑
i
q
bl
(
i
|
x
)
log
p
(
i
|
x
)
-
λ
∑
i
q
sp
(
i
|
x
)
log
p
(
i
|
x
)
where λ is a weight hyperparameter, q bl is a set of soft labels from the general teacher neural network, q sp is a set of soft labels from the plurality of specialized teacher neural networks, and p(i|x) is a probability of a class from the student model, with i indicating an index of context-dependent phonemes and x indicating an input signal.
7. The method of claim 4 , wherein the student model is a neural network that has fewer parameters than the general teacher neural network.
8. The method of claim 1 , further comprising performing speech recognition on a new utterance using the trained student model.
9. The method of claim 8 , further comprising performing a natural language task on recognized speech from the new utterance.
10. A non-transitory computer readable storage medium comprising a computer readable program for training a neural network, wherein the computer readable program when executed on a computer causes the computer to:
cluster, by a clustering neural network, a full set of training data samples into a plurality of specialized training clusters according to acoustic conditions estimated through a generated supervector using an average of phone alignment information, representing frame alignment of phonemes, by employing log likelihoods of the acoustic conditions concatenated with averaged log Mel-frequency features for context-dependent phonemes as features estimated by a general teacher neural network;
train a plurality of specialized teacher neural networks using respective specialized training clusters of the plurality of specialized training clusters;
generate soft labels for the full set of training data samples using the plurality of specialized teacher neural networks; and
train a student model using the full set of training data samples, the specialized training clusters, and the soft labels.
11. The non-transitory computer readable storage medium of claim 10 , wherein the training data samples include acoustic waveforms.
12. The non-transitory computer readable storage medium of claim 10 , wherein computer readable program further causes the computer to perform unsupervised deep clustering on the full set of training data samples.
13. The non-transitory computer readable storage medium of claim 10 , wherein computer readable program further causes the computer to train a general teacher neural network using the full set of training data samples.
14. The non-transitory computer readable storage medium of claim 13 , wherein computer readable program further causes the computer to use the general teacher neural network to generate soft labels.
15. The non-transitory computer readable storage medium of claim 14 , wherein computer readable program further causes the computer to a loss function to train the student model:
ℒ
=
-
(
1
-
λ
)
∑
i
q
bl
(
i
|
x
)
log
p
(
i
|
x
)
-
λ
∑
i
q
sp
(
i
|
x
)
log
p
(
i
|
x
)
where λ is a weight hyperparameter, qui is a set of soft labels from the general teacher neural network, q sp is a set of soft labels from the plurality of specialized teacher neural networks, and p(i|x) is a probability of a class from the student model, with i indicating an index of context-dependent phonemes and x indicating an input signal.
16. The non-transitory computer readable storage medium of claim 13 , wherein the student model is a neural network that has fewer parameters than the general teacher neural network.
17. The non-transitory computer readable storage medium of claim 10 , wherein computer readable program further causes the computer to perform speech recognition on a new utterance using the trained student model.
18. A system for training a neural network, comprising:
a hardware processor; and
a memory that stores computer program code which, when executed by the hardware processor, implements:
a student model;
a clustering neural network that clusters a full set of training data samples into a plurality of specialized training clusters according to acoustic conditions estimated through a generated supervector using an average of phone alignment information, representing frame alignment of phonemes, by employing log likelihoods of the acoustic conditions concatenated with averaged log Mel-frequency features for context-dependent phonemes as features estimated by a general teacher neural network;
a plurality of specialized teacher neural networks that together generate soft labels for the full set of training data samples; and
a model trainer that trains the plurality of specialized teacher neural networks using respective specialized training clusters of the plurality of specialized training clusters, and that trains a student model using the full set of training data samples, the specialized training clusters, and the soft labels.