Training of student neural network with switched teacher neural networks
A student neural network may be trained by a computer-implemented method, including: selecting a teacher neural network among a plurality of teacher neural networks, inputting an input data to the selected teacher neural network to obtain a soft label output generated by the selected teacher neural network, and training a student neural network with at least the input data and the soft label output from the selected teacher neural network.
1. A computer-implemented method comprising:
selecting a teacher neural network among a plurality of teacher neural networks,
inputting at least an input data to the selected teacher neural network to obtain a soft label output generated by the selected teacher neural network,
training a student neural network with at least the input data and the soft label output generated by the selected teacher neural network,
increasing a frequency of selecting the teacher neural network based on an accuracy of the soft label output in comparison with the correct data corresponding to the input data versus a frequency of iterations in which selecting the teacher neural network is random, as a number of the iterations increases,
repeating selecting the teacher neural network, inputting the input data to the selected teacher neural network, and training the student neural network.
2. The method of claim 1 , wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
randomly selecting the teacher neural network among the plurality of teacher neural networks.
3. The method of claim 1 , wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
selecting the teacher neural network that outputs the soft label output that is a closest to the correct data corresponding to the input data, among the plurality of teacher neural networks.
4. The method of claim 1 , wherein the input data is audio data and the soft label output is a classification of the audio data.
5. The method of claim 4 , wherein the classification of the audio data identifies phonemes.
6. The method of claim 1 , wherein training the student neural network with at least the input data and the soft label output generated by the selected teacher neural network includes:
training the student neural network with at least the input data, the soft label output generated by the selected teacher neural network, and a correct data corresponding to the input data.
7. The method of claim 1 , further comprising:
before selecting the teacher neural network, training the plurality of teacher neural networks by a server computer,
wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
receiving, by the server computer, the input data from a client computer, and
selecting, by the server computer, the teacher neural network among the plurality of teacher neural networks,
wherein inputting an input data to the selected teacher neural network to obtain the soft label output generated by the selected teacher neural network includes:
inputting, by the server computer, the input data to the selected teacher neural network to obtain the soft label output output from the selected teacher neural network, and
transmitting, by the server computer, the soft label output to the client computer,
wherein training the student neural network with at least the input data and the soft label output generated by the selected teacher neural network includes:
training, by the client computer, the student neural networks with at least the input data and the soft label output generated by the selected teacher neural network.
8. An apparatus comprising:
a processor or a programmable circuitry; and
one or more computer readable mediums collectively including instructions that, when executed by the processor or the programmable circuitry, cause the processor or the programmable circuitry to perform operations including:
selecting a teacher neural network among a plurality of teacher neural networks,
inputting at least an input data to the selected teacher neural network to obtain a soft label output generated by the selected teacher neural network,
training a student neural network with at least the input data and the soft label output generated by the selected teacher neural network,
increasing a frequency of selecting the teacher neural network based on an accuracy of the soft label output in comparison with the correct data corresponding to the input data versus a frequency of iterations in which selecting the teacher neural network is random, as a number of iterations increases, and
repeating selecting the teacher neural network, inputting the input data to the selected teacher neural network, and training the student neural network.
9. The apparatus of claim 8 , wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
randomly selecting the teacher neural network among the plurality of teacher neural networks.
10. The apparatus of claim 8 , wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
selecting the teacher neural network that outputs the soft label output that is a closest to the correct data corresponding to the input data, among the plurality of teacher neural networks.
11. The apparatus of claim 8 , wherein the input data is audio data and the soft label output is a classification of the audio data.
12. A computer program product including one or more computer readable storage mediums collectively storing program instructions that are executable by a processor or programmable circuitry to cause the processor or programmable circuitry to perform operations comprising:
selecting a teacher neural network among a plurality of teacher neural networks,
inputting at least an input data to the selected teacher neural network to obtain a soft label output generated by the selected teacher neural network,
training a student neural network with at least the input data and the soft label output generated by the selected teacher neural network,
increasing a frequency of selecting the teacher neural network based on an accuracy of the soft label output in comparison with the correct data corresponding to the input data versus a frequency of iterations in which selecting the teacher neural network is random, as a number of iterations increases, and
repeating selecting the teacher neural network, inputting the input data to the selected teacher neural network, and training the student neural network.
13. The computer program product of claim 12 , wherein selecting the teacher neural network among the plurality of teacher neural networks includes:
randomly selecting the teacher neural network among the plurality of teacher neural networks.
14. The computer program product of claim 12 , wherein the selecting the teacher neural network among the plurality of teacher neural networks includes:
selecting the teacher neural network that outputs the soft label output that is a closest to the correct data corresponding to the input data, among the plurality of teacher neural networks.