Solution for training a neural network system
Disclosed is a computer-implemented method for training a neural network system including an original neural network and a label generator. The method is based on an idea that the neural network system is trained by a sequence of training steps where at each training step at least one of a plurality of operations is performed and each of the operations gets performed at least once during training of the neural network system. Also disclosed are a neural network system and a computer program product.
1. A computer-implemented method for training a neural network system comprising an original neural network and a label generator, the method comprises:
obtaining a number of training cases comprising input data and wherein at least one training case is labeled; and
training the neural network system by a sequence of training steps where at each training step at least one of the following operations is performed:
training the original network by processing a subset of the labeled training cases with labels,
generating a label with the label generator for a subset of the training cases and training the original network with the generated label, or
updating weights of the label generator based on its current weights and weights of the original network in response to an outcome of the training of the original network, wherein updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises determining each of the weights of the label generator as a weighted average of a corresponding one of the current weights of the label generator and a corresponding one of the weights of the original network,
wherein each of the operations gets performed at least once during training of the neural network system.
2. The computer-implemented method of claim 1 , wherein training the original network comprises minimizing a combination of a classification cost between a predicted label by the original network and the original label and a consistency cost between the predicted label by the original network and the generated label by the label generator.
3. The computer-implemented method of claim 2 , wherein the combination of the classification cost and the consistency cost is a weighted average of the classification cost and the consistency cost.
4. The computer-implemented method of claim 1 , wherein the qweighted average is determined using a smoothing coefficient.
5. The computer-implemented method of claim 1 , wherein the updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises updating the weights of the label generator based only on the current values of the weights of the label generator and corresponding ones of the weights of the original network.
6. The computer-implemented method of claim 1 , wherein the weights of the label generator are initialized to match the weights of the original network.
7. The computer implemented method of claim 1 , wherein generating a label with the label generator comprises mutating at least some outputs of intermediate layers of the label generator.
8. A neural network system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform:
obtaining a number of training cases comprising input data and wherein at least one training case is labeled; and
training the neural network system by a sequence of training steps where at each training step at least one of the following operations is performed:
training the original network by processing a subset of the labeled training cases with labels,
generating a label with the label generator for a subset of the training cases and training the original network with the generated label, or
updating weights of the label generator based on its current weights and weights of the original network in response to an outcome of the training of the original network, wherein updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises determining each of the weights of the label generator as a weighted average of a corresponding one of the current weights of the label generator and a corresponding one of the weights of the original network,
wherein each of the operations gets performed at least once during training of the neural network system.
9. The neural network system of claim 8 , wherein
training the original network comprises minimizing a combination of a classification cost between a predicted label by the original network and the original label and a consistency cost between the predicted label by the original network and the generated label by the label generator.
10. The neural network system of claim 9 , wherein the combination of the classification cost and the consistency cost is a weighted average of the classification cost and the consistency cost.
11. The neural network system of claim 8 , wherein the weighted average is determined using a smoothing coefficient.
12. The neural network system of claim 8 , wherein the updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises updating the weights of the label generator based only on the current values of the weights of the label generator and corresponding ones of the weights of the original network.
13. The neural network system of claim 8 , wherein the weights of the label generator are initialized to match the weights of the original network.
14. The neural network system of claim 8 , wherein generating a label with the label generator comprises mutating at least some outputs of intermediate layers of the label generator.
15. A non-transitory computer-readable storage device including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations comprising:
obtaining a number of training cases comprising input data and wherein at least one training case is labeled; and
training the neural network system by a sequence of training steps where at each training step at least one of the following operations is performed:
training the original network by processing a subset of the labeled training cases with labels,
generating a label with the label generator for a subset of the training cases and training the original network with the generated label, or
updating weights of the label generator based on its current weights and weights of the original network in response to an outcome of the training of the original network, wherein updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises determining each of the weights of the label generator as a weighted average of a corresponding one of the current weights of the label generator and a corresponding one of the weights of the original network,
wherein each of the operations gets performed at least once during training of the neural network system.
16. The non-transitory computer-readable storage device of claim 15 , wherein training the original network comprises minimizing a combination of a classification cost between a predicted label by the original network and the original label and a consistency cost between the predicted label by the original network and the generated label by the label generator.
17. The non-transitory computer-readable storage device of claim 16 , wherein the combination of the classification cost and the consistency cost is a weighted average of the classification cost and the consistency cost.
18. The non-transitory computer-readable storage device of claim 15 , wherein the weighted average is determined using a smoothing coefficient.
19. The non-transitory computer-readable storage device of claim 15 , wherein the updating the weights of the label generator based on its current weights and the weights of the original network in response to an outcome of the training of the original network comprises updating the weights of the label generator based only on the current values of the weights of the label generator and corresponding ones of the weights of the original network.
20. The non-transitory computer-readable storage device of claim 15 , wherein the weights of the label generator are initialized to match the weights of the original network.
21. The non-transitory computer-readable storage device of claim 15 , wherein generating a label with the label generator comprises mutating at least some outputs of intermediate layers of the label generator.