Soft-tying nodes of a neural network
A machine learning system includes a coach machine learning system that uses machine learning to help a student machine learning system learn its system. By monitoring the student learning system, the coach machine learning system can learn (through machine learning techniques) “hyperparameters” for the student learning system that control the machine learning process for the student learning system. The machine learning coach could also determine structural modifications for the student learning system architecture. The learning coach can also control data flow to the student learning system.
1. A method of training a first neural network, wherein:
the first neural network comprises a plurality of layers;
each layer comprises at least one node;
training data examples for training the first neural network are assigned to one of a plurality of training data example clusters; and
the plurality of training data example clusters comprises a first training data example cluster,
the method comprising backpropagating, by a first computer system, partial derivatives of a cost function through the first neural network, wherein backpropagating the partial derivatives comprises soft-tying, by the first computer system, activation values of a subject node of the first neural network across all training data examples assigned to the first training data example cluster.
2. The method of claim 1 , wherein soft-tying the activation values comprises including, by the first computer system, a regularization term in the cost function for the subject node during the backpropagation of the partial derivatives of the cost function.
3. The method of claim 2 , wherein the regularization term is based on a difference between the activation value of the subject node for a first training data example that is assigned to the first training data example cluster and an average activation value for the subject node across all training data examples assigned to the first training data example cluster.
4. The method of claim 3 , wherein the regularization term is based on a product of a relaxation strength parameter and the difference between the activation value of the subject node for the first training data example and the average activation value for the subject node across all training data examples assigned to the first cluster.
5. The method of claim 1 , wherein the subject node comprises a feature node.
6. The method of claim 5 , further comprising inserting, by the computer system, the feature node into a second neural network.
7. A method of training a first neural network, wherein:
the first neural network comprises a plurality of layers; and
each layer comprises at least one node;
the method comprising backpropagating, by a first computer system, partial derivatives of a cost function through the first neural network, wherein backpropagating the partial derivatives comprises soft-tying, by the first computer system, an activation value of a subject node of the first neural network to an activation value of a second node.
8. The method of claim 7 , wherein soft-tying the activation values comprises including, by the first computer system, a regularization term in the cost function for the subject node and the second node for a common data training example.
9. The method of claim 8 , wherein the regularization term is based on a mean activation value of the subject node and the second node for the common data training example.
10. The method of claim 9 , wherein the regularization term is based on a product of a relaxation strength parameter and the mean activation value.
11. The method of claim 10 , further comprising determining, by a learning coach system, the relaxation strength parameter.
12. The method of claim 11 , wherein determining the relaxation strength parameter comprises changing, by the learning coach, the relaxation strength parameter during the training.
13. The method of claim 8 , wherein the second node is part of the first neural network.
14. The method of claim 8 , wherein the second node is part of a second neural network that is different from the first neural network.
15. The method of claim 8 , wherein soft-tying comprises soft-tying, by the first computer system, the activation value of the subject node of the first neural network to activation values of a plurality of soft-tied nodes, wherein the plurality of soft-tied nodes comprises the second node.
16. The method of claim 15 , wherein the regularization term is based on a mean of the activation values of the subject node and the plurality of soft-tied nodes for the common data training example.
17. The method of claim 16 , wherein:
the second node is part of a second neural network that is different from the first neural network; and
the plurality of soft-tied nodes comprises a third node that is part of a third neural network.
18. The method of claim 17 , wherein:
the second network comprises a classifier; and
the third network comprises a feature detector.
19. A computer system for training a first neural network, wherein:
the first neural network comprises a plurality of layers;
each layer comprises at least one node;
training data examples for training the first neural network are assigned to one of a plurality of training data example clusters; and
the plurality of training data example clusters comprises a first training data example cluster,
the computer system comprising:
a processor core; and
a memory in communication with the processor core, wherein the memory stores computer instructions that, when executed by the processor core, cause the processor to backpropagate partial derivatives of a cost function through the first neural network by soft-tying activation values of a subject node of the first neural network across all training data examples assigned to the first training data example cluster.
20. The computer system of claim 19 , wherein the memory stores further computer instructions that, when executed by the processor core, cause the processor core to soft-tie the activation values by including a regularization term in the cost function for the subject node during the backpropagation of the partial derivatives of the cost function.
21. The computer system of claim 20 wherein the regularization term is based on a difference between the activation value of the subject node for a first training data example that is assigned to the first training data example cluster and an average activation value for the subject node across all training data examples assigned to the first training data example cluster.
22. The computer system of claim 21 , wherein the regularization term is based on a product of a relaxation strength parameter and the difference between the activation value of the subject node for the first training data example and the average activation value for the subject node across all training data examples assigned to the first cluster.
23. The computer system of claim 19 , wherein the subject node comprises a feature node.
24. The computer system of claim 23 , wherein the memory stores further computer instructions that, when executed by the processor core, cause the processor core to insert the feature node into a second neural network.
25. A computer system for training a first neural network, wherein:
the first neural network comprises a plurality of layers; and
each layer comprises at least one node;
the computer system comprising:
a processor core; and
a memory in communication with the processor core, wherein the memory stores computer instructions that, when executed by the processor core, cause the processor to backpropagate partial derivatives of a cost function through the first neural network by soft-tying an activation value of a subject node of the first neural network to an activation value of a second node.
26. The computer system of claim 25 , wherein the memory stores further computer instructions that, when executed by the processor core, cause the processor core to soft-tie the activation values by including a regularization term in the cost function for the subject node and the second node for a common data training example.
27. The computer system of claim 26 , wherein the regularization term is based on a mean activation value of the subject node and the second node for the common data training example.
28. The computer system of claim 27 , wherein the regularization term is based on a product of a relaxation strength parameter and the mean activation value.
29. The computer system of claim 28 , further comprising a learning coach system for determining the relaxation strength parameter.
30. The computer system of claim 29 , wherein the learning coach is for changing the relaxation strength parameter during the training.
31. The computer system of claim 26 , wherein the second node is part of the first neural network.
32. The computer system of claim 26 , wherein the second node is part of a second neural network that is different from the first neural network.
33. The computer system of claim 26 , wherein the memory stores further computer instructions that, when executed by the processor core, cause the processor core to soft-tie by soft-tying the activation value of the subject node of the first neural network to activation values of a plurality of soft-tied nodes, wherein the plurality of soft-tied nodes comprises the second node.
34. The computer system of claim 33 , wherein the regularization term is based on a mean of the activation values of the subject node and the plurality of soft-tied nodes for the common data training example.
35. The computer system of claim 34 , wherein:
the second node is part of a second neural network that is different from the first neural network; and
the plurality of soft-tied nodes comprises a third node that is part of a third neural network.
36. The computer system of claim 35 , wherein:
the second network comprises a classifier; and
the third network comprises a feature detector.