AGGRESSIVE DEVELOPMENT WITH COOPERATIVE GENERATORS
Various systems and methods are described herein for improving the aggressive development of machine learning systems. In machine learning, there is always a trade-off between allowing a machine learning system to learn as much as it can from training data and overfitting on the training data. This trade-off is important because overfitting usually causes performance on new data to be worse. However, various systems and methods can be utilized to separate the process of detailed learning and knowledge acquisition and the process of imposing restrictions and smoothing estimates, thereby allowing machine learning systems to aggressively learn from training data, while mitigating the effects of overfitting on the training data.
1 . A computer-implemented method of restricting learning by a neural network, wherein the neural network comprises a first node, the method comprising:
training, by a computer system, the neural network on a training data set; and
adding, by the computer system during training, a relaxation term to a back-propagated derivative of an objective function with respect to a computed value of each of the first node of the neural network and a second node, the relaxation term adding a penalty to a cost function of each of the computed value of the first node and the computed value of the second node according to whether the computed values for the first and second nodes diverge from each other.
2 . The method of claim 1 , further comprising controlling, by the computer system, a weight of the relaxation term via a hyperparameter.
3 . The method of claim 2 , wherein the hyperparameter comprises a multiplicative scale factor applied to the relaxation term.
4 . The method of claim 1 , wherein the neural network comprises the second node.
5 . The method of claim 1 , wherein:
the neural network comprises a first neural network; and
a second neural network comprises the second node.
6 . The method of claim 1 , further comprising:
adding, by the computer system during training, a second relaxation term to a back-propagated derivative of an objective function with respect to a computed value of each of the first node and a third node, the relaxation term adding a penalty to a cost function of each of the first node and the third node according to whether the computed values for the first and third nodes diverge from each other.
7 . The method of claim 1 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the computed value of each of the first node and the second node for each data example in the training data set.
8 . The method of claim 1 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the computed value of each of the first node and the second node for a subset of data examples in the training data set.
9 . The method of claim 8 , wherein the subset of data examples for which the relaxation term is added corresponds to a classification category into which the training data set has been divided.
10 . The method of claim 8 , wherein the subset of data examples for which the relaxation term is added corresponds to a data cluster into which the training set has been divided by a machine learning system trained to cluster the training data set according to cluster assignment values.
11 . The method of claim 1 , wherein the computed value comprises a connection weight of each of the first node and the second node.
12 . The method of claim 11 , wherein the relaxation term requires that the connection weights of the first node and the second node be equal.
13 . A computer system for restricting learning of a neural network comprising a first node, the computer system comprising:
one or more processor cores;
one or more memories coupled to the one or more processor cores, the one or more memories storing the neural network and instructions that, when executed by the one or more processor cores, cause the computer system to:
train the neural network on a training data set; and
add, during training, a relaxation term to a back-propagated derivative of an objective function with respect to a computed value of each of the first node of the neural network and a second node, the relaxation term adding a penalty to a cost function of each of the computed value of the first node and the computed value of the second node according to whether the computed values for the first and second nodes diverge from each other.
14 . The computer system of claim 13 , wherein the instructions, when executed by the one or more processor cores, further cause the computer system to control a weight of the relaxation term via a hyperparameter.
15 . The computer system of claim 14 , wherein the hyperparameter comprises a multiplicative scale factor applied to the relaxation term.
16 . The computer system of claim 13 , wherein the neural network comprises the second node.
17 . The computer system of claim 13 , wherein:
the neural network comprises a first neural network; and
a second neural network comprises the second node, the second neural network stored by the one or more memories.
18 . The computer system of claim 13 , wherein the instructions, when executed by the one or more processor cores, further cause the computer system to:
add, during training, a second relaxation term to a back-propagated derivative of an objective function with respect to a computed value of each of the first node and a third node, the relaxation term adding a penalty to a cost function of each of the first node and the third node according to whether the computed values for the first and third nodes diverge from each other.
19 . The computer system of claim 13 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the computed value of each of the first node and the second node for each data example in the training data set.
20 . The computer system of claim 13 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the computed value of each of the first node and the second node for a subset of data examples in the training data set.
21 . The computer system of claim 20 , wherein the subset of data examples for which the relaxation term is added corresponds to a classification category into which the training data set has been divided.
22 . The computer system of claim 20 , wherein the subset of data examples for which the relaxation term is added corresponds to a data cluster into which the training set has been divided by a machine learning system trained to cluster the training data set according to cluster assignment values.
23 . The computer system of claim 13 , wherein the computed value comprises a connection weight of each of the first node and the second node.
24 . The computer system of claim 23 , wherein the relaxation term requires that the connection weights of the first node and the second node be equal.
25 . The computer system of claim 13 , further comprising:
a plurality of computer nodes interconnected via connections having varying data bandwidths;
wherein the one or more processor cores and the one or more memories are distributed across the computer nodes;
wherein the memory of each of the plurality of computer nodes stores instructions that, when executed by the one or more processor cores, cause the computer nodes to transmit data between the computer nodes according to the data bandwidth associated with respective connections between the computer nodes.
26 . A computer-implemented method of restricting learning by a neural network, wherein the neural network comprises a first node, the method comprising:
training, by a computer system, the neural network on a training data set; and
adding, by the computer system during training, a relaxation term to a back-propagated derivative of an objective function with respect to an activation value of each of the first node of the neural network and a second node, the relaxation term adding a penalty to a cost function for each of the first node and the second node according to whether the activation values for the first and second nodes diverge from each other.
27 . The method of claim 26 , further comprising controlling, by the computer system, a weight of the relaxation term via a hyperparameter.
28 . The method of claim 27 , wherein the hyperparameter comprises a multiplicative scale factor applied to the relaxation term.
29 . The method claim 26 , wherein the neural network comprises the second node.
30 . The method claim 26 , wherein:
the neural network comprises a first neural network; and
a second neural network comprises the second node.
31 . The method of claim 26 , further comprising:
adding, by the computer system during training, a second relaxation term to a back-propagated derivative of an objective function with respect to a learned parameter of each of the first node and a third node, the relaxation term adding a penalty to a cost function of each of the first node and the third node according to whether the learned parameters for the first and third nodes diverge from each other.
32 . The method of claim 26 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the activation value of each of the first node and the second node for each data example in the training data set.
33 . The method of claim 26 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the activation value of each of the first node and the second node for a subset of data examples in the training data set.
34 . The method of claim 33 , wherein the subset of data examples for which the relaxation term is added corresponds to a classification category into which the training data set has been divided.
34 . (canceled)
35 . A computer system for restricting learning of a neural network comprising a first node, the computer system comprising:
one or more processor cores;
one or more memories coupled to the one or more processor cores, the one or more memories storing the neural network and instructions that, when executed by the one or more processor cores, cause the computer system to:
train the neural network on a training data set; and
add, during training, a relaxation term to a back-propagated derivative of an objective function with respect to an activation value of each of the first node of the neural network and a second node, the relaxation term adding a penalty to a cost function for each of the first node and the second node according to whether the activation values for the first and second nodes diverge from each other.
36 . The computer system of claim 35 , wherein the instructions, when executed by the one or more processor cores, further cause the computer system to control a weight of the relaxation term via a hyperparameter.
37 . The computer system of claim 36 , wherein the hyperparameter comprises a multiplicative scale factor applied to the relaxation term.
38 . The computer system claim 35 , wherein the neural network comprises the second node.
39 . The computer system claim 35 , wherein:
the neural network comprises a first neural network; and
a second neural network comprises the second node, the second neural network stored by the one or more memories.
40 . The computer system of claim 35 , wherein the instructions, when executed by the one or more processor cores, further cause the computer system to:
add, during training, a second relaxation term to a back-propagated derivative of an objective function with respect to a learned parameter of each of the first node and a third node, the relaxation term adding a penalty to a cost function of each of the first node and the third node according to whether the learned parameters for the first and third nodes diverge from each other.
41 . The computer system of claim 35 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the activation value of each of the first node and the second node for each data example in the training data set.
42 . The computer system of claim 35 , wherein the relaxation term is added to the back-propagated derivative of the objective function with respect to the activation value of each of the first node and the second node for a subset of data examples in the training data set.
43 . The computer system of claim 42 , wherein the subset of data examples for which the relaxation term is added corresponds to a classification category into which the training data set has been divided.
44 . The computer system of claim 42 , wherein the subset of data examples for which the relaxation term is added corresponds to a data cluster into which the training set has been divided by a machine learning system trained to cluster the training data set according to cluster assignment values.
45 . The computer system of claim 35 , further comprising:
a plurality of computer nodes interconnected via connections having varying data bandwidths;
wherein the one or more processor cores and the one or more memories are distributed across the computer nodes;
wherein the memory of each of the plurality of computer nodes stores instructions that, when executed by the one or more processor cores, cause the computer nodes to transmit data between the computer nodes according to the data bandwidth associated with respective connections between the computer nodes.
46 - 444 . (canceled)
445 . The method of claim 1 , wherein the computed value comprises a learned parameter.
446 . The computer system of claim 13 , wherein the computed value comprises a learned parameter.