Using back propagation computation as data
Computer-implemented, machine-learning systems and methods relate to a neural network having at least two subnetworks, i.e., a first subnetwork and a second subnetwork. The systems and methods estimate the partial derivative(s) of an objective with respect to (i) an output activation of a node in first subnetwork, (ii) the input to the node, and/or (iii) the connection weights to the node. The estimated partial derivative(s) are stored in a data store and provided as input to the second subnetwork. Because the estimated partial derivative(s) are persisted in a data store, the second subnetwork has access to them even after the second subnetwork has gone through subsequent training iterations. Using this information, subnetwork 160 can compute classifications and regression functions that can help, for example, in the training of the first subnetwork.
1. A method of training a first deep neural network, the method comprising:
[a] training the first deep neural network with a first training data example, wherein training the first deep neural network with a first training data example comprises:
a feedforward computation through the first deep neural network; and
a back-propagation computation, with respect to a first network objective function for the first deep neural network, through the first deep neural network at least to a target node in a hidden layer of the first deep neural network, wherein the first network objective function is for measuring quality of output of the first deep neural network, wherein the back-propagation computation comprises, at least, an estimated partial derivative of the first network objective with respect to an output of the target node, an input to the target node, and/or a connection weight for a connection to the target node from another node in the first deep neural network;
[b] training a second deep neural network with a second network objective function that is different from the first network objective function, wherein the second network objective function is for measuring quality of output of the second deep neural network, wherein training the second deep neural network comprises inputting to an input layer of the second deep neural network at least the estimated partial derivative computed in the back-propagation computation for the first deep neural network relative to the target node, such that the second deep neural network is trained to compute an output based on estimated partial derivatives computed in the back-propagation computation for the first deep neural network relative to the target node; and
[c] improving the first deep neural network based on an output of the second deep neural network.
2. The method of claim 1 , wherein:
the first deep neural network comprises at least one connection from the second deep neural network; and
improving the first deep neural network comprises setting a customized hyperparameter for the first deep neural network with one or more estimated partial derivatives computed in the training of the second deep neural network.
3. The method of claim 1 , wherein improving the first deep neural network comprises:
monitoring, with a machine-learning learning coach, the output of the second deep neural network; and
determining, by the machine-learning learning coach, an enhancement to the first deep neural network based on the monitoring.
4. The method of claim 3 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises a revised hyperparameter for the first deep neural network.
5. The method of claim 3 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises a structural change for the first deep neural network.
6. The method of claim 3 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises selective data training for the first deep neural network.
7. The method of claim 1 , wherein the first and second deep neural networks are each a subnetwork of a main neural network.
8. The method of claim 7 , wherein there are no connections from the second deep neural network to the first deep neural network.
9. The method of claim 1 , further comprising storing the estimated partial derivative of the first network objective function in a data store accessible by the second deep neural network.
10. The method of claim 1 , further comprising repeating steps [a] and [b] for a plurality of training data examples, and step [c] comprises improving the first deep neural network based on computations by the second deep neural network in step [b] for the plurality of training data examples.
11. A computer system for training a first deep neural network, the computer system comprising one or more processors programmed to:
train the first deep neural network with a first training data example, wherein training the first deep neural network with a first training data example comprises:
a feedforward computation through the first deep neural network; and
a back-propagation computation, with respect to a first network objective function, through the first deep neural network at least to a target node in a hidden layer of the first deep neural network, wherein the first network objective function is for measuring quality of output of the first deep neural network, wherein the back-propagation computation comprises, at least, an estimated partial derivative of the first network objective with respect to an output of the target node, an input to the target node, and/or a connection weight for a connection to the target node from another node in the first deep neural network;
train a second deep neural network with a second network objective function that is different from the first network objective function, wherein the second network objective function is for measuring quality of output of the second deep neural network, and wherein training the second deep neural network comprises inputting to an input layer of the second deep neural network at least the estimated partial derivative computed in the back-propagation computation for the first deep neural network relative to the target node, such that the second deep neural network is trained to compute an output based on estimated partial derivatives computed in the back-propagation computation for the first deep neural network relative to the target node; and
improve the first deep neural network based on an output of the second deep neural network.
12. The computer system claim of 11 , wherein:
the first deep neural network comprises at least one connection from the second deep neural network; and
the one or more processors are further programmed to improve the first deep neural network by updating one or more learned parameters for the first deep neural network with one or more estimated partial derivatives computed in the training of the second deep neural network.
13. The computer system claim of 11 , the one or more processors are programmed to improve the first deep neural network by:
monitoring, with a machine-learning learning coach, the output of the second deep neural network; and
determining, by the machine-learning learning coach, an enhancement to the first deep neural network based on the monitoring.
14. The computer system of claim 13 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises a revised hyperparameter for the first deep neural network.
15. The computer system of claim 13 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises a structural change for the first deep neural network.
16. The computer system of claim 13 , wherein the enhancement to the first deep neural network determined by the machine-learning learning coach comprises selective data training for the first deep neural network.
17. The computer system of claim 11 , wherein the first and second subnetworks are each a subnetwork of a main neural network.
18. The computer system of claim 17 , wherein there are no connections from the second deep neural network to the first deep neural network.
19. The computer system of claim 11 , further comprising a data store accessible by the second deep neural networks for storing the estimated partial derivative of the first network objective function with respect the output activation of the target node.
20. The computer system of claim 11 , wherein the one or more processors comprise a GPU.