Creating diverse neural networks with node tying
A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.
1. A method of training a computer-implemented deep neural network to improve performance of the deep neural network by making the deep neural network more diverse, wherein the deep neural network comprises an input layer, an output layer, and one or more inner layers between the input and output layers, wherein the one or more inner layers and the output layer each comprise one or more nodes, such that the deep neural network comprises multiple nodes across, collectively, the one or more inner layers and the output layer, such that the neural network comprises at least a first node and a second node, wherein the first node has a first set of weights for one or more incoming connections to the first node and the second node has a second set of weights for one or more incoming connections to the second node, the method comprising:
iterative training, by a programmed computer system that comprises one or more processor cores, via machine learning, of the deep neural network, wherein the iterative training comprises;
for each training datum item in a set of training data items:
in a forward propagation computation, computing an activation value for the multiple nodes; and
in a back-propagation computation, computing partial derivatives of weights for the multiple nodes with respect to, at least, an objective for the neural network, wherein computing the partial derivatives of the weights for the multiple nodes comprises computing partial derivatives for the first set of weights for the first node and partial derivatives for the second set of weights for the second node; and
updating learned parameters for the nodes based on the partial derivatives, wherein updating the learned parameters for the nodes comprises updating the first and second sets of weights;
wherein the iterative training further comprises training the first and second nodes to be complementary, wherein training the first and second nodes to be complementary comprises:
detecting, by the programmed computer system, a first error-prone training data item in the set of training data items;
detecting that the activation values for the first and second nodes for the first error-prone training data item agree;
imposing a first regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the first error-prone training data item upon detection of the activation values for the first and second nodes agreeing for the first error-prone training data item, where the first regularization cost term penalizes the first and second nodes;
detecting, by the programmed computer system, a second error-prone training data item in the set of training data items;
detecting that the activation values for the first and second nodes for the second error-prone training data item do not agree; and
imposing a second regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the second error-prone training data item upon detection of the activation values for the first and second nodes not agreeing for the second error-prone training data item, where the second regularization cost term rewards the first and second nodes.
2. The method of claim 1 , wherein:
the deep neural network comprises first and second sub-networks;
the first sub-network comprises the first node; and
the second sub-network comprises the second node.
3. The method of claim 1 , wherein:
the deep neural network comprises an ensemble that comprises first and second ensemble members;
the first ensemble member comprises the first node; and
the second ensemble member comprises the second node.
4. The method of claim 3 , wherein the ensemble comprises an ensemble of classifiers, such that the first ensemble member comprises a first classifier and the second ensemble member comprises a second classifier.
5. The method of claim 4 , wherein:
the first node comprises an output node of the first classifier; and
the second node comprises an output node of the second classifier.
6. The method of claim 3 , wherein:
the first node comprises an output node of the first ensemble member; and
the second node comprises an output node of the ensemble member.
7. The method of claim 1 , wherein detecting the first error prone training data item comprises determining whether activation values for the first and second nodes are equal for a training data example.
8. The method of claim 5 , wherein detecting the first error prone training data item comprises determining whether the ensemble made a classification error for a training data item.
9. The method of claim 1 , wherein the first and second regularization costs terms are based on a counter-tying strength hyperparameter.
10. The method of claim 1 , wherein the iterative training further comprises:
detecting by the programmed computer system a non-error-prone training data item in the set of training data items; and
not imposing a regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the non-error-prone training data item.
11. The method of claim 1 , wherein:
the programmed computer system comprises a first set of one or more graphical processing units (GPUs) and second set of one or more GPUs; and
computing the activation values in the forward propagation computations for the multiple nodes comprises processing in parallel, by the first and second sets of GPUs, the training data items in the set of training data items.
12. A computer system for training a deep neural network to improve performance of the deep neural network by making the deep neural network more diverse, wherein the deep neural network comprises an input layer, an output layer, and one or more inner layers between the input and output layers, wherein the one or more inner layers and the output layer each comprise one or more nodes, such that the deep neural network comprises multiple nodes across, collectively, the one or more inner layers and the output layer, such that the neural network comprises at least a first node and a second node, wherein the first node has a first set of weights for one or more incoming connections to the first node and the second node has a second set of weights for one or more incoming connections to the second node, the computer system comprising:
one or more processing units; and
computer memory in communication with the one or more processing units, wherein the computer memory stores instructions executable by the one or more processing units, wherein the instructions, when executed by the one or more processing units, cause the one or more processing units to perform iterative training, via machine learning, of the deep neural network by at least;
for each training datum item in a set of training data items:
in a forward propagation computation, computing an activation value for the multiple nodes; and
in a back-propagation computation, computing partial derivatives of weights for the multiple nodes with respect to, at least, an objective for the neural network, wherein computing the partial derivatives of the weights for the multiple nodes comprises computing partial derivatives for the first set of weights for the first node and partial derivatives for the second set of weights for the second node; and
updating learned parameters for the nodes based on the partial derivatives, wherein updating the learned parameters for the nodes comprises updating the first and second sets of weights; and
wherein the instructions further cause the one or more processing units to perform the iterative training by at least training the first and second nodes to be complementary by at least:
detecting a first error-prone training data item in the set of training data items;
upon detecting that the activation values for the first and second nodes for the first error-prone training data item agree, imposing a first regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the first error-prone training data item upon detection of the activation values for the first and second nodes agreeing for the first error-prone training data item, where the first regularization cost term penalizes the first and second nodes; and
upon detecting that the activation values for the first and second nodes for the first error-prone training data item do not agree, imposing a second regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the first error-prone training data item upon detection of the activation values for the first and second nodes not agreeing for the first error-prone training data item, where the second regularization cost term rewards the first and second nodes.
13. The computer system of claim 12 , wherein:
the deep neural network comprises first and second sub-networks;
the first sub-network comprises the first node; and
the second sub-network comprises the second node.
14. The computer system of claim 12 , wherein:
the deep neural network comprises an ensemble that comprises first and second ensemble members;
the first ensemble member comprises the first node; and
the second ensemble member comprises the second node.
15. The computer system of claim 14 , wherein the ensemble comprises an ensemble of classifiers, such that the first ensemble member comprises a first classifier and the second ensemble member comprises a second classifier.
16. The computer system of claim 15 , wherein:
the first node comprises an output node of the first classifier; and
the second node comprises an output node of the second classifier.
17. The computer system of claim 14 , wherein:
the first node comprises an output node of the first ensemble member; and
the second node comprises an output node of the ensemble member.
18. The computer system of claim 12 , wherein computer memory stores instructions that when executed by the one or more processing units cause the one or more processing unit to detect the first error prone training data item by determining whether activation values for the first and second nodes are equal for a training data example.
19. The computer system of claim 16 , wherein computer memory stores instructions that when executed by the one or more processing units cause the one or more processing unit to detect the first error prone training data item by determining whether the ensemble made a classification error for a training data item.
20. The computer system of claim 12 , wherein the first and second regularization costs terms are based on a counter-tying strength hyperparameter.
21. The computer system of claim 12 , wherein computer memory stores instructions that when executed by the one or more processing units cause the one or more processing unit to perform the iterative training by:
detecting a non-error-prone training data item in the set of training data items; and
not imposing a regularization cost term for the first and second nodes in computing the partial derivatives of the weights for the first and second nodes for the non-error-prone training data item.
22. The computer system of claim 12 , wherein the one or more processing units comprise a first set of one or more graphical processing units (GPUs) and second set of one or more GPUs for, collectively, processing in parallel the training data items in the set of training data items.