Evaluating the value of connecting a selected pair of unconnected nodes of a nodal network

1. A method of training a neural network, the method comprising:

in an initial training of the neural network, training the neural network, by a computer system, wherein the neural network comprises a plurality of layers, including an input layer, an output layer, and at least one hidden layer, wherein each layer comprises at least one node, such that the neural network comprises a plurality of nodes, including a node A and a node B and a node C, wherein node A is not in the output layer and node B is not in the input layer, such that a weighted output from node A is not input to node B in a feedforward phase through the neural network such that the output from node A is not used in computing an activation value for node B in the feedforward phase, and wherein there is no direct connection from node A to node B in the neural network after the initial training and there is direct connection from node C to node B in the neural network after the initial training, such that a weighted output from node C is input to node B in a feedforward phase through the neural network such that the output from node C is used in computing the activation value for node B in the feedforward phase, and wherein the initial training comprises, for each training data item in a first set of training data:

computing, in the feedforward phase, activation values for nodes of the neural network for the training data item, wherein computing the activation for node B comprises computing the activation value for node B based on a weighted value of the activation value for node C due to the direct connection from node C to node B, where a weight for the activation value for node C used in the computation of the activation value for node B is a weight of the direction connection from node C to node B; and

computing, in a back-propagation phase, estimates of partial derivatives for each of nodes A, B and C with respect to an objective for the neural network, wherein the objective is the same for nodes A, B and C in the initial training;

after the initial training, evaluating, by the computer system, whether to add a direct connection from node A in the neural network to node B in the neural network, such that after adding the direct connection from node A to node B, the activation value of node A, weighted by a connection weight for the direct connection from node A to node B, would be used in computation of the activation value for node B, wherein evaluating whether to add the direct connection from node A to node B comprises estimating, by the computer system, an improvement in the objective of the neural network with the direct connection from node A to node B, wherein estimating the improvement in the objective comprises computing, by the computer system, a value of adding the direct connection from node A to node B, wherein computing the value comprises computing, by the computer system, a sum, over a second set of training data, of products of multiple factors, wherein the multiple factors comprise, for each item in the second set of training data, an activation value for node A and a partial derivative of an error loss function with respect to each input to node B; and

adding, by the computer system, the direct connection from node A to node B upon a determination by the computer system that an outcome of estimating the improvement in the objective of the neural network meets a criterion for adding the direct connection, such that, in a subsequent feedforward phase of a subsequent training, weighted outputs from both node A and node C are used in computing the activation value for node B.

2. The method of claim 1 , further comprising, in the subsequent training, training, by the computer system, the neural network after adding the direct connection from node A to node B.

3. The method of claim 2 , wherein the multiple factors further comprise a first hyperparameter.

4. The method of claim 3 , further comprising determining, by a computer-implemented learning coach, the first hyperparameter.

5. The method of claim 3 , wherein the first hyperparameter comprises a data influence weight for each item in the second set of training data.

6. The method of claim 3 , wherein computing the value of adding the direct connection from node A to node B further comprises adding, by the computer system, a second hyperparameter to the sum.

7. The method of claim 1 , wherein the second set of training data comprises a batch of training data for the neural network.

8. The method of claim 1 , wherein the neural network comprises a self-organizing partially ordered network.

9. A computer system comprising:

one or more processor cores; and

a memory in communication with the one or more processor cores, wherein the memory stores software that, when executed by the one or more processor cores, cause the one or more processor cores to:

train a neural network, in an initial training of the neural network, wherein the neural network comprises a plurality of layers, including an input layer, an output layer, and at least one hidden layer, where each layer comprises at least one node, such that the neural network comprises a plurality of nodes, including a node A and a node B and a node C, wherein node A is not in the output layer and node B is not in the input layer, such that a weighted output from node A is not input to node B in a feedforward phase through the neural network such that the output from node A is not used in computing an activation value for node B in the feedforward phase, and wherein there is no direct connection from node A to node B in the neural network after the initial training, and there is direct connection from node C to node B in the neural network after the initial training, such that a weighted output from node C is input to node B in a feedforward phase through the neural network such that the output from node C is used in computing the activation value for node B in the feedforward phase, and wherein the initial training comprises, for each training data item in a first set of training data:

computing, in the feedforward phase, activation values for nodes of the neural network for the training data item, wherein computing the activation for node B comprises computing the activation value for node B based on a weighted value of the activation value for node C due to the direct connection from node C to node B, where a weight for the activation value for node C used in the computation of the activation value for node B is a weight of the direction connection from node C to node B; and

computing, in a back-propagation phase, estimates of partial derivatives for each of nodes A, B and C with respect to an objective for the neural network, wherein the objective is the same for nodes A, B and C in the initial training;

after the initial training, evaluate whether to add a direct connection from node A in the neural network to node B in the neural network by estimating an improvement in an objective of the neural network with the direct connection from node A to node B, such that after adding the direct connection from node A to node B, the activation value of node A, weighted by a connection weight for the direct connection from node A to node B, would be used in computation of the activation value for node B, wherein evaluating whether to add the direct connection from node A to node B comprises estimating the improvement in the objective comprises computing, by the computer system, a value of adding the direct connection from node A to node B, wherein computing the value comprises computing, by the computer system, a sum, over a second set of training data, of products of multiple factors, wherein the multiple factors comprise, for each item in the set of training data, an activation value for node A and a partial derivative of an error loss function with respect to each input to node B; and

add the direct connection from node A to node B to the neural network upon a determination that an outcome of estimating the improvement in the objective of the neural network meets a criterion for adding the direct connection, such that, in a subsequent feedforward phase of a subsequent training, weighted outputs from both node A and node C are used in computing the activation value for node B.

10. The computer system of claim 9 , wherein the memory stores further software that, when executed by the one or more processors, cause the one or more processors to, in the subsequent training, train the neural network after adding the direct connection from node A to node B to the neural network.

11. The computer system of claim 10 , wherein the multiple factors further comprise a first hyperparameter.

12. The computer system of claim 11 , further comprising a computer-implemented learning coach in communication with the neural network, wherein the learning coach is configured to determine the first hyperparameter.

13. The computer system of claim 11 , wherein the first hyperparameter comprises a data influence weight for each item in the set of training data.

14. The computer system of claim 11 , wherein the memory stores further software that, when executed by the one or more processors, cause the one or more processors to compute the value of adding the direct connection from node A to node B by adding a second hyperparameter to the sum.

15. The computer system of claim 9 , wherein the set of training data comprises a batch of training data for the neural network.

16. The computer system of claim 9 , wherein the neural network comprises a self-organizing partially ordered network.