Artificial neural network training
An artificial neural network receives data for the inputs of a perceptron in the artificial neural network. The network determines an average of the data for each of the inputs of the perceptron, determines a standard deviation of the average for each of the inputs of the perceptron, and determines an average of the standard deviations for the perceptron. The network then sets a learning rate for the perceptron equal to the average of the standard deviations, and trains the artificial neural network using the learning rate for the perceptron.
1. A process for training an artificial neural network comprising:
(a) receiving input data for inputs of a perceptron in the artificial neural network;
(b) determining an average of the input data for each of the inputs of the perceptron;
(c) determining a standard deviation of the average for each of the inputs of the perceptron;
(d) determining an average of the standard deviations for the perceptron;
(e) setting a learning rate for the perceptron equal to the average of the standard deviations; and
(f) training the artificial neural network using the learning rate for the perceptron.
2. The process of claim 1 , comprising setting an initial weight for perceptrons in the artificial neural network before executing operations (a)-(e).
3. The process of claim 2 , wherein the initial weight is set to the same value for all the perceptrons in the artificial neural network.
4. The process of claim 3 , wherein the same value is equal to a maximum value that does not cause any perceptron in the network to saturate its activation function.
5. The process of claim 1 , comprising randomizing a plurality of weights prior to training the artificial neural network after the perceptron learning rates have been calculated.
6. The process of claim 1 , comprising executing operations (a)-(e) for each perceptron in each layer of the artificial neural network, thereby generating an individualized learning rate for each perceptron in the artificial neural network; and
training the artificial neural network using the individualized learning rate determined for each perceptron.
7. The process of claim 6 , wherein the same input data are used for each layer of the artificial neural network.
8. A non-transitory computer readable medium comprising instructions that when executed by a processor executes a process comprising:
(a) receiving input data for inputs of a perceptron in an artificial neural network;
(b) determining an average of the input data for each of the inputs of the perceptron;
(c) determining a standard deviation of the average for each of the inputs of the perceptron;
(d) determining an average of the standard deviations for the perceptron;
(e) setting a learning rate for the perceptron equal to the average of the standard deviations; and
(f) training the artificial neural network using the learning rate for the perceptron.
9. The non-transitory computer readable medium of claim 8 , comprising instructions for setting an initial weight for perceptrons in the artificial neural network before executing operations (a)-(e).
10. The non-transitory computer readable medium of claim 9 , wherein the initial weight is set to the same value for all the perceptrons in the artificial neural network.
11. The non-transitory computer readable medium of claim 10 , wherein the same value is equal to a maximum value that does not cause any perceptron in the network to saturate its activation function.
12. The non-transitory computer readable medium of claim 8 , comprising instructions for randomizing a plurality of weights prior to training the artificial neural network after the perceptron learning rates have been calculated.
13. The non-transitory computer readable medium of claim 8 , comprising instructions for executing operations (a)-(e) for each perceptron in each layer of the artificial neural network, thereby generating an individualized learning rate for each perceptron in the artificial neural network; and training the artificial neural network using the individualized learning rate determined for each perceptron.
14. The non-transitory computer readable medium of claim 13 , wherein the same input data are used for each layer of the artificial neural network.
15. A system comprising:
a computer processor; and
a computer memory coupled to the computer processor;
wherein the computer processor is operable for:
(a) receiving input data for inputs of a perceptron in an artificial neural network;
(b) determining an average of the input data for each of the inputs of the perceptron;
(c) determining a standard deviation of the average for each of the inputs of the perceptron;
(d) determining an average of the standard deviations for the perceptron;
(e) setting a learning rate for the perceptron equal to the average of the standard deviations; and
(f) training the artificial neural network using the learning rate for the perceptron.
16. The system of claim 15 , wherein the computer processor is operable for setting an initial weight for perceptrons in the artificial neural network before executing operations (a)-(e).
17. The system of claim 16 , wherein the initial weight is set to the same value for all the perceptrons in the artificial neural network; and wherein the same value is equal to a maximum value that does not cause any perceptron in the network to saturate its activation function.
18. The system of claim 15 , wherein the computer processor is operable for randomizing a plurality of weights prior to training the artificial neural network after the perceptron learning rates have been calculated.
19. The system of claim 15 , wherein the computer processor is operable for executing operations (a)-(e) for each perceptron in each layer of the artificial neural network, thereby generating an individualized learning rate for each perceptron in the artificial neural network; and training the artificial neural network using the individualized learning rate determined for each perceptron.
20. The system of claim 19 , wherein the same input data is used for each layer of the artificial neural network.