Non-convex optimization by gradient-accelerated simulated annealing

1. A computer-implemented method of training a neural network comprising:

identifying, by one or more processors, a set of parameters for a cost function for the neural network;

applying, by the one or more processors, a gradient optimization of the cost function to generate a set of intermediate optimized parameter values for the cost function;

initializing, by the one or more processors, a simulated annealing optimization of the cost function using the set of intermediate optimized parameter values for the cost function;

applying, by the one or more processors, the simulated annealing optimization of the cost function to generate a set of final optimized parameter values for the cost function, the simulating annealing optimization comprising iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next iteration that is a value of the parameter for a previous iteration added to a value that is randomized within a one percent of a range of permissible values for the parameter;

assigning, by the one or more processors, the set of final optimized parameter values as weights for connections of the neural network; and

using, by the one or more processors, the neural network to perform operations.

2. The computer-implemented method of claim 1 , wherein the set of intermediate optimized parameter values for the cost function correspond to a local minimum or a saddle point of the cost function.

3. The computer-implemented method of claim 1 , wherein the set of final optimized parameter values for the cost function correspond to a global minimum of the cost function.

4. The computer-implemented method of claim 1 , wherein the applying of the gradient optimization of the cost function comprises iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next gradient optimization iteration that is a value of the parameter for a previous gradient optimization iteration added to a predefined factor multiplied by a derivative of the cost function with respect to the parameter.

5. The computer-implemented method of claim 1 , wherein the operations performed by the neural network comprise classifying images.

6. The computer-implemented method of claim 1 , wherein the operations performed by the neural network comprise function approximation.

7. The computer-implemented method of claim 1 , wherein the operations performed by the neural network comprise pattern recognition.

8. The computer-implemented method of claim 1 , wherein the operations performed by the neural network comprise regression analysis.

9. A device comprising:

a memory storage comprising instructions; and

one or more processors in communication with the memory storage, wherein the one or more processors execute the instructions to perform:

identifying a set of parameters for a cost function for a neural network;

applying a gradient optimization of the cost function to generate a set of intermediate optimized parameter values for the cost function;

initializing a simulated annealing optimization of the cost function using the set of intermediate optimized parameter values for the cost function;

applying the simulated annealing optimization of the cost function to generate a set of final optimized parameter values for the cost function, the simulating annealing optimization comprising iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next iteration that is a value of the parameter for a previous iteration added to a value that is randomized within a one percent of a range of permissible values for the parameter;

assigning the set of final optimized parameter values as weights for connections of the neural network; and

using the neural network to perform operations.

10. The device of claim 9 , wherein the set of intermediate optimized parameter values for the cost function correspond to a local minimum or a saddle point of the cost function.

11. The device of claim 9 , wherein the set of final optimized parameter values for the cost function correspond to a global minimum of the cost function.

12. The device of claim 9 , wherein the applying of the gradient optimization of the cost function comprises iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next gradient optimization iteration that is a value of the parameter for a previous gradient optimization iteration added to a predefined factor multiplied by a derivative of the cost function with respect to the parameter.

13. The device of claim 9 , wherein the operations performed by the neural network comprise classifying images.

14. The device of claim 9 , wherein the operations performed by the neural network comprise function approximation.

15. The device of claim 9 , wherein the operations performed by the neural network comprise pattern recognition.

16. A non-transitory computer-readable medium storing computer instructions for training a neural network, that when executed by one or more processors, cause the one or more processors to perform steps of:

identifying a set of parameters for a cost function for the neural network;

applying a gradient optimization of the cost function to generate a set of intermediate optimized parameter values for the cost function;

initializing a simulated annealing optimization of the cost function using the set of intermediate optimized parameter values for the cost function;

applying the simulated annealing optimization of the cost function to generate a set of final optimized parameter values for the cost function, the simulating annealing optimization comprising iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next iteration that is a value of the parameter for a previous iteration added to a value that is randomized within a one percent of a range of permissible values for the parameter;

assigning the set of final optimized parameter values as weights for connections of the neural network; and

using the neural network to perform operations.

17. The non-transitory computer-readable medium of claim 16 , wherein the set of intermediate optimized parameter values for the cost function correspond to a local minimum or a saddle point of the cost function.

18. The non-transitory computer-readable medium of claim 16 , wherein the set of final optimized parameter values for the cost function correspond to a global minimum of the cost function.

19. The non-transitory computer-readable medium of claim 16 , wherein the applying of the gradient optimization of the cost function comprises iteratively defining, for each parameter of the set of parameters for the cost function, a value of the parameter for a next gradient optimization iteration that is a value of the parameter for a previous gradient optimization iteration added to a predefined factor multiplied by a derivative of the cost function with respect to the parameter.

20. The non-transitory computer-readable medium of claim 16 , wherein the operations performed by the neural network comprise classifying images.