Selecting a neural network architecture for a supervised machine learning problem
Systems and methods for selecting a neural network for a machine learning problem are disclosed. A method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with a machine learning problem and multiple untrained candidate neural networks for solving the machine learning problem. The method includes computing, for each untrained candidate neural network, at least one expressivity measure capturing an expressivity of the candidate neural network with respect to the machine learning problem. The method includes computing, for each untrained candidate neural network, at least one trainability measure capturing a trainability of the candidate neural network with respect to the machine learning problem. The method includes selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the machine learning problem. The method includes providing an output representing the selected at least one candidate neural network.
1. A system comprising:
processing hardware; and
a memory storing instructions which cause the processing hardware to perform operations comprising:
accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by a trained neural network;
computing, for each untrained candidate neural network, at least one expressivity measure based on data of the machine learning problem space, the expressivity measure capturing an expressivity of the candidate neural network with respect to the machine learning problem, the expressivity measure being computed without training the candidate neural network;
computing, for each untrained candidate neural network, at least one trainability measure based on data of the machine learning problem space, the trainability measure capturing a trainability of the candidate neural network with respect to the machine learning problem;
selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the machine learning problem, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressivity measure exceeding a threshold and the at least one trainability measure within a range; and
providing an output representing the selected at least one candidate neural network.
2. The system of claim 1 , wherein the at least one expressivity measure represents a measure of separation of samples from the machine learning problem space.
3. The system of claim 2 , wherein the measure of separation is a magnitude.
4. The system of claim 2 , wherein the measure of separation is an angle.
5. The system of claim 1 , wherein the at least one trainability measure represents a stochastic gradient descent of weights in the candidate neural network during a first phase of training.
6. The system of claim 1 , the operations further comprising:
training the selected at least one candidate neural network to solve the machine learning problem.
7. The system of claim 6 , the operations further comprising:
running the trained at least one candidate neural network on the machine learning problem space in order to solve the machine learning problem; and
providing a solution to the machine learning problem generated by the trained at least one candidate neural network.
8. A non-transitory machine-readable medium storing instructions which cause one or more machines to perform operations comprising:
accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by a trained neural network;
computing, for each untrained candidate neural network, at least one expressivity measure based on data of the machine learning problem space, the expressivity measure capturing an expressivity of the candidate neural network with respect to the machine learning problem, the expressivity measure being computed without training the candidate neural network;
computing, for each untrained candidate neural network, at least one trainability measure based on data of the machine learning problem space, the trainability measure capturing a trainability of the candidate neural network with respect to the machine learning problem;
selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the machine learning problem, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressivity measure exceeding a threshold and the at least one trainability measure within a range; and
providing an output representing the selected at least one candidate neural network.
9. The machine-readable medium of claim 8 , wherein the at least one expressivity measure represents a measure of separation of samples from the machine learning problem space.
10. The machine-readable medium of claim 9 , wherein the measure of separation is a magnitude.
11. The machine-readable medium of claim 9 , wherein the measure of separation is an angle.
12. The machine-readable medium of claim 8 , wherein the at least one trainability measure represents a stochastic gradient descent of weights in the candidate neural network during a first phase of training.
13. A method comprising:
accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by a trained neural network;
computing, for each untrained candidate neural network, at least one expressivity measure based on data of the machine learning problem space, the expressivity measure capturing an expressivity of the candidate neural network with respect to the machine learning problem, the expressivity measure being computed without training the candidate neural network;
computing, for each untrained candidate neural network, at least one trainability measure based on data of the machine learning problem space, the trainability measure capturing a trainability of the candidate neural network with respect to the machine learning problem;
selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the machine learning problem, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressivity measure exceeding a threshold and the at least one trainability measure within a range; and
providing an output representing the selected at least one candidate neural network.
14. The method of claim 13 , wherein the at least one expressivity measure represents a measure of separation of samples from the machine learning problem space.
15. The method of claim 14 , wherein the measure of separation is a magnitude.
16. The method of claim 14 , wherein the measure of separation is an angle.
17. The method of claim 13 , wherein the at least one trainability measure represents a stochastic gradient descent of weights in the candidate neural network during a first phase of training.