Automated and adaptive design and training of neural networks
Systems and methods are described for developing and using neural network models. An example method of training a neural network includes: oscillating a learning rate while performing a preliminary training of a neural network; determining, based on the preliminary training, a number of training epochs to perform for a subsequent training session, and training the neural network using the determined number of training epochs. The systems and methods can be used to build neural network models that efficiently and accurately handle heterogeneous data.
1 . A system, comprising:
one or more processors; and
at least one computer-readable medium having stored thereon executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
obtaining an initial schedule of a plurality of hyperparameters of a machine learning algorithm, the initial schedule including an initial number of training iterations of a machine learning algorithm and indicating, for each training iteration in the initial number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the plurality of hyperparameters include a learning rate and a momentum coefficient;
adaptively training a machine learning model via application of the machine learning algorithm to the machine learning model and a training data set, wherein adaptively training the machine learning model includes:
performing a plurality of sets of training iterations of the machine learning algorithm, including, for each set of training iterations, (i) varying the values of the plurality of hyperparameters of the machine learning algorithm across the training iterations within the respective set of training iterations, wherein the varying values are based in part on the values indicated by the initial schedule, and (ii) determining an accuracy of the machine learning model corresponding to the respective set of training iterations;
determining, based on a rate of change in the determined accuracy of the machine learning model across two or more of the sets of training iterations, an adapted number of training iterations;
generating an adapted schedule of the plurality of hyperparameters based on the initial schedule and the adapted number of training iterations, the adapted schedule including the adapted number of training iterations and indicating, for each training iteration in the adapted number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the learning rate has a single peak throughout the adapted number of training iterations; and
performing the adapted number of training iterations of the machine learning algorithm in accordance with the adapted schedule of the plurality of hyperparameters of the machine learning algorithm.
2 . The system of claim 1 , wherein performing a training iteration of a set of training of iterations of the plurality of sets of training iterations comprises training the machine learning model on a subset of the training data set corresponding to the training iteration.
3 . The system of claim 1 , wherein one or more first sets of training iterations in the plurality of sets of training iterations correspond, respectively, to a warm up phase and a general training phase of the initial schedule.
4 . The system of claim 1 , wherein varying the values of the plurality of hyperparameters across the training iterations within the respective set of training iterations includes:
increasing and decreasing values of the learning rate across the training iterations within the respective set of training iterations such that the learning rate peaks two or more times across the training iterations.
5 . The system of claim 3 , wherein a second set of training iterations of the plurality of sets of training iterations corresponds to a warm down phase of the initial schedule.
6 . The system of claim 1 , wherein determining the adapted number of training iterations based on the rate of change in the determined accuracy of the machine learning model across the two or more sets of training iterations comprises:
determining that a change in the determined accuracy of the machine learning model across the two or more sets of the sets of training iterations is less than a threshold value, and
calculating the adapted number of training iterations based on a total number of training iterations in the two or more sets of training iterations and in any of the sets of training iterations performed prior to the two or more sets of training iterations.
7 . The system of claim 1 , wherein varying the values of the plurality of hyperparameters of the machine learning algorithm across the training iterations within the respective set of training iterations comprises varying values of a learning rate between a maximum value of the learning rate and a minimum value of the learning rate indicated in the initial schedule.
8 . The system of claim 1 , wherein generating the adapted schedule of the plurality of hyperparameters based on the initial schedule and the adapted number of training iterations comprises stretching or compressing the initial schedule based on a difference between the adapted number of training iterations and the initial number of training iterations.
9 . The system of claim 1 , wherein the operations further include:
providing, for presentation via a user interface, a graph of the adapted schedule indicating the values of the learning rate across the adapted number of training iterations, wherein the learning rate has the single peak throughout the adapted number of training iterations.
10 . The system of claim 1 , wherein operations further include:
prior to performing the adapted number of training iterations of the machine learning algorithm, updating one or more values of a first hyperparameter of the plurality of hyperparameters of the adapted schedule based on input received via a user interface.
11 . A method, comprising:
obtaining, by a data processing system comprising one or more processors coupled with memory, an initial schedule of hyperparameters of a machine learning algorithm, the initial schedule including an initial number of training iterations of a machine learning algorithm and indicating, for each training iteration in the initial number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the plurality of hyperparameters include a learning rate and a momentum coefficient;
adaptively training, by the data processing system, a machine learning model via application of the machine learning algorithm to the machine learning model and a training data set, wherein adaptively training the machine learning model includes:
performing a plurality of sets of training iterations of the machine learning algorithm, including, for each set of training iterations, (i) varying the values of the plurality of hyperparameters of the machine learning algorithm across the training iterations within the respective set of training iterations, wherein the varying values are based in part on the values indicated by the initial schedule, and (ii) determining an accuracy of the machine learning model corresponding to the respective set of training iterations;
determining based on a rate of change in the determined accuracy of the machine learning model across two or more of the sets of training iterations, an adapted number of training iterations;
generating an adapted schedule of the plurality of hyperparameters based on the initial schedule and the adapted number of training iterations, the adapted schedule including the adapted number of training iterations and indicating, for each training iteration in the adapted number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the learning rate has a single peak throughout the adapted number of training iterations; and
performing the adapted number of training iterations of the machine learning algorithm in accordance with the adapted schedule of the plurality of hyperparameters of the machine learning algorithm.
12 . The method of claim 11 , wherein performing a training iteration of a set of training of iterations of the plurality of sets of training iterations comprises training the machine learning model on a subset of the training data set corresponding to the training iteration.
13 . The method of claim 11 , wherein one or more first sets of training iterations in the plurality of sets of training iterations correspond, respectively, to a warm up phase and a general training phase of the initial schedule.
14 . The method of claim 11 , wherein varying the values of the plurality of hyperparameters across the training iterations within the respective set of training iterations includes:
increasing and decreasing values of the learning rate across the training iterations within the respective set of training iterations such that the learning rate peaks two or more times across the training iterations.
15 . The method of claim 13 , wherein a second set of training iterations of the plurality of sets of training iterations corresponds to a warm down phase of the initial schedule.
16 . The method of claim 11 , wherein determining the adapted number of training iterations based on the rate of change in the determined accuracy of the machine learning model across the two or more sets of training iterations comprises:
determining that a change in the determined accuracy of the machine learning model across the two or more sets of the sets of training iterations is less than a threshold value, and
calculating the adapted number of training iterations based on a total number of training iterations in the two or more sets of training iterations and in any of the sets of training iterations performed prior to the two or more sets of training iterations.
17 . The method of claim 11 , wherein varying the values of the plurality of hyperparameters of the machine learning algorithm across the training iterations within the respective set of training iterations comprises varying values of a learning rate between a maximum value of the learning rate and a minimum value of the learning rate indicated in the initial schedule.
18 . The method of claim 11 , wherein generating the adapted schedule of the plurality of hyperparameters based on the initial schedule and the adapted number of training iterations comprises stretching or compressing the initial schedule based on a difference between the adapted number of training iterations and the initial number of training iterations.
19 . A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
obtaining an initial schedule of a plurality of hyperparameters of a machine learning algorithm, the initial schedule including an initial number of training iterations of a machine learning algorithm and indicating, for each training iteration in the initial number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the plurality of hyperparameters include a learning rate and a momentum coefficient;
adaptively training a machine learning model via application of the machine learning algorithm to the machine learning model and a training data set, wherein adaptively training the machine learning model includes:
performing a plurality of sets of training iterations of the machine learning algorithm, including, for each set of training iterations, (i) varying the values of the plurality of hyperparameters of the machine learning algorithm across the training iterations within the respective set of training iterations, wherein the varying values are based in part on the values indicated by the initial schedule, and (ii) determining an accuracy of the machine learning model corresponding to the respective set of training iterations;
determining, based on a rate of change in the determined accuracy of the machine learning model across two or more of the sets of training iterations, an adapted number of training iterations;
generating an adapted schedule of the plurality of hyperparameters based on the initial schedule and the adapted number of training iterations, the adapted schedule including the adapted number of training iterations and indicating, for each training iteration in the adapted number of training iterations, values of the plurality of hyperparameters during the respective training iteration, wherein the learning rate has a single peak throughout the adapted number of training iterations; and
performing the adapted number of training iterations of the machine learning algorithm in accordance with the adapted schedule of the plurality of hyperparameters of the machine learning algorithm.
20 . The non-transitory computer-readable medium of claim 19 , wherein performing a training iteration of a set of training of iterations of the plurality of sets of training iterations comprises training the machine learning model on a subset of the training data set corresponding to the training iteration.