Predicting deep learning scaling
As deep learning application domains grow, a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements is extremely beneficial. Presented herein are large-scale empirical study of error and model size growth as training sets grow. Embodiments of a methodology for this measurement are introduced herein as well as embodiments for predicting other metrics, such as compute-related metrics. It is shown herein that power-law may be used to represent deep model relationships, such as error and training data size. It is also shown that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.
1. A computer-implemented method for generating a learning curve to aid in estimating a metric value for a deep learning model, the method comprising:
splitting a data set into a set of shards such that sizes of the shards span multiple orders of magnitude;
training a set of models on each shard from the set of shards, in which models within the set of models vary in architecture, hyperparameters, or both;
using a validation set to identify a best model for each shard from among the set of models that have been trained, in which the best model for each shard has a validation accuracy that was highest for that shard;
fitting a power-law learning curve model using the validation accuracies and corresponding shard sizes of the best models selected for the shards, in which the power-law learning curve model comprises a power-law region, which follows a small data region and which indicates that new training data provides information that helps improve performance of a model in a predictable manner as indicated by the power-law region of the power-law learning curve model; and
given an output metric, using the power-law region of the fitted power-law learning curve to select a corresponding metric value for use in designing or training a deep learning model.
2. The computer-implemented method of claim 1 further comprising the step of randomly shuffling the data set to maximize likelihood that shards of the data set have similar data distribution to the data set.
3. The computer-implemented method of claim 1 wherein the step of splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude comprises splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude in steps of approximately twice a size of a prior shard's size.
4. The computer-implemented method of claim 1 wherein the selected metric value is improvement in accuracy for the deep learning model given an increase in training data set size for training the deep learning model.
5. The computer-implemented method of claim 1 wherein the selected metric value is one or more compute requirements for the deep learning model.
6. The computer-implemented method of claim 5 wherein a compute requirement for the deep learning model comprises a selected training data set size times a number of parameters of the deep learning model.
7. The computer-implemented method of claim 1 further comprising the step of using at least some of the data in the data set to form the validation set, in which none of the data in the validation set is shared with any of the shards.
8. A system for generating a learning curve to aid in predicting a metric value for a deep learning model, the system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
training a set of models on each shard from a set of shards in which the models from the set of models vary in architecture, hyperparameters, or both, and the set of shards having been generated such that a shard has a shard size and the shard sizes span multiple orders of magnitude;
using a validation set to identify a best model for each shard from among the set of models that have been trained, in which the best model for each shard has a validation accuracy that was highest for that shard;
fitting a power-law learning curve model using the validation accuracies and corresponding shard sizes of the best models selected for the shards, in which the power-law learning curve model comprises a power-law region, which follows a small data region and which indicates that new training data provides information that helps improve performance of a model in a predictable manner as indicated by the power-law region of the power-law learning curve model; and
using the power-law region of the fitted power-law learning curve, to determine a metric value for use in designing or training a deep learning model.
9. The system of claim 8 wherein the set of shards are generated from a data set of training data and the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
randomly shuffling the data set to maximize likelihood that shards of the data set have similar data distribution to the data set; and
splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude.
10. The system of claim 9 wherein the step of splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude comprises splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude in steps of approximately twice a size of a prior shard's size.
11. The system of claim 9 wherein the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
using at least some of the data in the data set to form the validation set, in which none of the data in the validation set is shared with any of the shards.
12. The system of claim 8 wherein the determined metric value is improvement in accuracy for the deep learning model given an increase in training data set size for training the deep learning model.
13. The system of claim 8 wherein the determined metric value is one or more compute requirements for the deep learning model.
14. The system of claim 13 wherein a compute requirement for the deep learning model comprises a determined training data set size times a number of parameters of the deep learning model.
15. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors processor, causes steps to be performed comprising:
training a set of models on each shard from a set of shards in which the models from the set of models vary in architecture, hyperparameters, or both, and the set of shards having been generated such that a shard has a shard size and the shard sizes span multiple orders of magnitude;
using a validation set to identify a best model for each shard from among the set of models that have been trained, in which the best model for each shard has a validation accuracy that was highest for that shard;
fitting a power-law learning curve model using the validation accuracies and corresponding shard sizes of the best models selected for the shards, in which the power-law learning curve model comprises a power-law region, which follows a small data region and which indicates that new training data provides information that helps improve performance of a model in a predictable manner as indicated by the power-law region; and
selecting, using the power-law region of the fitted power-law learning curve, a metric value for use in designing or training a deep learning model.
16. The non-transitory computer-readable medium or media of claim 15 wherein the set of shards are generated from a data set of training data and the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by the at least one processor, or more processors, causes steps to be performed comprising:
randomly shuffling the data set to maximize likelihood that shards of the data set have similar data distribution to the data set; and
splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude.
17. The non-transitory computer-readable medium or media of claim 16 wherein the step of splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude comprises splitting the data set into a set of shards such that the shard sizes span multiple orders of magnitude in steps of approximately twice a size of a prior shard's size.
18. The non-transitory computer-readable medium or media of claim 15 wherein the selected metric value is improvement in accuracy for the deep learning model given an increase in training data set size for training the deep learning model.
19. The non-transitory computer-readable medium or media of claim 15 wherein the selected metric value is one or more compute requirements for the deep learning model.
20. The non-transitory computer-readable medium or media of claim 15 wherein a compute requirement for the deep learning model comprises a selected training data set size times a number of parameters of the deep learning model.