Computer system for building, training and productionizing machine learning models
A system for building, training and productionizing machine learning models is disclosed. A model training specification is received, and a plurality of sets of hyper-parameters is obtained. Sets of training data and hyper parameter sets are distributed to distributed training systems. Models are trained in parallel using different sets of training data. Models are trained using multiple sets of hyper parameters. A candidate hyper-parameter set is selected, based on a measure of estimated effectiveness of the trained predictive models, and a production predictive model is generated by training a predictive model using the selected candidate hyper-parameter set and the complete set of training data.
1. A system comprising:
a device processor; and
a non-transitory computer readable medium storing instructions executable by the device processor to:
receive a model training specification;
determine a plurality of hyper parameter sets, the plurality of hyper parameter sets comprising a first hyper parameter set and a second hyper parameter set for training a type of predictive model specified in the model training specification;
distribute the first hyper parameter set to each of a first set of distributed training systems;
initiate fetching of raw training data from a plurality of data sources;
initiate merging and preprocessing of the fetched raw training data;
initiate merging of row sets from the merged and preprocessed fetched raw training data;
determine a plurality of folds of training data, each fold including a portion of the merged row sets;
initiate training of a first plurality of predictive models, in parallel, by the first set of distributed training systems, based on the received model training specification, using the first hyper parameter set, and using a different portion of a complete set of training data for each respective one of the first set of distributed training systems;
distribute the second hyper parameter set to each of a second set of distributed training systems;
initiate training of a second plurality of predictive models, in parallel, by the second set of distributed training systems, based on the received model training specification, using the second hyper parameter set, and using a different portion of a complete set of training data for each respective one of the second set of distributed training systems;
select a candidate hyper parameter set of the plurality of hyper parameter sets, based on a measure of estimated effectiveness of each of the first plurality of predictive models and second plurality of predictive models; and
generate a production predictive model by training a predictive model using the selected candidate hyper parameter set and the complete set of training data.
2. The system of claim 1 , wherein the model training specification specifies a machine learning algorithm for training predictive models.
3. The system of claim 1 , wherein the model training specification specifies a hyper-parameter search space for training predictive models.
4. The system of claim 1 , wherein the model training specification specifies a data source for training predictive models.
5. The system of claim 1 , wherein the instructions are executable by the device processor to:
initiate cleaning of the merged and preprocessed fetched raw training data.
6. The system of claim 1 , wherein the instructions are executable by the device processor to:
initiate storing of the merged and preprocessed fetched raw training data in a caching layer.
7. The system of claim 1 , wherein the distributed training systems comprise software containers that are configured based on the received model training specification.
8. The system of claim 1 , wherein the instructions are configured to initiate training of the first plurality of predictive models and second plurality of predictive models in parallel.
9. A non-transitory computer readable medium storing instructions that are executable by a device processor to:
receive a model training specification;
determine a plurality of hyper parameter sets, the plurality of hyper parameter sets comprising a first hyper parameter set and a second hyper parameter set for training a type of predictive model specified in the model training specification;
distribute the first hyper parameter set to each of a first set of distributed training systems;
initiate fetching of raw training data from a plurality of data sources;
initiate merging and preprocessing of the fetched raw training data;
initiate merging of row sets from the merged and preprocessed fetched raw training data;
determine a plurality of folds of training data, each fold including a portion of the merged row sets;
initiate training of a first plurality of predictive models, in parallel, by the first set of distributed training systems, based on the received model training specification, using the first hyper parameter set, and using a different portion of a complete set of training data for each respective one of the first set of distributed training systems;
distribute the second hyper parameter set to each of a second set of distributed training systems;
initiate training of a second plurality of predictive models, in parallel, by the second set of distributed training systems, based on the received model training specification, using the second hyper parameter set, and using a different portion of a complete set of training data for each respective one of the second set of distributed training systems;
select a candidate hyper parameter set of the plurality of hyper parameter sets, based on a measure of estimated effectiveness of each of the first plurality of predictive models and second plurality of predictive models; and
generate a production predictive model by training a predictive model using the selected candidate hyper parameter set and the complete set of training data.
10. The non-transitory computer readable medium of claim 9 , wherein the model training specification specifies a machine learning algorithm for training predictive models.
11. The non-transitory computer readable medium of claim 9 , wherein the model training specification specifies a hyper-parameter search space for training predictive models.
12. The non-transitory computer readable medium of claim 9 , wherein the model training specification specifies a data source for training predictive models.
13. The non-transitory computer readable medium of claim 9 , wherein the instructions are executable by the device processor to:
initiate cleaning of the merged and preprocessed fetched raw training data.
14. The non-transitory computer readable medium of claim 9 , wherein the instructions are executable by the device processor to:
initiate storing of the merged and preprocessed fetched raw training data in a caching layer.
15. The non-transitory computer readable medium of claim 9 , wherein the distributed training systems comprise software containers that are configured based on the received model training specification.
16. The non-transitory computer readable medium of claim 9 , wherein the instructions are configured to initiate training of the first plurality of predictive models and second plurality of predictive models in parallel.
17. A method of building, training and productionizing machine learning models, the method comprising:
receive a model training specification;
determine a plurality of hyper parameter sets, the plurality of hyper parameter sets comprising a first hyper parameter set and a second hyper parameter set for training a type of predictive model specified in the model training specification;
distribute the first hyper parameter set to each of a first set of distributed training systems;
initiate fetching of raw training data from a plurality of data sources;
initiate merging and preprocessing of the fetched raw training data;
initiate merging of row sets from the merged and preprocessed fetched raw training data;
determine a plurality of folds of training data, each fold including a portion of the merged row sets;
initiate training of a first plurality of predictive models, in parallel, by the first set of distributed training systems, based on the received model training specification, using the first hyper parameter set, and using a different portion of a complete set of training data for each respective one of the first set of distributed training systems;
distribute the second hyper parameter set to each of a second set of distributed training systems;
initiate training of a second plurality of predictive models, in parallel, by the second set of distributed training systems, based on the received model training specification, using the second hyper parameter set, and using a different portion of a complete set of training data for each respective one of the second set of distributed training systems;
select a candidate hyper parameter set of the plurality of hyper parameter sets, based on a measure of estimated effectiveness of each of the first plurality of predictive models and second plurality of predictive models; and
generate a production predictive model by training a predictive model using the selected candidate hyper parameter set and the complete set of training data.
18. The method of claim 17 , wherein the model training specification specifies a machine learning algorithm for training predictive models.
19. The method of claim 17 , wherein the model training specification specifies a hyper-parameter search space for training predictive models.
20. The method of claim 17 , wherein the model training specification specifies a data source for training predictive models.