Re-sizing data partitions for ensemble models in a mapreduce framework
Techniques are described for revising data partition size for use in generating predictive models. In one example, a method includes determining an initial number of base model partitions of data from a plurality of data sources; determining an initial base model partition size based at least in part on the initial number of base model partitions; and evaluating the initial base model partition size at least in part with reference to at least one base model partition size reference. The method further includes determining a finalized number of base model partitions based at least in part on the initial base model partition size; determining a revised base model partition size; and generating revised base models based at least in part on the revised base model partition size, including using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the base model partitions.
1. A method comprising:
determining a candidate adjustment factor for a number of base model partitions of data from a plurality of data sources based at least in part on a target size of an ensemble model;
determining an initial number of the base model partitions as a sum of a target size of the ensemble model and the lower of either the candidate adjustment factor or a constant;
determining an initial base model partition size based at least in part on the initial number of base model partitions;
evaluating the initial base model partition size at least in part with reference to at least one base model partition size reference;
determining a finalized number of base model partitions based at least in part on the evaluating of the initial base model partition size at least in part with reference to the at least one base model partition size reference;
determining a revised base model partition size based at least in part on the finalized number of base model partitions; and
generating revised base models based at least in part on the revised base model partition size, wherein generating the revised base models comprises using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the finalized number of base model partitions.
2. The method of claim 1 , wherein the predictive modeling framework comprises a MapReduce framework.
3. The method of claim 1 , wherein using the predictive modeling framework to randomly assign the input data records from the plurality of data sources into the finalized number of base model partitions comprises using the MapReduce framework to randomly assign the input data records from the plurality of data sources into the finalized number of base model partitions.
4. The method of claim 1 , further comprising determining the initial base model partition size by determining a floor of a total number of records in a dataset, minus a size of a holdout sample and a size of a validation sample V, then divided by a number of base partitions.
5. The method of claim 1 , wherein the initial number of base model partitions comprises an estimated target number of base model partitions, the method further comprising determining the initial base model partition size based also at least in part on a minimum number of base model partitions.
6. The method of claim 1 , further comprising generating an output based at least in part on the revised base model partition size.
7. The method of claim 1 , further comprising:
generating, using reduce operations, the plurality of base model partitions based on the plurality of training samples, prior to determining an initial number of base model partitions of data from a plurality of data sources.
8. The method of claim 1 , further comprising generating an ensemble model based on the plurality of revised base models.
9. The method of claim 6 , further comprising:
generating, using map operations for each of the data sources, a validation sample and a holdout sample from the data sources;
merging the holdout samples into a holdout dataset; and
generating, using a reduce operation, a reference model based on the validation samples.
10. The method of claim 9 , further comprising evaluating the ensemble model and the reference model with reference to the holdout dataset.
11. The method of claim 10 , further comprising generating a predictive model based at least in part on the evaluating of the ensemble model and the reference model with reference to the holdout dataset.