Optimizing analytic flows

1. A method of optimizing analytic flows comprising:

sampling, via a processor, source data using a sampling method;

executing, via the processor, a flow over the sampled data;

obtaining, via the processor, runtime statistics from the executed flow;

combining, via the processor, runtime statistics with historical statistics; and

optimizing, via the processor, the flow based on the combination of the runtime statistics and historical statistics.

2. The method of claim 1 , in which the sampling method comprises a reservoir sampling method or a sample rows method.

3. The method of claim 1 , further comprising:

identifying a number of statistical variants to be monitored during the execution of the flow over the sampled data;

identifying a number of source data stores; and

analyzing the source data stores according to a number of data characteristics.

4. The method of claim 1 , further comprising storing in a database the runtime statistics obtained from the executed flow.

5. The method of claim 4 , in which storing in a database the runtime statistics obtained from the executed flow comprises storing the runtime statistics as historical statistics.

6. The method of claim 1 , in which combining runtime statistics with historical statistics comprises assigning an importance ratio, R, comprising a ratio defining the degree at which the runtime statistics and historical statistics contribute to a cost model.

7. The method of claim 6 in which the importance ratio, R, is user-definable.

8. The method of claim 1 , further comprising updating a cost model within a flow optimization module using the combination of the runtime statistics and the historical statistics.

9. The method of claim 1 , further comprising updating a cost estimate within a flow optimization module using the combination of the runtime statistics and the historical statistics.

10. The method of claim 1 , further comprising re-executing the optimized flow over the source dataset.

11. A system for optimizing analytic flows comprising:

a processor;

a memory communicatively coupled to the processor; and

a statistical analysis module stored in the memory to:

obtain a flow, F;

identify a number of statistical variants, S V ;

identify a number of source data stores;

analyze the source data stores to identify a sampling method, S, for sampling the source data;

sample the source data using the sampling method, S to obtain sampled data;

execute the flow, F, using the sampled data;

obtain runtime statistics, S I , for the statistical variants, S V ; and

combine the runtime statistics with historical statistics.

12. The system of claim 11 , further comprising a historical statistics database.

13. The system of claim 11 , further comprising an output device through which a user interface is presented to a user.

14. A computer program product for optimizing analytic flows, the computer program product comprising:

a computer readable storage medium comprising computer usable program code embodied therewith, the computer usable program code comprising:

computer usable program code to, when executed by a processor, design an analytic flow;

computer usable program code to, when executed by a processor, combine runtime statistics of an executed instance of the flow over the sampled data;

computer usable program code to, when executed by a processor, combine the runtime statistics with historical statistics;

computer usable program code to, when executed by a processor, optimize the flow based on the combination of the runtime statistics and historical statistics; and

computer usable program code to, when executed by a processor, re-execute the flow based on the optimized flow over the source dataset.

15. The computer program product of claim 14 , further comprising computer usable program code to, when executed by a processor, present a user interface on an output device for tuning a cost model according to both the runtime statistics and historical statistics.