Apparatus and method for scheduling distributed workflow tasks
A server has a processor and a memory storing instructions executed by the processor to access scheduling tools including an entity workload profile with a work flow tasks schedule and work flow task dependencies. Processed data associated with a work flow task within the entity workload profile is identified. The work flow task dependencies are analyzed to alter the work flow tasks schedule to prioritize work flow tasks that rely upon the processed data.
1. A server, comprising:
a processor; and
a memory storing instructions executed by the processor to:
access scheduling tools including an entity workload profile with a work flow tasks schedule and work flow task dependencies,
identify processed data associated with a work flow task within the entity workload profile,
cache the processed data in cluster memory of a computer cluster,
analyze the work flow task dependencies to alter the work flow tasks schedule to prioritize work flow tasks that rely upon the processed data, the instructions to analyze include instructions
adding downstream jobs that use the processed data to a priority queue to pre-empt other scheduled jobs,
executing downstream jobs as soon as sources are available via in memory storage, such that the downstream jobs run faster by pulling source data from memory and the computer cluster uses fewer resources since the downstream jobs take less time and the source data is not accessed redundantly multiple times, and
selectively transition a work flow task running on a parallel, distributed Map Reduce cluster of computers to a work flow task running on an in-memory cluster of computers, wherein the in-memory cluster of computers has data loaded into cluster memory that is repeatedly queried.
2. The server of claim 1 wherein the scheduling tools include a data profile store with individual data profiles for individual data sources.
3. The server of claim 2 wherein the individual data profiles include column level and data set wide statistics.
4. The server of claim 2 wherein the individual data profiles include total record counts, minimum, maximum and mean values for numeric and date columns and cardinality estimations indicating the number of unique values in a column.
5. The server of claim 2 wherein the individual data profiles include frequency estimations for the most frequent values.
6. The server of claim 2 wherein the individual data profiles include a dataset footprint estimate.
7. The server of claim 2 further comprising instructions executed by the processor to estimate data growth and decay patterns for a work flow task.
8. The server of claim 1 wherein the scheduling tools include a preview engine to process a subset of data associated with a work flow task to develop analytics on the execution of the work flow task in its entirety.
9. The server of claim 1 wherein the scheduling tools include information on cluster resource availability.
10. The server of claim 1 wherein the scheduling tools include an operator composition analyzer to provide fine grain analysis of operators associated with work flow tasks to optimize the scheduling and execution of the operators.
11. The server of claim 1 wherein the scheduling tools include historical task execution profiles.