Apparatus and method for analytical optimization through computational pushdown
An analytic is deployed at the task driver, and the analytic performs one or more functions. At the task driver, an execution plan for the analytic is determined based at least in part upon an analysis of the functions. The execution plan determines a location for execution of each of the functions, and the location is one of the computational node or the data node. The functions are executed according to the execution plan.
1. A system that is configured to efficiently execute an analytic, the system comprising:
a computation cluster that includes a computational node and a task driver;
an analytic deployed at the task driver, the analytic performing an algorithm utilizing a first function and a second function, the first function and the second function performing a mathematical operation on data, performing a read operation or performing a write operation;
a data cluster including a remotely-located data node, wherein the computation cluster is remotely located from the data cluster, wherein the physical location of the remotely-located data cluster is different from the physical location of the computational cluster;
wherein the first function is different from the second function;
wherein the task driver is configured to determine an execution plan for the algorithm of the analytic based upon an amount of data required by the first and second functions, the physical location of the data required by the first and second functions, and locations of programmatic features utilized by the first and second functions, the execution plan determining a location for execution of the first and second functions;
wherein the execution plan specifies that the first function is executed on the computational node by fetching data required by the first function from the remotely located data node or writing data produced by the first function to the remotely located data node; and
wherein the execution plan specifies that the second function is executed at the remotely located data node to create a result, and the result is transmitted to the task driver at the computational node, wherein the first function is executed by fetching data required by the function from the remotely located data node or writing data produced by the function to the remotely located data node.
2. The system of claim 1 , wherein the computational node includes one or more processors.
3. The system of claim 1 , wherein the data node includes one or more processors.
4. The system of claim 1 , wherein some of the functions are executed by sending one or more queries to the data cluster.
5. The system of claim 1 , wherein the data cluster includes a router.
6. The system of claim 1 , wherein a third function is executed at the computation node creating first results and a fourth function is executed at the data node creating second results.
7. The system of claim 6 , wherein the first results and the second results are aggregated by the task driver.
8. A method for efficiently executing an analytic, portions of the analytic being executed at a computation cluster that includes a computational node and a task driver, and a data cluster that includes a data node, wherein the computation cluster is remotely located from the data cluster, wherein the physical location of the remotely-located data cluster is different from the physical location of the computational cluster, the method comprising:
deploying an analytic at the task driver, the analytic performing an algorithm utilizing a first function and a second function, the first function and the second function performing a mathematical operation on data, performing a read operation or performing a write operation, wherein the first function is different from the second function;
at the task driver, determining an execution plan for the algorithm of the analytic based upon an amount of data required by first and second functions, the physical location of the data required by the first and second functions, and locations of programmatic features utilized by the first and second functions, the execution plan determining a location for execution of the first and second functions;
wherein the execution plan specifies that the first function is executed on the computational node by fetching data required by the first function from the remotely located data node or writing data produced by the first function to the remotely located data node; and
wherein the execution plan specifies that the second function is executed at the remotely located data node to create a result, and the result is transmitted to the task driver at the computational node, wherein the first function is executed by fetching data required by the function from the remotely located data node or writing data produced by the function to the remotely located data node.
9. The method of claim 8 , wherein the computational node includes one or more processors.
10. The method of claim 8 , wherein the data node includes one or more processors.
11. The method of claim 8 , wherein executing the functions comprises executing at least some of the functions by sending one or more queries to the data cluster.
12. The method of claim 8 , wherein the data cluster includes a router.
13. The method of claim 8 , wherein a third function is executed at the computation node creating first results and a fourth function is executed at the data node creating second results.
14. The method of claim 13 , further comprising aggregating the first results and the second results at the task driver.