Apparatus, system, and method for cross-platform data processing
The present disclosure generally relates to a cost-based optimizer for efficiently processing data through the use of multiple different data processing platforms. The cost-based optimizer may receive an input plan for processing data that includes a number of base operators. The cost-based optimizer may then determine execution operators for each base operator, where each execution operator corresponds to a different data processing platform. From the execution operators, the cost-based optimizer may determine possible subplans for executing the input plan on one or more data processing platforms. The cost-based optimizer may determine the cost of executing each possible subplan and choose the subplan with the lowest cost.
1. A cross-platform data processing apparatus comprising:
a processor; and
a memory storing instructions which, when executed by the processor, cause the processor to:
receive an input plan to process a set of data, the input plan comprising a plurality of base operators;
determine at least one first execution operator for a first base operator from the plurality of base operators, the at least one first execution operator corresponding to a first data processing platform;
determine at least one second execution operator for the first base operator, the at least one second execution operator corresponding to a second data processing platform;
generate an inflated operator including the first base operator, each of the determined at least one first execution operator, and each of the determined at least one second execution operator;
generate a first subplan based on the inflated operator, the first subplan including the first execution operator but not the second execution operator;
generate a second subplan based on the inflated operator, the second subplan including the second execution operator but not the first execution operator;
calculate a cost value for each of the first subplan and the second subplan; and
select the first subplan or the second subplan that has the lowest cost value.
2. The apparatus of claim 1 , wherein the memory stores instructions which, when executed by the processor, cause the processor to determine a plurality of execution operators for the first base operator, each of the plurality of execution operators corresponding to a different data processing platform.
3. The apparatus of claim 1 , wherein the memory stores instructions which, when executed by the processor, cause the processor to determine a plurality of execution operators for each base operator in a plurality of base operators, each execution operator for each base operator corresponding to a different data processing platform.
4. The apparatus of claim 3 , wherein the memory stores instructions which, when executed by the processor, cause the processor to:
generate a plurality of subplans based on the plurality of execution operators;
calculate a cost value for each of the plurality of subplans; and
select the subplan with the lowest cost value.
5. The apparatus of claim 1 , wherein the memory stores instructions which, when executed by the processor, cause the processor to generate an inflated operator for each of the plurality of base operators.
6. The apparatus of claim 5 , wherein the memory stores instructions which, when executed by the processor, cause the processor to generate an inflated input plan, wherein the inflated input plan comprises a plurality of inflated operators.
7. The apparatus of claim 1 , wherein determining the first execution operator and the second execution operator comprises using a mapping dictionary to map the base operator to the first execution operator and the second execution operator.
8. The apparatus of claim 1 , wherein determining the first execution operator and the second execution operator comprises graph mapping the base operator to the first execution operator and the second execution operator.
9. The apparatus of claim 1 , wherein calculating the cost value of a plan comprises calculating an operating cost and a data movement cost of the plan.
10. The apparatus of claim 9 , wherein calculating the data movement cost comprises using a minimum cost conversion tree.
11. A method for efficient cross-platform data processing, the method comprising:
receiving an input plan to process a set of data, the input plan comprising a plurality of base operators;
determining at least one first execution operator for a first base operator from the plurality of base operators, the at least one first execution operator corresponding to a first data processing platform;
determining at least one second execution operator for the first base operator, the at least one second execution operator corresponding to a second data processing platform;
generating an inflated operator including the first base operator, each of the determined at least one first execution operator, and each of the determined at least one second execution operator;
generating a first subplan based on the inflated operator, the first subplan including the first execution operator but not the second execution operator;
generating a second subplan based on the inflated operator, the second subplan including the second execution operator but not the first execution operator;
calculating a cost value for each of the first subplan and the second subplan; and
selecting the first subplan or the second subplan with the lowest cost value.
12. The method of claim 11 , wherein the method comprises determining a plurality of execution operators for the first base operator, each of the plurality of execution operators corresponding to a different data processing platform.
13. The method of claim 11 , wherein the method comprises determining a plurality of execution operators for each base operator in a plurality of base operators, each execution operator for each base operator corresponding to a different data processing platform.
14. The method of claim 13 , wherein the method comprises:
generating a plurality of subplans based on the plurality of execution operators;
calculating a cost value for each of the plurality of subplans; and
selecting the subplan with the lowest cost value.
15. The method of claim 11 , wherein the method comprises generating an inflated operator for each of the plurality of base operators.
16. The method of claim 15 , wherein the method comprises generating an inflated input plan, wherein the inflated input plan comprises a plurality of inflated operators.
17. The method of claim 11 , wherein determining the first execution operator and the second execution operator comprises graph mapping the base operator to the first execution operator and the second execution operator.
18. The method of claim 11 , wherein calculating the cost value of a plan comprises calculating an operating cost and a data movement cost of the plan.
19. The method of claim 18 , wherein calculating the data movement cost comprises using a minimum cost conversion tree.
20. A non-transitory, computer-readable medium storing instructions which, when performed by a processor, cause the processor to:
receive an input plan to process a set of data, the input plan comprising a plurality of base operators;
determine at least one first execution operator for a first base operator from the plurality of base operators, the at least one first execution operator corresponding to a first data processing platform;
determine at least one second execution operator for the first base operator, the at least one second execution operator corresponding to a second data processing platform;
generate an inflated operator including the first base operator, each of the determined at least one first execution operator, and each of the determined at least one second execution operator;
generate a first subplan based on the inflated operator, the first subplan including the first execution operator but not the second execution operator;
generate a second subplan based on the inflated operator, the second subplan including the second execution operator but not the first execution operator;
calculate a cost value for each of the first subplan and the second subplan; and
select the first subplan or the second subplan that has the lowest cost value.
21. A cross-platform data processing apparatus comprising:
a processor; and
a memory storing instructions which, when executed by the processor, cause the processor to:
receive an input plan to process a set of data, the input plan comprising a plurality of base operators;
determine a first execution operator for a first base operator from the plurality of base operators, the first execution operator corresponding to a first data processing platform;
determine a second execution operator for the first base operator, the second execution operator corresponding to a second data processing platform;
generate an inflated operator for each of the plurality of base operators;
generate a first subplan based on the first execution operator;
generate a second subplan based on the second execution operator;
calculate a cost value for each of the first subplan and the second subplan; and
select the first subplan or the second subplan that has the lowest cost value.
22. A cross-platform data processing apparatus comprising:
a processor; and
a memory storing instructions which, when executed by the processor, cause the processor to:
receive an input plan to process a set of data, the input plan comprising a plurality of base operators;
determine a first execution operator for a first base operator from the plurality of base operators, the first execution operator corresponding to a first data processing platform;
determine a second execution operator for the first base operator, the second execution operator corresponding to a second data processing platform;
generate a first subplan based on the first execution operator;
generate a second subplan based on the second execution operator;
calculate a cost value for each of the first subplan and the second subplan; and
select the first subplan or the second subplan that has the lowest cost value,
wherein calculating the cost value for each of the first subplan and the second subplan includes calculating an operating cost and a data movement cost for each of the first subplan and the second subplan, and
wherein calculating the data movement cost comprises using a minimum cost conversion tree.