Learned resource consumption model for optimizing big data queries
Methods, systems, apparatuses, and computer program products are provided for evaluating a resource consumption of a query. A logical operator representation of a query generated to be executed (e.g., obtained from a query generating entity) may be determined. The logical operator representation may be transformed to a plurality of different physical operator representations for executing the query. A plurality of resource consumption models may be applied to each of the physical operator representations to determine a resource consumption estimate for the physical operator representation. The resource consumption models may be trained in different manners based at least on a history of query executions, such that each model may have different granularity, coverage and/or accuracy characteristics in estimating a resource consumption of a query. Based on the determined resource consumption estimates for the physical operator representations, a particular one of the physical operator representations may be selected to execute the query.
1 . A system for evaluating a resource consumption of a query, the system comprising:
one or more processors; and
one or more memory devices that store program code configured to be executed by the one or more processors, the program code configured to perform operations that comprise:
determining a logical operator representation of a query to be executed;
transforming the logical operator representation to two or more physical operator representations for executing the query, each physical operator representation corresponding to a query plan;
applying a plurality of machine learning-based resource consumption models trained based at least on a history of query executions to each of the physical operator representations to determine a resource consumption estimate of each of the physical operator representations, each of the plurality of machine learning-based resource consumption models being trained based on different physical operator tree characteristics; and
selecting a particular query plan to execute the query, the selected query plan corresponding to one of the physical operator representations based on the determined resource consumption estimates.
2 . The system of claim 1 , wherein the program code is further configured to perform operations that comprise selecting a partition count for each of the physical operator representations prior to the determining the resource consumption estimate of each of the physical operator representations, the partition count being selected based at least on a resource consumption estimate of a portion of each physical operator representation.
3 . The system of claim 1 , wherein each of the plurality of machine learning-based resource consumption models comprises a machine-learning model that is trained based at least on a feature set associated with the history of query executions, the feature set corresponding to each machine-learning model being weighted differently from one another.
4 . The system of claim 1 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common schema as a physical operator tree of at least one of the physical operator representations.
5 . The system of claim 1 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common root operator as a physical operator tree of at least one of the physical operator representations.
6 . The system of claim 1 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common root operator, a common set of leaf node inputs, and a same number of total operators as a physical operator tree of at least one of the physical operator representations.
7 . The system of claim 1 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common root operator, a common set of leaf node inputs, and a different number of total operators as a physical operator tree of at least one of the physical operator representations.
8 . The system of claim 1 , wherein the applying the plurality of machine learning-based resource consumption models comprises applying a combined model generated from the plurality of resource consumption models.
9 . The system of claim 8 , wherein the combined model is generated from a weighting of each of the plurality of machine learning-based resource consumption models.
10 . A computer-implemented method for evaluating a resource consumption of a query, the method comprising:
determining a logical operator representation of a query to be executed;
transforming the logical operator representation to two or more physical operator representations for executing the query, each physical operator representation corresponding to a query plan;
applying a plurality of machine learning-based resource consumption models trained based at least on a history of query executions to each of the physical operator representations to determine a resource consumption estimate of each of the physical operator representations, each of the plurality of machine learning-based resource consumption models being trained based on different physical operator tree characteristics; and
selecting a particular query plan to execute the query, the selected query plan corresponding to one of the physical operator representations based on the determined resource consumption estimates.
11 . The computer-implemented method of claim 10 , further comprising:
selecting a partition count for each of the physical operator representations prior to determining the resource consumption estimate of each of the physical operator representations, the partition count being selected based at least on a resource consumption estimate of a portion of each physical operator representation.
12 . The computer-implemented method of claim 10 , wherein each of the plurality of machine learning-based resource consumption models comprises a machine-learning model that is trained based at least on a feature set associated with the history of query executions, the feature set corresponding to each machine-learning model being weighted differently from one another.
13 . The computer-implemented method of claim 10 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common schema as a physical operator tree of at least one of the physical operator representations.
14 . The computer-implemented method of claim 10 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning- based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common root operator as a physical operator tree of at least one of the physical operator representations.
15 . The computer-implemented method of claim 10 , wherein the applying the plurality of machine learning-based resource consumption models comprises applying a combined model generated from the plurality of machine learning-based resource consumption models.
16 . A computer-readable memory having computer program code recorded thereon that when executed by at least one processor causes the at least one processor to perform a method comprising:
determining a logical operator representation of a query to be executed;
transforming the logical operator representation to two or more physical operator representations for executing the query, each physical operator representation corresponding to a query plan;
applying a plurality of machine learning-based resource consumption models trained based at least on a history of query executions to each of the physical operator representations to determine a resource consumption estimate of each of the physical operator representations, each of the plurality of machine learning-based resource consumption models being trained based on different physical operator tree characteristics; and
selecting a particular query plan to execute the query, the selected query plan corresponding to one of the physical operator representations based on the determined resource consumption estimates.
17 . The computer-readable memory of claim 16 , further comprising:
selecting a partition count for each of the physical operator representations prior to determining the resource consumption estimate of each of the physical operator representations, the partition count being selected based at least on a resource consumption estimate of a portion of each physical operator representation.
18 . The computer-readable memory of claim 16 , wherein each of the plurality of machine learning-based resource consumption models comprises a machine-learning model that is trained based at least on a feature set associated with the history of query executions, the feature set corresponding to each machine-learning model being weighted differently from one another.
19 . The computer-readable memory of claim 16 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common schema as a physical operator tree of at least one of the physical operator representations.
20 . The computer-readable memory of claim 16 , wherein at least one machine learning-based resource consumption model of the plurality of machine learning-based resource consumption models is trained based at least on prior query executions that comprise a physical operator tree with a common root operator as a physical operator tree of at least one of the physical operator representations.