Learning optimizer for shared cloud

1. A system, comprising:

at least one processor; and

a memory having computer-executable instructions stored thereupon which, when executed by the at least one processor, cause the system to:

analyze workload data to determine features of operator subgraphs of queries;

using a machine learning algorithm, train a particular cardinality model for a particular operator subgraph based at least on the features and actual runtime statistics included in the workload data for at least two different jobs that include different parameters or different data inputs, the particular operator subgraph comprising a sequence of operators that occurs in the at least two different jobs; and

store the particular cardinality model.

2. The system of claim 1 , wherein the extracted features comprise, for an occurrence of an individual operator subgraph in an individual job in the workload data, at least one of a job name of the individual job, a total cardinality of all inputs to the individual operator subgraph, a name of all input datasets to the individual operator subgraph, or one or more parameters in the individual operator subgraph.

3. The system of claim 1 , wherein the features comprise, for an occurrence of an individual operator subgraph in an individual job in the workload data, at least one of a normalized job name of the individual job, a square of an input cardinality of the individual operator subgraph, a square root of the input cardinality of the individual operator subgraph, a log of the input cardinality of the individual operator subgraph, or an average output row length.

4. The system of claim 1 , wherein the particular cardinality model is based on at least one of a linear regression algorithm, a Poisson regression algorithm, or a multi-layer perceptron neural network.

5. The system of claim 1 , the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:

store at least one of a priority level or a confidence level with the particular cardinality model.

6. The system of claim 1 , the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:

during training of another cardinality model for another operator subgraph, determine that the another operator subgraph is more computationally expensive than a full query plan and stop exploring join orders which involve that individual operator subgraph.

7. The system of claim 1 , the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:

during training of cardinality models for two equivalent operator subgraphs including the particular operator subgraph and another operator subgraph, select the particular operator subgraph for training based at least on a determination that the particular operator subgraph maximizes a number of new subgraphs observed relative to the another operator subgraph.

8. The system of claim 1 , the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the system to:

predict a cardinality of the particular operator subgraph for a received query using the retrieved particular cardinality model; and

determine whether to utilize the particular operator subgraph for execution of the received query based on the predicted cardinality.

9. A method, comprising:

determining features for operator subgraphs that can be used to execute a query, each operator subgraph comprising a different sequence of two or more operators;

retrieving cardinality models for the operator subgraphs;

predicting cardinalities of the operator subgraphs for executing the query using the features and the cardinality models; and

selecting a particular operator subgraph to utilize for execution of the query based at least on the predicted cardinalities.

10. The method of claim 9 , further comprising:

executing the query using the particular operator subgraph.

11. The method of claim 9 , wherein the features reflect at least one of job names, input cardinality, input dataset names, or operator subgraph parameters.

12. The method of claim 9 , wherein the features reflect at least one of normalized job names, squares of input cardinalities, square roots of input cardinalities, logs of input cardinalities, or average row length outputs.

13. The method of claim 9 , wherein the cardinality models are based on at least one of a linear regression algorithm, a Poisson regression algorithm, or a multi-layer perceptron neural network.

14. The method of claim 9 , wherein at least one of a priority level or a confidence level is stored with the cardinality models and used when selecting the particular operator subgraph to utilize for execution of the query.

15. A computer storage media storing computer-readable instructions that, when executed, cause a computing device to:

analyze workload data to determine features of a plurality of operator subgraphs, each operator subgraph comprising a sequence of two or more operators;

using a machine learning algorithm, train cardinality models for the operator subgraphs based at least on the features and actual runtime statistics included in the workload data for jobs that include individual operator subgraphs of the plurality;

receive a query having particular operator subgraphs that can be used to execute the query;

retrieve respective cardinality models for the particular operator subgraphs;

predict cardinalities of the particular operator subgraphs using the respective cardinality models; and

select a specific operator subgraph of the particular operator subgraphs to utilize for execution of the query based at least on the predicted cardinalities.

16. The computer storage media of claim 15 , storing further computer-readable instructions that, when executed, cause the computing device to:

execute the query using the specific operator subgraph.

17. The computer storage media of claim 15 , wherein the features identify job names of individual jobs which involved execution of the individual operator subgraphs.

18. The computer storage media of claim 15 , wherein the features identify average row output length.

19. The computer storage media of claim 15 , wherein the cardinality models are based on at least one of a linear regression algorithm, a Poisson regression algorithm, or a multi-layer perceptron neural network.

20. The computer storage media of claim 15 , wherein at least one of a priority level or a confidence level is stored with the cardinality models and used when selecting the specific operator subgraph to utilize for execution of the query.