Cloud based query workload optimization
The cloud-based query workload optimization system disclosed herein the cloud-based query workloads optimization system receives query logs from various query engines to a cloud data service, extracts various query entities from the query logs, parses query entities to generate a set of common workload features, generates intermediate representations of the query workloads, wherein the intermediate representations are agnostic to the language of the plurality of the queries, identifies a plurality of workload patterns based on the intermediate representations of the query workloads, categorizes the workloads in one or more workload type categories based on the workload patterns and the workload features, and selects an optimization scheme based on the category of workload pattern.
1. A physical article of manufacture including one or more non-transitory computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising:
receiving query logs from various query engines to a cloud data service;
extracting various query traces from the query logs, wherein the query traces including at least one of query metadata, query plans, and query runtime statistics;
parsing query traces using a plurality of parsers with each of the plurality of parsers configured to parse a different types of queries to generate a set of common workload features;
generating intermediate representations of the query workloads using the set of common workload features, wherein the intermediate representations are agnostic to the language of the plurality of the queries and are common across workloads and query engines;
identifying a plurality of workload patterns based on the intermediate representations of the query workloads;
categorizing the workloads in one or more workload type categories based on the workload patterns and the workload features; and
selecting an optimization scheme based on the category of workload pattern.
2. The physical article of manufacture of claim 1 , wherein the computer process further comprising optimizing the workload using the selected optimization scheme.
3. The physical article of manufacture of claim 1 , wherein categorizing the workloads further comprising categorizing the workloads based on recurring workload pattern, similarity workload pattern, and dependency workload pattern.
4. The physical article of manufacture of claim 1 , wherein selecting an optimization scheme based on the category of workload pattern further comprising selecting a dependency-driven optimization scheme if the workload pattern is a dependency pattern.
5. The physical article of manufacture of claim 1 , wherein selecting an optimization scheme based on the category of workload pattern further comprising selecting a learned optimization scheme if the workload pattern is a recurring pattern.
6. The physical article of manufacture of claim 1 , wherein selecting an optimization scheme based on the category of workload pattern further comprising selecting a multi-query optimization scheme if the workload pattern is a similarity pattern.
7. The physical article of manufacture of claim 1 , wherein the computer process further comprising generating query annotations that can be used by the query engines during query compilation.
8. The physical article of manufacture of claim 7 , wherein the query annotations include a signature, an action, and a set of parameters to be consumed by a query compiler.
9. The physical article of manufacture of claim 7 , wherein the query annotations are output as a file on a cloud storage location to be accessed by the query engine.
10. The physical article of manufacture of claim 1 , wherein the query traces further comprising query metadata, query plans, and query runtime statistics.
11. A method of providing cloud-based query workload optimization, the method comprising:
receiving query logs from various query engines to a cloud data service;
extracting various query traces from the query logs, wherein the query traces including at least one of query metadata, query plans, and query runtime statistics;
parsing query traces using a plurality of parsers with each of the plurality of parsers configured to parse a different types of queries to generate a set of common workload features;
generating intermediate representations of the query workloads using the set of common workload features, wherein the intermediate representations are agnostic to the language of the plurality of the queries and are common across workloads and query engines;
identifying a plurality of workload patterns based on the intermediate representations of the query workloads;
categorizing the workloads in one or more workload type categories based on the workload patterns and the workload features; and
selecting an optimization scheme based on the category of workload pattern.
12. The method of claim 11 , further comprising optimizing the workload using the selected optimization scheme.
13. The method of claim 11 , wherein categorizing the workloads further comprising categorizing the workloads based on recurring workload pattern, similarity workload pattern, and dependency workload pattern.
14. The method of claim 11 , further comprising:
selecting a dependency-driven optimization scheme if the workload pattern is a dependency pattern;
selecting a learned optimization scheme if the workload pattern is a recurring pattern; and
selecting a multi-query optimization scheme if the workload pattern is a similarity pattern.
15. The method of claim 11 , further comprising generating query annotations that can be used by the query engines during query compilation.
16. The method of claim 15 , wherein the query annotations include a signature, an action, and a set of parameters to be consumed by a query compiler.
17. The method of claim 15 , wherein the query annotations are output as a file on a cloud storage location to be accessed by the query engine.
18. In a computing environment, a system comprising:
memory;
one or more processor units;
a cloud-based query workload optimization system stored in the memory and executable by the one or more processor units, the cloud-based query workload optimization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising:
receiving query logs from various query engines to a cloud data service;
extracting various query traces from the query logs, wherein the query traces including at least one of query metadata, query plans, and query runtime statistics;
parsing query traces using a plurality of parsers with each of the plurality of parsers configured to parse a different types of queries to generate a set of common workload features;
generating intermediate representations of the query workloads using the set of common workload features, wherein the intermediate representations are agnostic to the language of the plurality of the queries and are common across workloads and query engines;
identifying a plurality of workload patterns based on the intermediate representations of the query workloads;
categorizing the workloads in one or more workload type categories based on the workload patterns and the workload features;
selecting an optimization scheme based on the category of workload pattern; and
optimizing the workload using the selected optimization scheme.
19. The system of claim 18 , wherein categorizing the workloads further comprising categorizing the workloads based on recurring workload pattern, similarity workload pattern, and dependency workload pattern.
20. The system of claim 18 , wherein the computer process further comprising:
selecting a dependency-driven optimization scheme if the workload pattern is a dependency pattern;
selecting a learned optimization scheme if the workload pattern is a recurring pattern; and
selecting a multi-query optimization scheme if the workload pattern is a similarity pattern.