Tracking evaluation of workload stability through performance indexing
Methods, systems, and computer programs are described for tracking evaluation of workload stability through performance indexing. A plurality of metric source data is received by at least one hardware processor. Based on this data, a workload is identified as a stable workload candidate. A performance index is then generated, reflecting the characteristics of the identified stable workload candidate. The performance index is continuously tracked over a period of time, enabling the detection and analysis of any modifications to the workload and the subsequent impact on system performance.
1 . A method comprising:
receiving, by at least one hardware processor, a plurality of metric source data;
identifying a workload, based at least in part on the plurality of metric source data, as a stable workload candidate, the stable workload candidate comprising a workload that satisfies at least one predefined stability threshold for one or more production metrics over a running time window;
generating a performance index based on the workload being identified as the stable workload candidate, the performance index comprising at least one of: a global performance index or a performance sub-index associated with the workload; and
tracking the performance index over a period of time.
2 . The method of claim 1 , wherein:
the performance sub-index is generated based on a subcategory of the workload; and
the global performance index comprises one or more performance sub-indexes.
3 . The method of claim 1 , wherein identifying the workload as the stable workload candidate further comprises:
identifying the workload based on production workloads; and
defining the stable workload candidate to isolate impacts associated with changes to the production workloads from impacts associated with changes to a data platform.
4 . The method of claim 1 , wherein tracking the performance index over the period of time further comprises:
detecting, in an automatic manner, a definition change in the workload;
regenerating the performance index based on the definition change in the workload;
assessing an impact of the definition change in the workload; and
identifying the impact to the workload as a performance regression or a performance improvement.
5 . The method of claim 1 , wherein tracking the performance index over the period of time further comprises:
identifying adjacent comparison windows of the performance index, wherein the adjacent comparison windows include a first comparison window and a second comparison window;
generating a first performance index based on the workload during the first comparison window;
generating a second performance index based on the workload during the second comparison window; and
identifying overlapping workloads between the first performance index and the second performance index to measure relative performance of the workload over the period of time.
6 . The method of claim 5 , wherein tracking the performance index comprises:
comparing a value of the first performance index to a value of the second performance index, wherein the value of the first performance index and the value of the second performance index are based on one of: a previous period of time, a previous parameter setting, a previous metric, or a previous definition.
7 . The method of claim 6 , further comprising:
generating a next performance index based on a result of the comparing of the value of the first performance index and the value of the second performance index.
8 . The method of claim 1 , wherein identifying the stable workload candidate further comprises:
identifying a stable warehouse;
identifying a stable recurrent query;
identifying stable data sources; and
categorizing the workload as a stable workload, based on at least one of the identified stable warehouse, the identified stable recurrent query, or the identified stable data sources.
9 . The method of claim 8 , wherein the stable workload is automatically detected based on production query patterns.
10 . The method of claim 1 , further comprising:
causing the performance index to be displayed in a user interface.
11 . A system comprising:
one or more hardware processors of a machine; and
at least one memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:
receiving a plurality of metric source data;
identifying a workload, based at least in part on the plurality of metric source data, as a stable workload candidate, the stable workload candidate comprising a workload that satisfies at least one predefined stability threshold for one or more production metrics over a running time window;
generating a performance index based on the workload being identified as the stable workload candidate, the performance index comprising at least one of: a global performance index or a performance sub-index associated with the workload; and
tracking the performance index over a period of time.
12 . The system of claim 11 , wherein:
the performance sub-index is generated based on a subcategory of the workload; and
the global performance index comprises one or more performance sub-indexes.
13 . The system of claim 11 , wherein identifying the workload as the stable workload candidate further comprises:
identifying the workload based on production workloads; and
defining the stable workload candidate to isolate impacts associated with changes to the production workloads from impacts associated with changes to a data platform.
14 . The system of claim 11 , wherein tracking the performance index over the period of the time further comprises:
detecting, in an automatic manner, a definition change in the workload;
regenerating the performance index based on the definition change in the workload;
assessing an impact of the definition change in the workload; and
identifying the impact to the workload as a performance regression or a performance improvement.
15 . The system of claim 11 , wherein tracking the performance index over the period of time further comprises:
identifying adjacent comparison windows of the performance index, wherein the adjacent comparison windows include a first comparison window and a second comparison window;
generating a first performance index based on the workload during the first comparison window;
generating a second performance index based on the workload during the second comparison window; and
identifying overlapping workloads between the first performance index and the second performance index to measure relative performance of the workload over the period of time.
16 . The system of claim 15 , wherein tracking the performance index comprises:
comparing a value of the first performance index to a value of the second performance index, wherein the value of the first performance index and the value of the second performance index are based on one of: a previous period of time, a previous parameter setting, a previous metric, or a previous definition.
17 . The system of claim 16 , the operations further comprising:
generating a next performance index based on a result of the comparing of the value of the first performance index and the value of the second performance index.
18 . The system of claim 11 , wherein identifying the stable workload candidate further comprises:
identifying a stable warehouse;
identifying a stable recurrent query;
identifying stable data sources; and
categorizing the workload as a stable workload, based on at least one of the identified stable warehouse, the identified stable recurrent query, or the identified stable data sources.
19 . The system of claim 11 , wherein the stable workload is automatically detected based on production query patterns.
20 . The system of claim 11 , the operations further comprising:
causing the performance index to be displayed in a user interface.
21 . A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a plurality of metric source data;
identifying a workload, based at least in part on the plurality of metric source data, as a stable workload candidate, the stable workload candidate comprising a workload that satisfies at least one predefined stability threshold for one or more production metrics over a running time window;
generating a performance index based on the workload being identified as the stable workload candidate, the performance index comprising at least one of: a global performance index or a performance sub-index associated with the workload; and
tracking the performance index over a period of time.
22 . The computer-storage medium of claim 21 , wherein:
the performance sub-index is generated based on a subcategory of the workload; and
the global performance index comprises one or more performance sub-indexes.
23 . The computer-storage medium of claim 21 , wherein identifying the workload as the stable workload candidate further comprises:
identifying the workload based on production workloads; and
defining the stable workload candidate to isolate impacts associated with changes to the production workloads from impacts associated with changes to a data platform.
24 . The computer-storage medium of claim 21 , wherein tracking the performance index over the period of the time further comprises:
detecting, in an automatic manner, a definition change in the workload;
regenerating the performance index based on the definition change in the workload;
assessing an impact of the definition change in the workload; and
identifying the impact to the workload as a performance regression or a performance improvement.
25 . The computer-storage medium of claim 21 , wherein tracking the performance index over the period of time further comprises:
identifying adjacent comparison windows of the performance index, wherein the adjacent comparison windows include a first comparison window and a second comparison window;
generating a first performance index based on the workload during the first comparison window;
generating a second performance index based on the workload during the second comparison window; and
identifying overlapping workloads between the first performance index and the second performance index to measure relative performance of the workload over the period of time.
26 . The computer-storage medium of claim 25 , wherein tracking the performance index comprises:
comparing a value of the first performance index to a value of the second performance index, wherein the value of the first performance index and the value of the second performance index are based on one of: a previous period of time, a previous parameter setting, a previous metric, or a previous definition.
27 . The computer-storage medium of claim 26 , the operations further comprising:
generating a next performance index based on a result of the comparing of the value of the first performance index and the value of the second performance index.
28 . The computer-storage medium of claim 21 , wherein identifying the stable workload candidate further comprises:
identifying a stable warehouse;
identifying a stable recurrent query;
identifying stable data sources; and
categorizing the workload as a stable workload, based on at least one of the identified stable warehouse, the identified stable recurrent query, or the identified stable data sources.
29 . The computer-storage medium of claim 28 , wherein the stable workload is automatically detected based on production query patterns.
30 . The computer-storage medium of claim 21 , the operations further comprising:
causing the performance index to be displayed in a user interface.