Optimized identification of performance regression
A baseline run of a workload and a target run of the workload are compared to identify whether there is a performance regression in the target run, wherein the target run implements a feature that is not implemented in the baseline run. In response to identifying the performance regression, the target run is executed to identify whether the performance regression still exists. The performance regression is flagged as a false positive in response to identifying that the performance regression no longer exists when the target run is executed.
1. A system for testing a feature on a database, the system comprising:
a processor to:
compare a baseline run of a workload and a target run of the workload to identify whether there is a performance regression in the target run, wherein the target run implements a feature that is not implemented in the baseline run;
in response to identifying the performance regression, identify a query of the baseline run and the target run that executes at a slower execution rate when using the target run as compared to when using the baseline run;
re-execute the query using the target run on separate isolated resources to identify whether the performance regression still exists; and
flag the performance regression as a false positive in response to identifying that the performance regression no longer exists when the query is re-executed using the target run on the separate isolated resources.
2. The system of claim 1 , wherein the comparison of the baseline run and the target run is based on a previous execution of the baseline run and the target run by the processor.
3. The system of claim 1 , wherein the feature comprises procedural logic.
4. The system of claim 1 , wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
5. The system of claim 4 , the processor to:
deduplicate historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.
6. The system of claim 1 , wherein the feature modifies a function of the database.
7. The system of claim 1 , wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.
8. The system of claim 1 , wherein the separate isolated resources use lower concurrency to re-execute the query using target run.
9. The system of claim 1 , wherein the processor to identify one or more errors in response to comparing the baseline run and the target run, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein to execute the target run, the processor is further to identify whether any of the one or more errors still exists; and
wherein to flag the performance regression as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.
10. The system of claim 9 , wherein the processor is further to generate a report comprising data about the target run, wherein the report does not comprise an indication of any performance regression or error that was flagged as a false positive.
11. The system of claim 1 , wherein the processor is further to obfuscate results of the baseline run and the target run such that a user does not have visibility into any database data associated with the workload.
12. The system of claim 1 , wherein the processor to execute the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
13. The system of claim 1 , wherein the processor further to determine a version of a client's database data that existed when a historical client query was run by the client, and wherein the processor to execute on the version of the client's database data that existed when the historical client query of the one or more historical client queries was run by the client.
14. A method of testing a feature on a database, the method comprising:
comparing a baseline run of a workload and a target run of the workload to identify whether there is a performance regression in the target run, wherein the target run implements a feature that is not implemented in the baseline run;
in response to identifying the performance regression, identify a query of the baseline run and the target run that executes at a slower execution rate when using the target run as compared to when using the baseline run;
re-executing the query using the target run on separate isolated resources to identify whether the performance regression still exists; and
flagging, by a processor, the performance regression as a false positive in response to identifying that the performance regression no longer exists when the query is re-executed using the target run on the separate isolated resources.
15. The method of claim 14 , wherein the comparison of the baseline run and the target run is based on a previous execution of the baseline run and the target run by the processor.
16. The method of claim 14 , wherein the feature comprises procedural logic.
17. The method of claim 14 , wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
18. The method of claim 17 , further comprising:
deduplicating historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
pruning the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
selecting a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.
19. The method of claim 14 , wherein the feature modifies a function of the database.
20. The method of claim 14 , wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.
21. The method of claim 14 , wherein the separate isolated resources use lower concurrency to re-execute the query using target run.
22. The method of claim 14 , further comprising identifying one or more errors in response to comparing the baseline run and the target run, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein to execute the target run, the processor is further to identify whether any of the one or more errors still exists; and
wherein to flag the performance regression as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.
23. The method of claim 22 , further comprising:
generating a report comprising data about the target run, wherein the report does not comprise an indication of any performance regression or error that was flagged as a false positive.
24. The method of claim 14 , further comprising:
executing the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
25. A non-transitory computer readable storage media, programmable to execute instructions that, when executed, cause a processor to:
compare a baseline run of a workload and a target run of the workload to identify whether there is a performance regression in the target run, wherein the target run implements a feature that is not implemented in the baseline run;
in response to identifying the performance regression, identify a query of the baseline run and the target run that executes at a slower execution rate when using the target run as compared to when using the baseline run;
re-execute the query using the target run on separate isolated resources to identify whether the performance regression still exists; and
flag, by the processor, the performance regression as a false positive in response to identifying that the performance regression no longer exists when the query is re-executed using the target run on the separate isolated resources.
26. The non-transitory computer readable storage media of claim 25 , wherein the comparison of the baseline run and the target run is based on a previous execution of the baseline run and the target run by the processor.
27. The non-transitory computer readable storage media of claim 25 , wherein the feature comprises procedural logic.
28. The non-transitory computer readable storage media of claim 25 , wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
29. The non-transitory computer readable storage media of claim 28 , the processor to:
deduplicate historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.
30. The non-transitory computer readable storage media of claim 25 , the processor to:
execute the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.