Optimized identification of performance regression
A baseline run of a workload and a target run of the workload are compared to identify whether there is a performance regression in the target run, wherein the target run implements a feature that is not implemented in the baseline run. In response to identifying the performance regression, the target run is executed to identify whether the performance regression still exists. The performance regression is flagged as a false positive in response to identifying that the performance regression no longer exists when the target run is executed.
1. A system for testing a feature on a database, the system comprising:
a processor to:
repeatedly execute a test run of a workload using resources of a cloud environment at one or more execution rates to identify whether there is a performance difference in the test run;
in response to identifying the performance difference, increase the resources of the cloud environment;
re-execute the test run using the increased resources of the cloud environment and at a slower speed than the one or more execution rates to identify whether the performance difference still exists; and
determine that the performance difference is attributed to variations in the cloud environment in response to identifying that the performance difference no longer exists.
2. The system of claim 1 , wherein the re-execute the test run using the scaled resources of the cloud environment to identify whether the performance difference still exists comprises:
selecting a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.
3. The system of claim 1 , wherein the feature comprises procedural logic.
4. The system of claim 1 , wherein the processor to repeatedly execute the test run of the workload using the resources of the cloud environment to identify whether there is a performance difference in the test run comprises:
compare a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
5. The system of claim 1 , wherein the feature modifies a function of the database.
6. The system of claim 1 , wherein the separate isolated resources use lower concurrency to re-execute the query using target run is executed using separate isolated resources with lower concurrency.
7. The system of claim 1 , wherein the processor to execute the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
8. The system of claim 1 , wherein the processor further to determine a version of a client's database data that existed when a historical client query was run by the client, and wherein the processor to execute on the version of the client's database data that existed when the historical client query of the one or more historical client queries was run by the client.
9. The system of claim 4 , wherein the processor to:
deduplicate historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.
10. The system of claim 4 , wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.
11. The system of claim 4 , wherein the processor to identify one or more errors in response to comparing the baseline run and the target run, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein to execute the target run, the processor is further to identify whether any of the one or more errors still exists; and
wherein to flag the performance difference as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.
12. The system of claim 4 , wherein the processor is further to obfuscate results of the baseline run and the target run such that a user does not have visibility into any database data associated with the workload.
13. The system of claim 11 , wherein the processor is further to generate a report comprising data about the target run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.
14. A method of testing a feature on a database, the method comprising:
repeatedly executing a test run of a workload using resources of a cloud environment at one or more execution rates to identify whether there is a performance difference in the test run;
in response to identifying the performance difference, increasing the resources of the cloud environment;
re-executing the test run using the increased resources of the cloud environment and at a slower speed than the one or more execution rates to identify whether the performance difference still exits; and
determine that the performance difference is attributed to variations in the cloud environment in response to identifying that the performance difference no longer exists.
15. The method of claim 14 , wherein the re-executing the test run using the scaled resources of the cloud environment to identify whether the performance difference still exist comprises:
selecting a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.
16. The method of claim 14 , wherein the feature comprises procedural logic.
17. The method of claim 14 , wherein repeatedly executing the test run of the workload using resources of the cloud environment to identify whether there is the performance difference in the test run comprises:
comparing a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
18. The method of claim 14 , wherein the feature modifies a function of the database.
19. The method of claim 14 , wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.
20. The method of claim 14 , wherein the separate isolated resources use lower concurrency to re-execute the query using target run is executed using separate isolated resources with lower concurrency.
21. The method of claim 14 , further comprising identifying one or more errors in response to comparing the baseline run and the target run, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein to execute the target run, the processor is further to identify whether any of the one or more errors still exists; and
wherein to flag the performance difference as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.
22. The method of claim 14 , further comprising:
executing the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
23. The method of claim 17 , further comprising:
deduplicating historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
pruning the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
selecting a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.
24. The method of claim 21 , further comprising:
generating a report comprising data about the target run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.
25. A non-transitory computer readable storage media, programmable to execute instructions that, when executed, cause a processor to:
repeatedly execute a test run of a workload using resources of a cloud environment at one or more execution rates to identify whether there is a performance difference in the test run;
in response to identifying the performance difference, increase the resources of the cloud environment;
re-execute the test run using the increased resources of the cloud environment and at a slower speed than the one or more execution rates to identify whether the performance difference still exists; and
determine that the performance difference is attributed to variations in the cloud environment in response to identifying that the performance difference no longer exists.
26. The non-transitory computer readable storage media of claim 25 , wherein the processor to re-execute the test run using the scaled resources of the cloud environment to identify whether the performance difference still exists comprises:
select a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.
27. The non-transitory computer readable storage media of claim 25 , wherein the feature comprises procedural logic.
28. The non-transitory computer readable storage media of claim 25 , wherein the processor to repeatedly execute the test run of the workload using the resources of the cloud environment to identify whether there is a performance difference in the test run comprises:
compare a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.
29. The non-transitory computer readable storage media of claim 25 , the processor to:
execute the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
30. The non-transitory computer readable storage media of claim 28 , the processor to:
deduplicate historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.