Incremental feature development and workload capture in database systems
Systems, methods, and devices for incremental feature development and workload capture in database systems are disclosed. The method includes determining a workload based on one or more client queries to be rerun for testing the feature release. The method further includes repeatedly executing a test run of the workload to identify whether there is a performance difference in the test run. The method further includes re-execute, in response to identifying the performance difference, the test run using resources with a different concurrency to identify whether the performance difference still exists. The method further includes flagging the performance difference as a false positive in response to identifying that the performance difference no longer exists when the test run is re-executed using the resources with different concurrency.
1. A system for testing a feature release, the system comprising:
a processor to:
determine a workload based on one or more client queries to be rerun to test the feature release;
repeatedly execute a test run of the workload to identify whether there is a performance difference in the test run;
re-execute, in response to identifying the performance difference, the test run using resources with a different concurrency to identify whether the performance difference still exists; and
flag the performance difference as a false positive in response to identifying that the performance difference no longer exists when the test run is re-executed using the resources with different concurrency.
2. The system of claim 1 , wherein the processor to determine the workload further comprises the processor to filter out unwanted client queries based on a workload selection configuration comprising one or more parameters input by a user.
3. The system of claim 2 , wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested a client query; or
a query schema.
4. The system of claim 2 , wherein to determine the workload, the processor to:
deduplicate client queries that satisfy the workload selection configuration to determine a grouping of client queries.
5. The system of claim 1 , wherein the processor to identify one or more errors in response to identifying the performance difference, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the test run;
an internal error in the test run; or
a user error in the test run; and
wherein to rerun the test run, the processor is further to identify whether any of the one or more errors still exists.
6. The system of claim 1 , wherein the processor is further to generate a report comprising data about the test run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.
7. The system of claim 1 , wherein the processor is further to obfuscate results of the test run such that a user does not have visibility into any database data associated with the one or more client queries of the workload.
8. The system of claim 1 , wherein the processor to execute the test run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
9. The system of claim 1 , wherein the processor further to determine a version of a client's database data that existed when a client query of the one or more client queries was executed by the client.
10. The system of claim 1 , wherein the resources are separate isolated resources.
11. A method of testing a feature on a database, the method comprising:
determining a workload based on one or more client queries to be rerun for testing the feature release;
repeatedly executing a test run of the workload to identify whether there is a performance difference in the test run;
re-execute, in response to identifying the performance difference, the test run using resources with a different concurrency to identify whether the performance difference still exists; and
flagging the performance difference as a false positive in response to identifying that the performance difference no longer exists when the test run is re-executed using the resources with different concurrency.
12. The method of claim 11 , wherein determining the workload comprises filtering out unwanted client queries based on a workload selection configuration comprising one or more parameters input by a user.
13. The method of claim 12 , wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested an historical client query;
a query schema.
14. The method of claim 12 , wherein determining the workload comprises:
deduplicating client queries that satisfy the workload selection configuration to determine a grouping of client queries.
15. The method of claim 11 , further comprising:
identifying one or more errors in response to identifying the performance difference, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the test run;
an internal error in the test run; or
a user error in the test run; and
wherein rerunning the test run comprises identify whether any of the one or more errors still exists.
16. The method of claim 11 , further comprising:
generating a report comprising data about the test run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.
17. The method of claim 11 , further comprising:
obfuscating results of the test run such that a user does not have visibility into any database data associated with the one or more client queries of the workload.
18. The method of claim 11 , further comprising:
executing the test run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
19. The method of claim 11 , further comprising:
determining a version of a client's database data that existed when a client query of the one or more client queries was executed by the client.
20. The method of claim 11 , wherein the resources are separate isolated resources.
21. A non-transitory computer readable storage media, programmable to execute instructions that, when executed, cause a processor to:
determine a workload based on one or more client queries to be rerun to test the feature release;
repeatedly execute a test run of the workload to identify whether there is a performance difference in the test run;
re-execute, in response to identifying the performance difference, the test run using resources with a different concurrency to identify whether the performance difference still exists; and
flag the performance difference as a false positive in response to identifying that the performance difference no longer exists when the test run is re-executed using the resources with different concurrency.
22. The non-transitory computer readable storage media of claim 21 , wherein the processor to determine the workload further comprises the processor to filter out unwanted client queries based on a workload selection configuration comprising one or more parameters input by a user.
23. The non-transitory computer readable storage media of claim 22 , wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested a client query; or
a query schema.
24. The non-transitory computer readable storage media of claim 22 , wherein the processor to:
deduplicate client queries that satisfy the workload selection configuration to determine a grouping of client queries.
25. The non-transitory computer readable storage media of claim 21 , wherein the processor to identify one or more errors in response to identifying the performance difference, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the test run;
an internal error in the test run; or
a user error in the test run; and
wherein to rerun the test run, the processor is further to identify whether any of the one or more errors still exists.
26. The non-transitory computer readable storage media of claim 21 , wherein the processor to generate a report comprising data about the test run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.
27. The non-transitory computer readable storage media of claim 21 , wherein the processor to obfuscate results of the test run such that a user does not have visibility into any database data associated with the one or more client queries of the workload.
28. The non-transitory computer readable storage media of claim 21 , wherein the processor to execute the test run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.
29. The non-transitory computer readable storage media of claim 21 , wherein the processor to determine a version of a client's database data that existed when a client query of the one or more client queries was executed by the client.
30. The non-transitory computer readable storage media of claim 21 , wherein the resources are separate isolated resources.