Production data in continuous integration flows
The present disclosure relates to software development and, more specifically, to continuous integration (CI) based software development systems, methods, and computer program products in which a system-under-test (SUT) is populated with production data that has been filtered or transformed, or both. In accordance with this approach, the filtered or transformed production test data can be automatically generated and seamlessly integrated into a continuous integration process. In some examples, the resulting filtered or transformed test data closely reflects real production data while, at the same time, consuming less data storage space and protecting an individual's sensitive information.
1. A method of developing an application comprising, by one or more computing devices comprising computer hardware:
detecting, by a continuous integration system, a commit of an application code in a repository, wherein the commit represents a change in state of the application code;
compiling, by the continuous integration system, a build of the application based at least in part on the detected commit;
accessing, by a test data generation system, production data produced by a production system and organized in a database comprising records stored in multiple tables, including parent records stored in a parent table and dependent records, which are dependent from respective ones of the parent records, in one or more dependent tables, each record comprising one or more attribute values;
filtering the production data, by the test data generation system, to produce a filtered set of the production data;
by the test data generation system, selecting a subset of the production data in the database comprising one or more of the parent records in the filtered set of the production data and a plurality of the dependent records selected by traversing a respective hierarchy of the records to choose the dependent records in one or more of the dependent tables that are dependent from each of the one or more of the parent records;
transforming, by the test data generation system, one or more of the attribute values of the records in the selected subset to produce one or more changed values; and
testing, by a test computer system, the build of the application, wherein the testing comprises populating parameters of the application code commit with one or more of the attribute values and changed values.
2. The method of claim 1 , wherein the accessed production data is copied from source production data, and the filtering is performed on the production data copied from the source production data.
3. The method of claim 1 , wherein the accessing comprises retrieving the production data from a backup copy of the production data.
4. The method of claim 1 , wherein filtering the production data comprises, by the test data generation system, choosing the records to be included in the filtered set of the production data.
5. The method of claim 4 , wherein choosing the records comprises applying a query on the production data to produce the filtered set of the production data.
6. The method of claim 1 , wherein the selecting comprises selecting a specified number of parent records in the filtered set of the production data.
7. The method of claim 1 , wherein the selecting comprises selecting one or more of parent records in the filtered set of the production data randomly.
8. The method of claim 1 , wherein the transforming comprises replacing one or more of the attribute values with different respective replacement values.
9. The method of claim 8 , wherein the transforming comprises anonymizing one or more of the attribute values comprising personally identifiable information.
10. An application development system, comprising:
a continuous integration system to detect a commit of an application code in a repository, and to compile a build of the application based at least in part on the detected commit;
a test data generation system to access production data produced by a production system and organized in a database comprising records stored in multiple tables, including parent records stored in a parent table and dependent records, which are dependent from respective ones of the parent records, in one or more dependent tables, each record comprising one or more attribute values, filter the production data to produce a filtered set of the production data, select a subset of the production data in the database comprising one or more of the parent records in the filtered set of the production data and a plurality of the dependent records selected by traversing a respective hierarchy of the records to choose the dependent records in one or more of the dependent tables that are dependent from each of the one or more of the parent records, and transform one or more of the attribute values of the records in the selected subset to produce one or more changed values; and
a test computer system to populate parameters of the application code commit with one or more of the attribute values and changed values, and test the build of the application.
11. The application development system of claim 10 , wherein the accessed production data is copied from source production data, and the test data generation system filters the production data copied from the source production data.
12. The application development system of claim 11 , wherein the accessed production data is retrieved from a backup copy of the production data.
13. The application development system of claim 10 , wherein the subset is selected by selecting one or more of the parent records in the filtered set of the production data randomly.
14. The application development system of claim 10 , wherein the one or more of the attribute values are transformed by replacing the one or more of the attribute values with different respective replacement values.
15. The application development system of claim 10 , wherein the one or more of the attribute values are transformed by anonymizing one or more of the attribute values comprising personally identifiable information.
16. A computer program product for execution by a computer system and comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising:
an executable code portion configured to detect a commit of an application code in a repository, wherein the commit represents a change in state of the application code;
an executable code portion configured to compile a build of the application based at least in part on the detected commit;
an executable code portion configured to access production data produced by a production system and organized in a database comprising records stored in multiple tables, including parent records stored in a parent table and dependent records, which are dependent from respective ones of the parent records, in one or more dependent tables, each record comprising one or more attribute values;
an executable code portion configured to filter the production data to produce a filtered set of the production data;
an executable code portion configured to select a subset of the production data in the database comprising of one or more of the parent records in the filtered set of the production data and a plurality of the dependent records selected by traversing a respective hierarchy of the records to choose the dependent records in one or more of the dependent tables that are dependent from each of the one or more of the parent records;
an executable code portion configured to transform one or more of the attribute values of the records in the selected subset to produce one or more changed values;
an executable code portion configured to populate parameters of the application code commit with one or more of the attribute values and changed values; and
an executable code portion configured to test the build of the application.
17. The computer program product of claim 16 , wherein the accessed production data is copied from source production data.
18. The computer program product of claim 16 , wherein the executable code portion configured to select a subset of one or more parent records in the filtered set of the production data is further configured to randomly select one or more of the parent records in the filtered set of the production data.
19. The computer program product of claim 16 , wherein the executable code portion configured to transform one or more of the attribute values to produce one or more changed values is further configured to replace the one or more of the attribute values with different respective replacement values.
20. The computer program product of claim 19 , wherein the executable code portion configured to transform one or more of the attribute values to produce one or more changed values is further configured to anonymize one or more of the attribute values comprising personally identifiable information.