Systems and methods for improved structured data duplication in a database
Systems and methods for duplicating data in a data management system. As data is being updated in the data management system, a copy of the updated data is being written to a table for collecting the data changes. The data changes are flattened by a flattener to generate one or more flat files. A packaging controller may package the one or more flat files and generate a data change file. A listing API may list the data change files that are available. The data change file may be accessed via a data access API.
1 . A computer-implemented method for improved data duplication from a data management system, wherein the data management system comprises at least one data repository and a data duplication controller, the computer-implemented method comprising:
ingesting study file format data into the data management system, the study file format data comprising clinical study data and clinical study definition data;
generating a delta table for each object in the study file format data, wherein as data is updated in the data management system, a copy of the updated data is written to the delta table for collecting data changes;
determining whether a full operations or an incremental operation is to be performed;
extracting each data change by a data extractor in the data duplication controller;
running a data flattener in the data duplication controller to generate flattened data for each data change, and creating a first flat file for the flattened data for a first extracted data change and a second flat file for the flattened data for a second extracted data change, wherein the first flat file includes data of the first object but not the second object, and the second flat file includes data of the second object but not the first object, and wherein the first extract is configured to extract a first object in the data management system and the second extract is configured to extract a second object in the data management system;
generating a data change file to package the first flat file and the second flat file for the flattened data for the extracted data changes by a packaging controller in the data duplication controller; and
presenting the data change file with a listing API in the data duplication controller to enable access to the data change file.
2 . The method of claim 1 , wherein the first predetermined time interval is 15 minutes.
3 . The method of claim 1 , further comprising: running the data flattener at a second predetermined time interval to extract the data changes within the second predetermined time interval and generating a first incremental change flat file.
4 . The method of claim 3 , wherein the second predetermined time interval is 15 minutes.
5 . The method of claim 1 , further comprising: maintaining a full copy of the flattened data.
6 . The method of claim 1 , wherein the flat file is a CSV file, a JSON file, an XML file, or a Parquet file.
7 . The method of claim 1 , wherein the generating flattened data for the extracted data changes further comprises concatenating the data by the data flattener.
8 . The method of claim 1 , wherein the generating flattened data for the extracted data changes further comprises de-duplicating the data by the data flattener.
9 . The method of claim 1 , wherein the data is duplicated from the data management system to a data storage system.
10 . The method of claim 9 , wherein the data storage system is a data warehouse.
11 . The method of claim 1 , wherein the first data change file is a zip file.
12 . The method of claim 1 , wherein the first data change file is accessed by a data access API.
13 . The method of claim 12 , wherein the data access API is an REST API.
14 . The method of claim 1 , further comprising: cleaning the delta table at a third predetermined time interval.
15 . The method of claim 14 , wherein the third predetermined time interval is 3 days.
16 . The method of claim 1 , further comprising: creating a flat file for each delta table.
17 . The method of claim 1 , further comprising: storing data updates in a third flat file.
18 . The method of claim 17 , further comprising: storing data deletes in a fourth flat file.
19 . The method of claim 1 , wherein the first flat file has a first start time and a first stop time, and the first flat file comprises data changes on or after the first start time and on or before the first stop time.
20 . The method of claim 1 , wherein the first repository is a data repository.
21 . The method of claim 1 , wherein the first repository is a data and content repository.
22 . A data management system, comprising a processor, a memory, at least one repository and a data duplication controller, wherein the data duplication controller comprises:
a data extractor for extracting a data change in the data management system, wherein the data change modifies an object in the at least one data repository and records the data change to a delta table for collecting the data changes in the data management system, wherein the delta table comprises associated fields corresponding to the object, and a delta indicator specifying whether the data change is an addition, update, or deletion;
a data flattener for generating flattened data for the extracted data changes, and creating a first flat file and a second flat file for the flattened data for the extracted data changes, wherein the first flat file includes data of the first object but not the second object, and the second flat file includes data of the second object but not the first object, wherein the first flat file is for a first extract and the second flat file is for a second extract, and wherein the first extract is configured to extract a first object in the data management system and the second extract is configured to extract a second object in the data management system;
a packaging controller for receiving the first and second flat files from the data flattener at a first predetermined time interval, and creating a first data change file to package the first flat file and the second flat file for the flattened data for the extracted data changes; and
a listing API for presenting the first data change file to enable access to the data change file.