Data abstraction layer for interfacing with reporting systems
View Patent ↗A data transformation system receives data from one or more external source systems and stores and transforms the data for providing to reporting systems. The data transformation system maintains multiple versions of data received from an external source system. The data transformation system can combine data from different versions of data and provide to the reporting system. As a result, external source systems that do not maintain data in a format appropriate for reporting systems and/or do not maintain sufficient historical data to generate different types of reports are able to generate these reports. The data transformation system can also enhance older versions of data stored in the system or exclude portions of data from reports. The data transformation system can purge older versions of data so that older data that is less frequently requested is maintained at a lower frequency than recent data.
1. A computer implemented method performed by a computing device comprising at least one processor, the method comprising:
receiving data from a plurality of external source systems, the receiving comprising, from each external source system, receiving a plurality of versions of data, each version of data associated with a time interval associated with the external source system, the versions of data represented in a first format;
storing the data received from the plurality of external source systems along with metadata describing each version of data;
receiving a request to generate report associated with an external source system, the report based on a set of versions of data obtained from the external source system;
transforming the data of the set of versions of data to a second format associated with the report; and
generating a report based on the transformed data.
2. The method of claim 1 , wherein the first format of the versions of data represents data as one of: row based changes, column based changes, a full snapshot, or information identifying deleted rows.
3. The method of claim 1 , wherein the second format associated with the requested report represents data as one of: a full snapshot, aggregation of data, change logs, or change logs for a particular column.
4. The method of claim 1 , wherein the data received from the plurality of external source systems is stored in a columnar format, the method further comprising:
converting data received from the external source system from the first format to the columnar format.
5. The method of claim 1 , wherein the first format of the external source system represents the data as change logs, the method further comprising:
applying the change logs to a previous version of data of the external source system to obtain a new version of data.
6. The method of claim 1 , wherein the second format represents data as changed logs, the method further comprising:
determining change logs by comparing at least two versions of data of the external source system.
7. The method of claim 1 , wherein the second format comprises an aggregation of information across two or more versions of data, the method further comprising:
retrieving data from the two or more versions of data of the external source system; and
determining requested aggregation of data over the two or more versions of data.
8. The method of claim 1 , wherein storing the data received from the plurality of external source systems comprises:
purging older versions of data so as to store more versions of data for recent time intervals compared to older time intervals.
9. The method of claim 1 , further comprising:
receiving a request to modify data previously loaded from the external source system;
identifying a version of data stored comprising the data to be modified; and
modifying the identified data in conformance with the received request.
10. The method of claim 1 , further comprising:
receiving a request to exclude data previously loaded from the external source system;
identifying a version of data stored comprising the data to be excluded; and
excluding the identified data from the data used for generating the requested report.
11. A non-transitory computer-readable storage medium containing instructions for execution on a computing device comprising at least one processor, the instructions for:
receiving data from a plurality of external source systems, comprising, from each external source system, receiving a plurality of versions of data, each version of data associated with a time interval associated with the external source system, the versions of data represented in a first format;
storing the data received from the plurality of external source systems along with metadata describing each version of data;
receiving a request to generate report associated with an external source system, the report based on a set of versions of data obtained from the external source system;
transforming the data of the set of versions of data to a second format associated with the report; and
generating a report based on the transformed data.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the first format of the versions of data represents data as one of: row based changes, column based changes, a full snapshot, or information identifying deleted rows.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the second format associated with the requested report represents data as one of: a full snapshot, aggregation of data, change logs, or change logs for a particular column.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the data received from the plurality of external source systems is stored in a columnar format, further comprising instructions for:
converting data received from the external source system from the first format to the columnar format.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the first format of the external source system represents the data as change logs, further comprising instructions for:
applying the change logs to a previous version of data of the external source system to obtain a new version of data.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the second format represents data as changed logs, further comprising instructions for:
determining change logs by comparing at least two versions of data of the external source system.
17. The non-transitory computer-readable storage medium of claim 11 , wherein the second format comprises an aggregation of information across two or more versions of data, further comprising instructions for:
retrieving data from the two or more versions of data of the external source system; and
determining requested aggregation of data over the two or more versions of data.
18. The non-transitory computer-readable storage medium of claim 11 , wherein storing the data received from the plurality of external source systems comprises:
purging older versions of data so as to store more versions of data for recent time intervals compared to older time intervals.
19. The non-transitory computer-readable storage medium of claim 11 , further comprising instructions for:
receiving a request to modify data previously loaded from the external source system;
identifying a version of data stored comprising the data to be modified; and
modifying the identified data in conformance with the received request.
20. The non-transitory computer-readable storage medium of claim 11 , further comprising instructions for:
receiving a request to exclude data previously loaded from the external source system;
identifying a version of data stored comprising the data to be excluded; and
excluding the identified data from the data used for generating the requested report.