Method and system for persisting data
A method and system for persisting data are provided. Batch data is periodically extracted via a computer system from at least one primary data source. Batch data is transformed via the computer system. The batch data is loaded, via the computer system, into a database. Real-time data is continuously received from at least one secondary data source. The received real-time data is indexed in an index. Business intelligence reports are generated from both the real-time data in the index and the batch data in the database.
1. A method for persisting and reconciling one or more data sources on a computer system, the method comprising:
periodically extracting batch data from at least one primary data source,
transforming the batch data into a first format specified by a data schema for a database;
loading the transformed batch data into the database
continuously receiving real-time data from at least one secondary data source;
indexing the received real-time data to generate an index;
generating business intelligence reports comprising the real-time data in the index and the transformed batch data in the database;
storing the real-time data by key-value in a distributed key-value store for subsequent integration into the database, the key-value bucketing a time range of the real-time data having time-series information within the time range of the key-value, and
transforming the real-time data stored in the distributed key-value store into the first format specified by the data schema for the database.
2. The method of claim 1 , further comprising storing the extracted batch data in an interim data store.
3. The method of claim 1 , wherein transforming the batch data comprises at least one of discarding certain data, translating the data to account for differences in a representation of the data, determining the age of the data, sorting the data, joining the data where the data is from multiple sources, and aggregating the data.
4. The method of claim 1 , wherein the at least one primary data source does not publish or push the data immediately.
5. The method of claim 1 , further comprising analyzing the data in the database for generating the business intelligence reports, the analyzing comprising at least one of statistical analysis, machine learning techniques, and data mining.
6. The method of claim 1 , further comprising periodically reconciling the real-time data against the batch data.
7. The method of claim 1 , wherein the real-time data is acquired via at least one of REST requests and a user datagram protocol.
8. The method of claim 1 , wherein the real-time data is pushed onto a distributed message queue system.
9. A system for persisting and reconciling one or more data sources, the system comprising one or more processors and a data storage, the one or more processors configured to execute, or direct to be executed:
an extract-transform-load module to:
periodically extract batch data from at least one primary data source;
transform the batch data into a first format specified by a data schema for a database; and
load the transformed batch data into the database;
a real-time distributed processing module to:
continuously receive real-time data from at least one secondary data source;
index the received real-time data to generate an index;
store the real-time data by key-value in a distributed key-value store for subsequent integration into the database, the key-value bucketing a time range of the real-time data having time-series information within the time range of the key-value; and
transform the real-time data stored in the distributed key-value store into the first format specified by the data schema for the database,
and
at least one business intelligence tool to generate business intelligence reports comprising the real-time data in the index and the transformed batch data in the database.
10. The system of claim 9 , the extract-transform-load module further executable to store the extracted batch data in an interim data store.
11. The system of claim 9 , wherein transforming the batch data by the extract-transform-load module comprises at least one of discarding certain data, translating the data to account for differences in a representation of the data, determining the age of the data, sorting the data, joining the data where the data is from multiple sources, and aggregating the data.
12. The system of claim 9 , wherein the at least one primary data source does not publish or push the data immediately.
13. The system of claim 9 , further comprising a data analysis module executable to analyze the data in the database for generating the business intelligence reports, the analysis comprising at least one of statistical analysis, machine learning techniques, and data mining.
14. The system of claim 9 , wherein the at least one business intelligence tool periodically reconciles the real-time data against the batch data.
15. The system of claim 9 , wherein the real-time data is acquired by the extract-transform-load module via at least one of REST requests and a user datagram protocol.
16. The system of claim 9 , further comprising a distributed message queue system, the extract-transform-load module pushes the real-time data onto the distributed message queue system.