Method and system for delivering data to a batch consumer and a streaming consumer
A method and system delivers data to a batch consumer and a streaming consumer. The method and system include retrieving data from a plurality of data centers, storing the data in a first directory, bundling the data into plurality of batches in the first directory, transferring each batch of the plurality of batches to the second directory, delivering the each batch of the plurality of batches in the second directory to the batch consumer and the streaming consumer and delivering the data in the first directory to the streaming consumer.
1. A method for providing data to a batch consumer and a streaming consumer, the method comprising:
performing with a data analysis engine:
retrieving data from a plurality of data centers, wherein the data is continuously streamed from the plurality of data centers;
storing the data in a first directory;
bundling the data into a plurality of batches in the first directory, wherein bundling the data is in accordance with a predefined time period;
transmitting the plurality of batches to a second directory, wherein each batch of the plurality of batches is immutable;
delivering each batch of the plurality of batches in the second directory to the batch consumer; and
delivering the data in the first directory and each batch of the plurality of batches in the second directory to the streaming consumer.
2. The method of claim 1 , wherein the predefined time period is defined by a Service Level Agreement (SLA).
3. The method of claim 1 , wherein the delivery of the data in the first directory to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer comprises parallel delivery.
4. A system for providing data to a batch consumer and a streaming consumer, the system comprising:
a data analysis engine comprising a processing center having a data retrieval module, wherein the data retrieval module is configured to retrieve data from a plurality of data centers;
the processing center of the data analysis engine further having a storage module, wherein the storage module is configured to store the data, wherein the storage module comprises a first directory and a second directory;
a bundler, wherein the bundler is configured to bundle the data into a plurality of batches in the first directory;
the processing center of the data analysis engine further having a transmitting module, wherein the transmitting module is configured to transmit each batch of the plurality of batches to the second directory; and
the processing center of the data analysis engine further having a delivery module, wherein the delivery module is configured to deliver the data in the first directory to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer.
5. The system of claim 4 , wherein the bundler bundles the data received within a predefined time period in the first directory.
6. The system of claim 4 , wherein during operation the transmitting module transmits the each batch of the plurality of batches to the secondary directory at the end of the predefined time period.
7. The system of claim 4 , wherein during operation the delivery module delivers the each batch of the plurality of batches in the second directory to the batch consumer.
8. The system of claim 4 , wherein during operation delivery module delivers the data in the first directory and the each batch of the plurality of batches in the second directory to the streaming consumer.