Method and system for delivering data to a batch consumer and a streaming consumer
A method and system delivers data to a batch consumer and a streaming consumer. The method and system include retrieving data from a plurality of data centers, storing the data in a first directory, bundling the data into plurality of batches in the first directory, transferring each batch of the plurality of batches to the second directory, delivering the each batch of the plurality of batches in the second directory to the batch consumer and the streaming consumer and delivering the data in the first directory to the streaming consumer.
1. A method for providing data to a batch consumer and a streaming consumer, the method comprising:
performing with a data analysis engine:
retrieving data from a plurality of data centers, wherein the data is electronically transferred from the plurality of data centers;
storing the data in a first memory location;
bundling the data into a plurality of batches in the first memory location, wherein bundling the data is in accordance with a time period;
transmitting the plurality of batches to a second memory location, wherein each batch of the plurality of batches is immutable;
delivering each batch of the plurality of batches in the second memory location to the batch consumer; and
delivering the data in the first memory location and each batch of the plurality of batches in the second memory location to the streaming consumer.
2. The method of claim 1 , wherein the time period is a predefined time period is defined by a Service Level Agreement (SLA).
3. The method of claim 1 , wherein the delivery of the data in the first memory location to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer comprises parallel delivery.
4. The method of claim 1 , wherein the first memory location comprises a first directory and the second memory location comprises a second directory.
5. The method of claim 1 , wherein the data is electronically transferred comprises the data is continuously streamed from the plurality of data centers.
6. The method of claim 1 , wherein the data is electronically transferred comprises the data is transferred via data batch processes from the plurality of data centers over a predefined period of time and the data analysis engine comprises a batch analysis.
7. The method of claim 6 , wherein the data analysis engine comprises a batch analysis.
8. The method of claim 1 , wherein delivering the data further comprises:
delivering the data in the first memory location and each batch of the plurality of batches in the second memory location to the streaming consumer and a batch consumer.
9. A system for providing data to a batch consumer and a streaming consumer, the system comprising:
a processor; and
a memory, coupled to the processor, including instructions executable by the processor to:
retrieve data from a plurality of data centers;
bundle the data into a plurality of batches in a first memory location in the memory;
transmit each batch of the plurality of batches to a second memory location in the memory; and
deliver the data in the second memory location to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer.
10. The system of claim 9 , wherein the bundler bundles the data received within a predefined time period in the first directory.
11. The system of claim 9 , wherein the instructions are further executable by the processor to transmit the each batch of the plurality of batches to the secondary memory location at the end of the predefined time period.
12. The system of claim 9 , wherein the instructions are further executable by the processor to deliver the each batch of the plurality of batches in the second memory location to the batch consumer.
13. The system of claim 9 , wherein the instructions are further executable by the processor to deliver the data in the first memory location and the each batch of the plurality of batches in the second memory location to the streaming consumer.
14. The system of claim 9 , wherein the first memory location comprises a first directory and the second memory location comprises a second directory.
15. A non-transitory, computer program product comprising instructions stored therein, wherein the instructions are executable by a processor of a data processing system to cause the data processing system to:
retrieve data from a plurality of data centers, wherein the data is electronically transferred from the plurality of data centers;
store the data in a first memory location;
bundle the data into a plurality of batches in the first memory location, wherein bundling the data is in accordance with a time period;
transmit the plurality of batches to a second memory location, wherein each batch of the plurality of batches is immutable;
deliver each batch of the plurality of batches in the second memory location to the batch consumer; and
deliver the data in the first memory location and each batch of the plurality of batches in the second memory location to the streaming consumer.
16. The computer program product of claim 15 , wherein the time period is a predefined time period is defined by a Service Level Agreement (SLA).
17. The computer program product of claim 15 , wherein the delivery of the data in the first memory location to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer comprises parallel delivery.
18. The computer program product of claim 15 , wherein the first memory location comprises a first directory and the second memory location comprises a second directory.
19. The computer program product of claim 15 , wherein the data is electronically transferred comprises the data is continuously streamed from the plurality of data centers.
20. The computer program product of claim 15 , wherein to deliver the data further comprises to:
deliver the data in the first memory location and each batch of the plurality of batches in the second memory location to the streaming consumer and a batch consumer.