Data lake loader
A data loader for loading event data from a streaming platform to an object storage, the data loader comprising: a memory for caching event data; a local file system for storing event data; and a data processor, wherein the data processor is configured to: receive a plurality of batches of event data from the streaming platform; for each batch of event data from the plurality of batches of event data: cache the batch of event data to the memory; write the batch of event data from the memory to the local file system; and combine the plurality of batches of event data stored in the local file system into an output file; and load the output file from the local file system to the object storage.
1 . A data loader for loading event data from a streaming platform to an object storage, the data loader comprising:
a memory for caching event data;
a local file system for storing event data; and
a data processor, wherein the data processor is configured to:
receive a plurality of batches of event data from the streaming platform;
for each batch of event data from the plurality of batches of event data:
cache the batch of event data to the memory; and
write the batch of event data from the memory to the local file system;
periodically combine the plurality of batches of event data stored in the local file system into an output file according to a predetermined combining time interval; and
load the output file from the local file system to the object storage;
wherein said loading comprises transforming the output file into a data warehouse compatible format with separate data and metadata files and partitioned by an event field.
2 . A data loader according to claim 1 , wherein the data processor is further configured to write the batch of event data from the memory to the local file system when the memory capacity is full.
3 . A data loader according to claim 1 , wherein the data processor is further configured to periodically load at least one batch of event data from the output file to the object storage according to a predetermined loading time interval.
4 . A data loader according to claim 1 , wherein the data warehouse compatible format allows a data warehouse to create derived tables by querying the output file stored in the object storage.
5 . A data loader according to claim 1 , wherein the data processor is configured to update the metadata of the output file.
6 . A data loader according to claim 1 , wherein the data processor is further configured to process the plurality of batches of event data from the streaming platform, wherein processing the event data from the streaming platform comprises:
dynamically determining a schema of the plurality of batches of event data; and/or
determining a metadata of the plurality of batches of event data.
7 . A data loader according to claim 1 , wherein the data processor is configured to load any remaining event data stored in the local file system to the object storage if the streaming platform fails.
8 . A data loader according to claim 1 , wherein the data processor is configured to return an acknowledgement message to the streaming platform in response to receiving a batch of event data from the streaming platform.
9 . A data loader according to claim 8 , wherein the data processor is configured to return the acknowledgement message only if all the event data of the batch of event data from the streaming platform has been loaded to the object storage.
10 . A system comprising a plurality of data loaders in accordance with claim 1 for loading event data from a streaming platform to an object storage, each data loader receiving batches of event data from the streaming platform and loading output files to the object storage.
11 . A system according to claim 10 , further comprising a database table for providing a locking mechanism between the plurality of data loaders to prevent conflict between concurrent read and/or write commands from the plurality of data loaders to the object storage.
12 . A data loader according to claim 1 , wherein the event field is a date of an event and/or a name of an event.
13 . A data loader according to claim 1 , wherein the predetermined combining time interval is five minutes.
14 . A data loader according to claim 1 , wherein the data processor is an Apache Spark Cluster and the output file is a resilient distributed dataset (RDD).
15 . A computer implemented method for loading event data from a streaming platform to an object storage via a data loader comprising a memory for caching event data, a local file system for storing event data and a data processor, the method comprising:
receiving, by the data processor, a plurality of batches of event data from the streaming platform;
for each batch of event data from the plurality of batches of event data:
caching, by the data processor, the batch of event data to the memory; and
writing, by the data processor, the batch of event data from the memory to the local file system;
periodically combining, by the data processor, the plurality of batches of event data stored in the local file system into an output file according to a predetermined combining time interval; and
loading, by the data processor, the output file from the local file system to the object storage;
wherein said loading comprises transforming, by the data processor, the output file into a data warehouse compatible format with separate data and metadata files and partitioned by an event field.
16 . A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 15 .